System Design : Building Electronic Stock Exchange

Step 1: Functional and Non-Functional Requirements

Functional Requirements:

1. Trading Operations: Support placing/canceling limit orders.
2. Order Book: Real-time order book visibility for clients
3. ?Trade Matching: Real-time trade execution on order match.
4. Risk Management: User limit (e.g., max 1 million shares/day)
5. Funds Management: Ensure sufficient funds and withhold for pending orders.
6. Stock Trading: The system will only support stock trading. Advanced features such as hedging, leveraging, speculative trading, and commodity or FX trading (including currency pairs, futures, options, swaps, and forwards) are not included in this design.
7. Trading Hours: Operate during regular market hours (9:30 am?—?4:00 pm ET).

Non-Functional Requirements:

1. Availability: 99.99% uptime
2. Fault Tolerance: Fast recovery from incidents
3. Latency: Millisecond-level round-trip time, focus on the 99th percentile.
4. Throughput: Support billions of orders/day, peak QPS up to 215,000.
5. Scalability: Easily extendable to more users, symbols, and volume
6. Security: KYC checks, DDoS protection.
7. Compliance: Adhere to regulatory risk management rules.

Step 2: High Level?Design?

Step 3: LLD?—?API?Design

Step 3: LLD?—?Data?Model

Step 3: LLD?—?Deployment And Performance Optimizations

Based on the diagrams, the stock exchange system design effectively manages the following key aspects:

Performance

• ?Critical Path Optimization?: The critical path, involving components like the Order Manager, Sequencer, and Matching Engine, is optimized for performance by reducing the number of tasks and minimizing network and disk usage. Communication is handled using mmap, which allows memory-mapped communication, minimizing latency.
• CPU Pinning: Each critical service (Order Manager, Sequencer, Execution Service) is pinned to specific CPUs to avoid context switching, ensuring high throughput and low-latency execution.

Event Sourcing

• Sequencer: The Sequencer plays a critical role in event sourcing, ensuring all events (orders and executions) are logged in a deterministic order. These events are stored in the Event Store for future replay and recovery.
• Immutable Event Log: The Event Store keeps an immutable log of all state-changing events, which helps in reconstructing the system’s state by replaying events. This provides traceability and makes it easier to debug issues by tracing how the state evolved.

High Availability:

• ?Warm Matching Engine: The Backup Server hosts a warm instance of the Matching Engine, which processes the same data as the primary instance but does not send out data. In case of failure, it can immediately take over without significant downtime.
• ?Replication: The event sourcing design ensures that all events are replicated across instances, guaranteeing minimal data loss and near-instant recovery during failover scenarios.

Fault Tolerance

• Hot-Warm Setup: The hot-primary and warm-backup design ensures that even if the primary system fails, the backup can take over immediately, reducing the risk of downtime.
• Cluster Leader Election: In case of multiple failures, the system uses a leader election algorithm (like Raft) to elect a new leader, ensuring that a single point of failure does not cause the entire system to collapse.

Matching Algorithms

• ?Order Matching: The Matching Engine is responsible for maintaining the Order Book and performing buy/sell matching based on predefined algorithms. This ensures trades are executed efficiently, minimizing latency between the placement of orders and their execution.
• Market Data Publisher: After orders are matched, the Market Data Publisher updates order books and candlestick charts, which are critical for real-time market insights.

Determinism

• Functional Determinism: Through event sourcing and the sequencer, the system ensures that given the same sequence of events, the outcome will always be the same, which is critical for consistency.
• Latency Determinism: The system strives to achieve consistent latency, particularly focusing on the 99th percentile latency. Techniques like CPU pinning and avoiding disk and network access help achieve near-constant latency for each trade.

Market Data Publisher Optimization

• ?Efficient Memory Management: The Market Data Publisher uses ring buffers, which are efficient, lock-free data structures that minimize memory allocation overhead and ensure that real-time market data is processed without bottlenecks.
• Multicast: The Market Data Publisher distributes data to multiple subscribers simultaneously using multicastover reliable UDP, ensuring all clients receive market updates in near real-time.

Fairness

• ?Multicast Fairness: The Market Data Publisher ensures that all subscribers receive market data simultaneously through multicast. This avoids scenarios where certain clients might receive critical data ahead of others, ensuring a level playing field.
• Colocation Service: VIP clients can reduce latency by physically placing their servers in the same data center as the exchange through colocation services. This minimizes the network latency between their systems and the exchange’s matching engine.

Multicast

• Market Data Distribution: The exchange uses multicast to distribute real-time market data (order books, candlestick charts) to multiple clients at once. This is essential to ensure all clients get the same data at the same time, preventing unfair advantages.
• Reliable UDP: Multicast is implemented over reliable UDP, which provides low-latency communication while ensuring that packets reach all subscribers.

Colocation

• Minimal Latency: By offering colocation services, the exchange allows certain clients (hedge funds, brokers) to minimize the physical distance between their systems and the exchange’s core services. This ensures minimal network latency, giving these clients an edge in ultra-low-latency trading environments.

Network Security

• Firewall and DDoS Protection: The Firewall component protects the system from external attacks, such as DDoS attacks. Public and private services are separated to ensure that even if public services are targeted, the core trading system remains unaffected
• Rate Limiting: The Client Gateway includes rate limiting to protect the system from malicious clients attempting to overwhelm the exchange with requests.