From Monolith to Thread-Per-Core: Lessons Scaling a Real-Time Exchange
The Latency Cliff of Shared-Memory Concurrency
In traditional multi-threaded systems, concurrency is managed through shared state protected by mutexes, condition variables, and thread pools. Under moderate traffic, this model functions adequately; but as throughput scales to millions of orders per second, cache-line bouncing, context switching, and lock contention inevitably trigger crippling latency spikes.
When our team redesigned our core matching engine, we abandoned the shared-memory paradigm entirely in favor of an asymmetric, thread-per-core architecture written in bare-metal Rust.
Architectural Principles of Thread-Per-Core Engines
By treating each physical CPU core as an independent, isolated computer, the architecture achieves deterministic execution guarantees:
- Core Pinning & CPU Isolation: Each worker thread is strictly bound to a dedicated physical core with operating system interrupts and kernel scheduling shielded away.
- Lockless Ring Buffers: Cross-core communication occurs strictly via single-producer single-consumer (SPSC) circular ring buffers aligned to cache-line boundaries (64 bytes), eliminating false sharing entirely.
- Run-to-Completion Polling: Threads continuously poll IO queues using DPDK and io_uring, avoiding the costly overhead of sleep/wakeup cycles and system calls.
Production Results and Key Takeaways
The migration transformed our latency distribution: p99.9 latency fell from 14.2 milliseconds to an ultra-consistent 8.4 microseconds, with zero tail outliers even during extreme market volatility. For high-throughput infrastructure, shared-nothing concurrency is the definitive path forward.
