Storage and I/O Tuning in Linux for Ultra-Low Latency Trading

In my previous articles, I covered CPU tuning, Network Optimization using Solarflare NICs, and Memory tuning for ultra-low latency trading. Often, storage and I/O tuning are overlooked compared to CPU, memory, and network optimization. However, optimizing storage and I/O is crucial. Imagine hardware interrupts stacking up due to a poorly optimized disk. A single disk-intensive thread can block others, causing delays.

When optimizing for performance, ensure data integrity is not compromised, as compliance in trading and exchange systems is legally mandatory.

Use Fast Storage:

The backbone of low-latency systems is fast storage. NVMe SSDs offer significantly lower latencies than traditional SSDs or HDDs. They provide direct CPU access via the PCIe bus, reducing read/write times and ensuring high throughput. Always monitor vendor releases for the latest hardware and features. It’s not just a recommendation but a necessity to consult with vendors and read their manuals regularly to stay updated on the latest innovations in storage.

Filesystem Selection:

XFS and EXT4 are the most suitable filesystems due to their low overhead and efficient journaling. Journaling helps maintain filesystem consistency by logging changes before applying them, allowing for quick recovery after crashes. Use data=writeback mode for the lowest latency if you can tolerate some risk. Adjust the journal commit interval to balance performance and data safety.

Other Filesystems:

• ZFS: Offers advanced caching (ARC and L2ARC) but requires careful tuning for low latency.
• Btrfs: Provides features like copy-on-write and built-in RAID but may introduce latency.

EXT4 and XFS still remain the most reliable choices for trading systems.

I/O Scheduler Tuning:

Linux I/O schedulers manage read/write queue processing:

• none: Bypasses scheduling, ideal for NVMe devices, minimizing latency.
• mq-deadline: Ensures I/O request deadlines, reducing spikes. Suitable for modern multi-queue storage.
• bfq: Fair allocation of I/O bandwidth, mainly for multi-tenant environments.

For ultra-low latency, use none for NVMe and mq-deadline for other storage types.

Direct I/O and Buffer Bypass:

Use Direct I/O (O_DIRECT) to bypass kernel buffers and avoid unnecessary data copying between user space and kernel space. This approach is highly beneficial for applications requiring predictable and low-latency I/O.

Disk Write Caching:

Enable disk write caching to boost write performance. However, understand that while it increases speed, it risks data loss in case of power failure. For critical data, ensure write caching is backed by reliable power sources or UPS systems. Needless to say DO NOT LOSE DATA in the process of improving the performance.

Reduce Dirty Page Ratios:

Tuning the Linux virtual memory system can reduce I/O spikes:

Adjust dirty background ratio and dirty ratio to lower values, ensuring data is written to disk more frequently. This prevents large write bursts and helps maintain consistent I/O latency. I wrote more about this in Memory tuning article.

Asynchronous I/O (AIO):

Leverage Asynchronous I/O to allow non-blocking read/write operations. By decoupling I/O processing from application threads, AIO can enhance throughput and lower latency, especially for applications handling multiple I/O requests.

Align Partitions Properly:

Improper partition alignment can cause additional I/O operations, affecting performance. Align partitions to the disk's physical block size (typically 4K or 8K) to ensure optimal read and write efficiency.

I/O Affinity and NUMA:

Bind I/O operations to local NUMA nodes, similar to CPU and memory tuning, to prevent cross-node latencies. Tools like numactl can be used to bind storage processes to specific NUMA nodes.

Final Thoughts:

Do not lose data. Optimizing storage and I/O configurations is essential in achieving ultra-low latency trading performance my experience. By leveraging fast storage, selecting appropriate filesystems, using direct I/O, and reducing memory pressure, you can minimize I/O-related latencies. When combined with CPU, Network, and memory tuning, these practices ensure a well-rounded low-latency trading infrastructure.