How cool is NVMe ? Part 2 -> Throughput

So I've decided to put down my snark hunt for the moment, and get technical. There are lots of claims about how much faster NVMe is than boring old fashioned SCSI and SAS, which are for the most part universally accepted as being true. One good example of this is the whitepaper from nvmeexpress.org, you’ll see them say

“By supporting PCI Express and Fabrics such as RDMA and Fibre Channel, NVM Express can support much higher bandwidths than SATA or SAS (e.g., a PCI Express Gen3 x4 delivers 4 GB/s)”

I’m not suggesting that the overall thrust of what they’re saying is wrong, NVMe devices do see better throughputs, but the example they used isn’t great because the theoretical bandwidth you can get out of 12 Gbit SAS is pretty impressive, certainly better than 4GB/sec.

If you’re interested in some technical reading Broadcom put out a good document here https://docs.broadcom.com/docs/12353459 that says the following

Like PCIe, SAS uses lanes, and high-performance storage systems normally aggregate multiple SAS lanes to support higher data rates. A third generation SAS x4 wide port, for example, provides a maximum half-duplex aggregate transfer rate of 48Gb/s or 4800MB/s, as shown in Table 2.

So a 4-lane SAS port is going to give you more throughput than a 4-lane PCIe / NMVe interface .. and based on the following graph a 12Gbit SAS card can more than saturate a PCIe 3.0 slot running with 8 “lanes”, though I’m not aware of any SAS cards that run more than 4 lanes, so this graph seems a bit “marketing theoretical”, but it’s interesting to see in the face of similar marketing claims from the NMVe side of the house when they say “NVM Express can support much higher bandwidths than SATA or SAS”

There are improvements to both of these technologies in the pipeline and the SCSI trade association said here that they will retain a theoretical throughput advantage over PCIe 

“Kutcipal claimed the 19.2 Gbps bandwidth would have a 21.5% per-lane performance advantage over non-volatile memory express (NVMe) running on top of PCI Express (PCIe) 4.0. The maximum bandwidth for single-lane PCIe 4.0 is 15.8 Gbps“

That whole search storage article is interesting for this discussion but at the same time it reminds me of the arguments of Infinband vs Fibre Channel vs Ethernet .. technically interesting, but for the most part people don’t really care because the decision about which of those technologies you use has more to do with IT organisational boundaries (Storage Team vs Network Team vs HPC team) than the actual technology.

It’s important in this discussion to remember that NVMe and SAS are communication protocols, not device types, and the main justification for the “NMVe has better throughput” claim is that SAS attached devices (SSDs and Disks) appear to use only one SAS lane, whereas high performance NMVe attached devices typically use four PCIe lanes, and some already use 8 lanes. If you're interested in some measured "real world" figures, theres an interesting comparison at Tom's IT Pro from about a year ago here and I've tabulated some of those results from a throughput perspective here

So the per device improvements for NVMe are impressive, though its worth noting that none of those devices, which were tested a year ago, are hitting the theoretical throughput of any of the interfaces they're attached to, with the SAS device getting the closest at around 85% and the 8x lane device is only getting about 50% of the potential while consuming 8 PCI lanes (keeping in mind that your typical consumer Intel processor can only address 16 lanes in total)

When you’re talking about installing two or three devices dedicated to a single server those per device improvements are going to be significant but you're going to have to budget your PCIe lane capacity to make sure you're getting the best out of them. However when you’re attaching those devices to an array controller where you are typically aggregating the throughput of twenty or more devices the relative improvement isn’t going to be nearly as great, probably a useful incremental bump, but hardly the orders of magnitude improvements we saw when moving from disk to flash.

Hopefully that's put some useful perspective on the throughput improvements you can expect of out NVMe compared to 12G SAS. In my next post I'm going to cover something much more exciting .. latency, and maybe really really exciting stuff like queue depths and littles law and queuing theory .. I bet you can't wait .. right :-)

Other Posts in this series

• How cool is NVMe ? Part 1 - Snark Hunting
• How cool is NVMe ? Part 2 -> Throughput
• How Cool is NVMe ? - Part 3 - No waiting in Queues
• How cool is NVMe ? – Part 4 – CPU and Software Efficiency
• How cool is NVMe – A summary of benefits for external storage arrays