How cool is NVMe ? Part 2 -> Throughput
So I've decided to put down my snark hunt for the moment, and get technical. There are lots of claims about how much faster NVMe is than boring old fashioned SCSI and SAS, which are for the most part universally accepted as being true. One good example of this is the whitepaper from nvmeexpress.org, you’ll see them say
“By supporting PCI Express and Fabrics such as RDMA and Fibre Channel, NVM Express can support much higher bandwidths than SATA or SAS (e.g., a PCI Express Gen3 x4 delivers 4 GB/s)”
I’m not suggesting that the overall thrust of what they’re saying is wrong, NVMe devices do see better throughputs, but the example they used isn’t great because the theoretical bandwidth you can get out of 12 Gbit SAS is pretty impressive, certainly better than 4GB/sec.
If you’re interested in some technical reading Broadcom put out a good document here https://docs.broadcom.com/docs/12353459 that says the following
Like PCIe, SAS uses lanes, and high-performance storage systems normally aggregate multiple SAS lanes to support higher data rates. A third generation SAS x4 wide port, for example, provides a maximum half-duplex aggregate transfer rate of 48Gb/s or 4800MB/s, as shown in Table 2.
So a 4-lane SAS port is going to give you more throughput than a 4-lane PCIe / NMVe interface .. and based on the following graph a 12Gbit SAS card can more than saturate a PCIe 3.0 slot running with 8 “lanes”, though I’m not aware of any SAS cards that run more than 4 lanes, so this graph seems a bit “marketing theoretical”, but it’s interesting to see in the face of similar marketing claims from the NMVe side of the house when they say “NVM Express can support much higher bandwidths than SATA or SAS”
There are improvements to both of these technologies in the pipeline and the SCSI trade association said here that they will retain a theoretical throughput advantage over PCIe
“Kutcipal claimed the 19.2 Gbps bandwidth would have a 21.5% per-lane performance advantage over non-volatile memory express (NVMe) running on top of PCI Express (PCIe) 4.0. The maximum bandwidth for single-lane PCIe 4.0 is 15.8 Gbps“
That whole search storage article is interesting for this discussion but at the same time it reminds me of the arguments of Infinband vs Fibre Channel vs Ethernet .. technically interesting, but for the most part people don’t really care because the decision about which of those technologies you use has more to do with IT organisational boundaries (Storage Team vs Network Team vs HPC team) than the actual technology.
It’s important in this discussion to remember that NVMe and SAS are communication protocols, not device types, and the main justification for the “NMVe has better throughput” claim is that SAS attached devices (SSDs and Disks) appear to use only one SAS lane, whereas high performance NMVe attached devices typically use four PCIe lanes, and some already use 8 lanes. If you're interested in some measured "real world" figures, theres an interesting comparison at Tom's IT Pro from about a year ago here and I've tabulated some of those results from a throughput perspective here
So the per device improvements for NVMe are impressive, though its worth noting that none of those devices, which were tested a year ago, are hitting the theoretical throughput of any of the interfaces they're attached to, with the SAS device getting the closest at around 85% and the 8x lane device is only getting about 50% of the potential while consuming 8 PCI lanes (keeping in mind that your typical consumer Intel processor can only address 16 lanes in total)
When you’re talking about installing two or three devices dedicated to a single server those per device improvements are going to be significant but you're going to have to budget your PCIe lane capacity to make sure you're getting the best out of them. However when you’re attaching those devices to an array controller where you are typically aggregating the throughput of twenty or more devices the relative improvement isn’t going to be nearly as great, probably a useful incremental bump, but hardly the orders of magnitude improvements we saw when moving from disk to flash.
Hopefully that's put some useful perspective on the throughput improvements you can expect of out NVMe compared to 12G SAS. In my next post I'm going to cover something much more exciting .. latency, and maybe really really exciting stuff like queue depths and littles law and queuing theory .. I bet you can't wait .. right :-)
Other Posts in this series
- How cool is NVMe ? Part 1 - Snark Hunting
- How cool is NVMe ? Part 2 -> Throughput
- How Cool is NVMe ? - Part 3 - No waiting in Queues
- How cool is NVMe ? – Part 4 – CPU and Software Efficiency
- How cool is NVMe – A summary of benefits for external storage arrays
Private Cloud Engineer
6 年Thanks for posting.
The idea is not only an application that needs it but the consolidation of multiple applications across a smaller number of large SSD's. For example, what is the performance numbers 160TB of data spread across Netappp's 15.8TB SSD's using 12Gb SAS. Not good! That's where NVMe shines with many threads and many queues. I can tell you it's huge compared to 12Gb SAS with real workloads not synthetic
John your messaging seems at odds with that of some of your architects and engineers. There is follow up to this article both are well written and explain why NVMe is so important to the storage industry as a whole just the same as when we went from FCAL to SAS at the back end was replacing a serialised protocol with a massively parallel protocol that sits closer to the CPU is having huge impact already. https://community.netapp.com/t5/Technology/NVMe-A-Step-into-the-Future/ba-p/127309
If you have a problem that is impossible to solve, ask me about it. We help people do the impossible everyday.
7 年Don't confuse marketing with Math :-)