E1.S - The future of high performance, high density flash

Author: Jason Adrian - Microsoft Azure Storage Principal Hardware Architect & Open Compute Project Storage Chair

E1.S – Enabling high performance flash for now and the future

High capacity (>= 8TB) ????High performance (>10 GB/s, 100k’s IOPS)? ?? Front serviceable? ?? Integrated thermal solution? ???

TLDR: E1.S is the new form factor taking over the data center. It takes the best of both the U.2 and M.2 formfactors and enables the next era of SSDs including PCIe Gen 5 and beyond.?This isn’t a niche space either, there are at least 8 SSD vendors with E1.S SSDs! E1.S enables incredibly high-performance drives with power envelopes exceeding 25W thanks to integrated heatsink solutions. While the largest consumers of E1.S are Microsoft Azure and Facebook, there are many systems available or soon to be available to the general public from systems providers like Supermicro, Viking, and more.

NAND flash offers an amazing amount of performance at low latencies, which has unlocked many disruptive applications over the last decade. Originally in the U.2 formfactor, and later the M.2 formfactor, there are plenty of modular options for capacities and performance. U.2 has good thermal capabilities to unlock the performance of flash, but it takes up a lot of front bezel space. M.2 offers a compact formfactor, but lacks thermal management. What if you don’t want to compromise? The E1.S formfactor was created to address these tradeoffs. Enterprise and Data Center SSD Form Factor (EDSFF) - A Primer

E1.S – a better M.2

It is expected that the E1.S form factor will completely displace new designs with M.2 in the hyper scale datacenters. It solves some of the challenges that are faced today with regards to thermal management and performance. Many might expect that M.2s can be easily integrated into hyper scale storage systems, but the dirty little secret is that they require adapters and heatsinks to make them functional.

In order to adopt M.2s, hyper scalers resort to using?elaborate adapters and heat sink designs with thermal interface materials to cool the M.2s. One such example is the OCP AVA card, which houses 4 M.2s on a PCI express add-in card. There is a lot of cost and space overhead in the solution, both from the PCB itself as well as the heatsinks that are on the M.2s and PCB. In comparison, E1.S has the heatsink and thermal interface material built-in to the SSD. From an all-up solution cost, the total bill of materials of leveraging 4x E1.S instead of 4 M.2 in the complex AVA solution should be less expensive. Not only that, replacing E1.S is much more?user friendly when mounted at the front of the system. Replacing an M.2 in an AVA card requires powering off the system, removing the system from the rack, pulling out the card, removing the heatsink and M.2, putting in a new SSD and thermal interface material, and redoing the entire assembly to get the server back in operation. This is a messy and time-consuming process. With E1.S, this is a simple drive swap. When you are managing a fleet of millions of systems, time is money, and this simplicity of service is critical. With modern CPUs and PCIe switches supporting PCIe downstream port containment, this service can even be an online operation.

The E1.S formfactor increases the overall dimensions compared to M.2 in order to offer more NAND locations and larger controllers. Marvell’s Bravera SC5 controller reference design is a great example of what is inside an E1.S SSD vs an M.2. As you can see in the image below, an E1.S and an M.2 22x110 are about the same length. However, the E1.S module is wider which allows for 2X or more NAND placements. This is how we enable substantially larger SSD’s as well as enabling high-performance controllers that can address 16 NAND channels. When future E1.S SSD take PCIe Gen 5 controllers to production, such as the Marvell Bravera SC5, we are going to see SSD’s capable of over 10GB/s and 100k’s of IOPS which is remarkable for such a small device.?

E1.S is here and now

Form factor transitions often take years, with limited options at the outset. This is not the case with E1.S, as 8+ SSD suppliers have products that are launching as we speak or in the next few quarters. This gives the end customer an incredible variety of products to choose from once they have designed in the slot form factor. This is particularly?timely during semiconductor shortages or peak periods of demand such as what we are currently experiencing.

If you are looking at the E1.S specification, and are concerned that there are too many options, there is some good news – they are all the same PCB. The only difference in the five versions you see in the specification are the heatsinks to enable various levels of performance. If you simply want a high capacity boot drive or general purpose storage device, the two non-cased versions will do just fine. These are good options to embed inside of a system but they are not easily hot pluggable, and cannot be used for high-performance without heatsink solutions. For higher performance, you need to look at the three cased variants: 9.5mm, 15mm, and 25mm. The 9.5mm will enable the highest density in a front service server or storage system. However, if you end up putting a heavy workload on these?probably going to drive the temperatures up fairly high and might start having to throttle. For the best performance you really want to stick to the 15 or 25 mm variants.

The 15 mm variant arguably is the best compromise on density and performance. This form factor enables up to 24 frontloaded SSDs in a traditional 1U server. This is going to give you an IOPS density that simply can’t be matched leveraging U.2 form factor. The 15mm variant was the result of an open collaboration within the Open Compute Storage workgroup and was a collaboration of 30+ companies. The group evaluated how we drive an optimized width between the 9.5mm and 25mm options as defined by EDSFF. The goal was to enable a >= 25W SSD (30W would enable even more flexibility) with moderate airflow requirements, and the 15mm variant seemed to hit that sweet spot. It enabled the highest IOPS and power density while also opening up the ability to cool up to 30W! Thus, the group converged on this proposal, and partnered with the EDSFF workgroup within SNIA to get this into the TA-1006 rev 1.4 specification. ( https://www.snia.org/technology-communities/sff/specifications )

One of the most remarkable things is how both the OCP Storage group and the SSD manufactures adopted the 15mm variant. The image below showcases 7 vendors supporting 15mm E1.S SSDs, and these are all real SSDs, not mechanical samples. On the NAND based SSD front, we have Intel, NGD, Samsung, SK Hynix, Micron, Fadu, and Kioxia. While most of the NAND SSDs shown here range from 1-8TB, NGD managed to fully use the real-estate to create a 12TB drive! Intel even has their Optane drives available in the E1.S 15mm format, which offers some exciting options to mix in high performance and high endurance 3DXP. Not shown in the montage below is Western Digital, which does have E1.S SSDs, but currently the 25mm variant as seen at top of the second image below.

E1.S – Enabling Dense & Innovative Systems

E1.S is enabling a whole new generation of systems, with higher IOPS density and simple serviceability. Many of today’s 1U servers can fit 10x U.2 SSDs at best. With E1.S 15mm, you can run 24 SSDs with a substantial increase in performance. That offers unprecedented IOPS density in a rack.

What about the early adopters? Supermicro was one of the first to adopt the E1.S formfactor, with a system that housed a whopping 32 9.5mm E1.S drives. At the time, the 15mm variant was not available, but neither were PCIe Gen 4 SSDs, thus 9.5mm is a reasonable formfactor for today’s PCIe Gen 3 SSDs. Looking forward, Supermicro is working on a new latch to enable the 15mm E1.S to plug into todays systems. This highlights the beauty of the E1.S formfactor, a slot can handle a 9.5mm, 15mm, or 25mm depending on how many adjacent slots you are willing to give up. In the current Supermicro system, you can now use 16 15mm E1.S SSDs in the box with no-changes to the system, but they are working on some new latches that will balance the airflow instead of leaving empty slots. For their future platforms, they are investigating solutions with 24 15mm drives.

Looking forward to new designs, Viking has a very interesting offering that really highlights the capabilities of combining the 15mm E1.S with the high PCIe lane count from AMD’s Epyc (Milan) CPUs. Here they have put 2 single socket systems into a 1U chassis with 24 15mm E1.S. Each server directly connects to 12x E1.S, with x4 Gen 4 lanes. By avoiding the need for PCIe switches, this is an excellent example of a cost and performance optimized platform. This really highlights the capabilities of the fantastic Epyc Milan CPU’s and the 128 PCIe Gen 4 lanes, as well as the new PCIe Gen 4 SSDs that are capable of amazing performance both in terms of IOPS as well as sequential transfer speeds.

Much ado about latches

One of the interesting quirks with E1.S is that it has the LEDs (green/amber) built into the SSD, along with a mounting location to install a latching mechanism. The standardized LED feature is certainly welcome, something that was non-existent on U.2 and M.2. The latches, however, are left to the consumer to supply. In the EDSFF specs the mount tab and screw holes are specified, but each vendor has their own designs to match their systems industrial design. Shown here, in order, are the Microsoft, Supermicro, and Viking latches.

Samsung brought a very interesting proposal to the OCP Storage track last year that would bring an open-source option for a tool-less latch. This is the type of innovation and industry wide collaboration that is happening around E1.S, including entire system contributions in this space.?

E1.S – Not just a data center formfactor

Over the last few years, the consumer market has also seen challenges with M.2’s. Most motherboards have multiple M.2 slots, but consumers started seeing performance throttling when they were doing any substantial workloads. The market adjusted, with motherboards bundling heatsinks for the M.2’s, or shipping PCIe adapter cards to house the M.2’s with heatsinks. This is an excellent space that is ripe for manufactures targeting high-end desktops and workstations to enable E1.S solutions. It will be interesting to see how E1.S gets enabled in these high performance machines over the coming years.?System designers in this space, the ball is in your court.

E1.S Adoption – A rapid shift from m.2

The E1.S form factor should quickly replace M.2 in the datacenter, but it is also likely to take some of the market using U.2 today. The thermal capabilities, performance options, and the broad vendor adoption are fueling this rapid change. IDC projects that E1.S is going to make a big splash in 2022, and eclipse M.2 by 2023!?

Conclusion

The E1.S formfactor has been a testament to the EDSFF workgroup and the wider industry who have quickly brought this form factor from concept to reality. Building on the original specs, the OCP Storage workgroup demonstrated how an open collaboration across the industry could align the hyper-scalers, OEMs, system designers, and SSD vendors around a new 15mm variant. Here we are, 18 months since the group aligned on 15mm, and we have nearly every SSD supplier debuting their devices. This is enabling high density servers and storage systems with unparalleled IOPS density and serviceability. The 15mm variant is going to unlock some amazing performance with its 25W+ capabilities combined with PCIe Gen 5 SSDs. It is great to see the industry embrace this formfactor, and we are looking forward to the next generation of platforms taking advantage of the capabilities.