A Server Designed for 2x200GbE!

It appears Dell's engineers may have collaborated with NVIDIA when designing their new Intel Sapphire Rapids server. Looking closely at the photos, with the proper selection of a PCIe riser, the R760 may correctly house and fully utilize NVIDIA's double-wide BlueField-3 2x200G DPU, and that's saying something!?

This week Intel allowed Dell to quote regular customers pricing for their new Sapphire Rapids servers. Of specific interest is Dell's industry-leading 2U R760 server. The R760 can support up to eight PCIe Slots spread across four riser bays, some with multiple riser options offering numerous slots. Dell calls out sixteen uniquely different PCIe slot configurations in their Technical Guide for this server.

Furthermore, the Dell Specification Sheet for this server identifies slot seven as specifically supporting SNAPI. Slot seven is a single wide PCIe Gen5x16 Full-height, Half-length. For those unfamiliar with SNAPI, it is a clever way to split the PCIe load of a DPU between the two CPU sockets on a dual CPU server. Typically, servers without SNAPI send all the DPU's networking traffic to the single CPU socket managing all the PCIe lanes of the DPU's PCIe slot. To be clear, Dell uses the term SNAPI which, according to the IEEE Workgroup, stands for Storage Network Architecture and Parallel I/Os. NVIDIA uses SNAP, Software-defined Network Accelerated Processing, and then explains how BlueField maps to storage.?

Specifically, it appears that Dell's R760 "4P Riser," shown below, was explicitly designed for the NVIDIA BlueField-3 2x200G DPU. This riser exposes just slot seven (no slot eight) as a PCIe Gen5x16 into a double-wide space ideal for this card. As we can see from this inside shot of the server, it even looks like the bulk of the airflow is channeled toward the top of the BlueField-3 server, where all the Heatsink fins reside. ?

Unfortunately, slot seven is also one of the two primary GPU slots, so GPU power should also be available, but this reduces the sever from two double-wide DPUs to a single double-wide. Riser option "4P-FL" appears to be designed to support full-length, double-wide accelerator cards like FPGAs and future DPUs.?

Finally, riser option "4R" might be worth exploring for Accelerators that require significant PCIe I/O as it offers slot seven's PCIe Gen5x16 and slot eight's PCIeGen5x8 (even though the picture displays a physical x16 wide connector Dell's documentation calls it out as x8). Having these additional eight lanes for I/O would give an accelerator card another 256 Gbps to the host. This would mean that the "4R" riser card option could potentially open up a whopping 768 Gbps path from the Ethernet to the server's CPUs to support 2x400GbE operation, WOW!

Soon I'll explore Lenovo and SMC's Sapphire Rapids offerings.