Different types of chassis on the Networking side (Switches and Routers)

Different types of chassis on the Networking side (Switches and Routers):

—------------------------------------------------------------------------------

Here are the form factors i have come across in my experience:

1. Fixed form factor switches - Typically in the 1RU/2RU/3RU/4RU form factor
2. Modular switches - Typically in the 5RU+ category
3. Fixed-Modular (Flexible Modular) switches - Typically in the 2RU to 4RU range.
4. Distributed Disaggregated Chassis (DDC) - Multiple smaller chassis joined together to appear like one large chassis.

Let me go through these one at a time along with the plus and minus for these. There is no one size that fits all and they all have a role to play depending on the customers and their individual use cases.

1. Fixed form factor switches:

As ASICs pack more bandwidth in a single chip, a single switch system based on one ASIC can meet the connectivity requirements for a traditional rack. Fewer components mean fewer failures, lower cost and power. Since they take up less power, cooling them is much easier compared to the modular switches. Fixed form factor switches offer the best port density and lower price points since they pack so many components in their 1RU-4RU form factor.

Another major advantage with the fixed form factor switches is the availability of the Original Design Manufacturer (ODM) choices and lower COGS (Cost of Goods) to purchase them due to the competition from the ODMs. Most of these switches have no Non-Recurring Cost (NRE) when they are purchased as white boxes or with some minor changes in memory (DRAM) or CPU or storage (SSD). This helps in having a lower upfront cost for the OEMs who want to buy these from the ODMs.?

Vast majority of the switches in the Enterprise or the Data Center space are in 1RU and some in 2RU fixed form factor. Fixed form factor switches comprise more than 75% of the data center switching market by revenue. The Top of the Rack (ToR) or the leaf switches are typically in the 1RU form factor since they connect to the servers in the rack and the rack space will limit the number of servers. Leaf or the aggregation switches are in 1RU or in the 2RU form factor. Rarely they go up to 4RU - when they are used in the middle of the row or in the end of the row where they aggregate multiple racks.

2. Modular switches:

Fully modular switches have dual Supervisors or Route Processors for redundancy along with the switch fabric redundancy also. These chassis have line card slots, and we can have 4 to 16 slots allocated for these line cards. If the line cards are of half width (example, Brocade/Extreme’s MLXe-32), then 32-line cards can fit in one chassis.

Modular chassis models are good if you have varying types of line cards like Ethernet, SONET, ATM or serial interface line cards. Even in Ethernet, you can have 1 Gbps, 10 Gbps, 25 Gbps, 40 Gbps, 100 Gbps, 400 Gbps or 800 Gbps line cards separately addressing the various price points. Since the world has mostly converged on Ethernet, the value from the Modular switches has diminished over time. Still anyone wanting the ‘pay as you grow’ model and want single management for a large number of ports can opt for the modular switches.

The modular chassis can be used for the end-of-row or border use cases in a data center. So, features like VxLAN, MPLS, larger route tables, internet peering and other advanced routing features become very important.?

Big modular switches come with big price tags too but the redundancy they offer is very important to protect the whole chassis which is a big failure domain. Modular chassis are expensive to start with and they consume a lot higher power, hence needing bigger fans to cool them. Needless to say, that they need a lot more real estate to place them. This is the reason why the growth with the modular switches is actually declining while the growth with the fixed form factor switches is still very good. Some customers in Asia, Europe or the Federal customers still love to buy the big modular switches for the redundancy and for the future growth perspectives.?

3. Fixed-Modular (Flexible Modular) switches:

These are the switches that look modular, but they don’t have the redundant Supervisor or Route Processor that are present in the fully modular switches/routers for true High Availability (HA) or for In-Service Software Upgrade (ISSU). However, they do offer the ‘Pay as you grow’ model where individual line cards can be added as the need grows. This is a future protection for those customers who are expecting future growth but don’t want to buy all the ports up front.

These are generally available in the 2RU/3RU/4RU form factor and could be used in the middle of the row in the data centers to aggregate multiple racks. Examples: VSP 8400 from Extreme Networks (acquired from Avaya) is in 2RU while the Extreme 9920 is in 4RU form factor. Arista’s 7368X4 and 7388X5 are also fixed-modular 4RU switches. Arista calls them as flexible modular switches since this modularity is meant for flexibility of the interfaces.

4. Distributed Disaggregated Chassis (DDC) - targeted for the AI traffic:

Distributed Disaggregated Chassis (DDC) is a way to achieve the functionality of a modular chassis but without purchasing that huge expensive chassis upfront. Instead, a virtual chassis is formed by adding multiple fixed form factor switches as leaf switches which then connect to the fabric in the spine layer. This is basically a cluster of switches but acts as one large switch. You can view this as a gigantic modular switch where the leaf switches behave like the line cards while the fabric gets moved externally to the spine layer. Splitting the system into multiple chassis is easy with the power and cooling requirements instead of having a huge modular chassis that needs huge power supplies and fans to cool the heat generated from them.

Broadcom has been promoting this idea with their recent ASICs - Jericho2C+ leaf switches and ramon spine that scales to 18,000 x 400 Gbps ports and their newer Jericho3-AI based leaf switches and ramon3 spine that scales to 32,000 x 800 Gbps ports. They also have the choice of Tomahaw5 based leaf and spine but that needs the intelligent end points like the intelligent NICs as scheduler. Broadcom seems to promote the ‘switch scheduled’ approach where the Jericho2C+ or the Jericho3-AI leaf switches make all the decisions while their ramon or ramon3 fabric is used just for connectivity for these leaf switches.

For those interested in more details, Broadcom’s Tech Field Day video on DDC is an interesting one to listen to -

Broadcom Jericho3 AI Ethernet Fabric - YouTube

DriveNets, an Israeli startup has already jumped into this DDC bandwagon. They use their DriveNets OS and their Orchestration software for DDC while using the hardware from their ODM partners like Ufispace and Accton (Edgecore). This is basically a disaggregated software that can run in any white box hardware. DriveNets is promoting this architecture as their solution for AI Networking. For those interested can read the information from this link and listen to their YouTube video at?

CloudNets - AI Networking Solutions - DriveNets

While the DDC above looks very close to a Clos based Ethernet fabric (leaf/spine/super-spine topologies), there are some key differences -?

1) DDC for AI has near perfect ECMP load balancing due to traffic getting sprayed as cells (Broadcom’s own format) equally in all links. So, there is no risk of saturating a few links due to the flow-based load balancing with some large incast traffic.

2) Zero traffic loss since the traffic doesn’t get sent in the fabric till the receiver is ready - yes, yes, way different from the usual Ethernet’s CSMA/CD where it is all done via best effort and retransmissions ??

The main difference between the regular Ethernet traffic and the AI traffic is that AI traffic has fewer flows, but they are all long lasting elephant flows. Job Completion Time (JCT) is really critical for the AI networks and so networks are supposed to be highly efficient and with ultra-low latency. Typically, this is the domain for the InfiniBand, but Ethernet for AI is picking up steam with the efforts from the Ultra Ethernet Consortium (UEC) who are trying to promote the RoCEv2 and the cell-based traffic streaming for near perfect load balancing. This UEC is a whole new exciting initiative and let’s stay tuned for more to unfold!