Cisco ACI Traffic Forwarding Simplified | Part V

This article is part five of a series dedicated to exploring how ACI manages various types of traffic, including Layer 2, Layer 3, and ARP traffic.

Previous Articles:

Part 1 : https://www.dhirubhai.net/pulse/cisco-aci-forwarding-simplified-part-i-mohammed-khalefa-xvkwf/

Part 2 : https://www.dhirubhai.net/pulse/cisco-aci-traffic-forwarding-simplified-part-ii-mohammed-khalefa-ayndf/

Part 3 : https://www.dhirubhai.net/pulse/cisco-aci-traffic-forwarding-simplified-part-iii-mohammed-khalefa-urzjc/

Part 4 : https://www.dhirubhai.net/pulse/cisco-aci-traffic-forwarding-simplified-part-iv-mohammed-khalefa-05anf/

Switched vs. Routed Traffic

Let’s revisit a fundamental point that simplifies everything about handling traffic in an ACI fabric.

It all comes down to one critical question:

Is the traffic switched/bridged, or is it routed?

That’s the single most important rule to remember.

Why This Rule Matters:

When determining how traffic is handled in the fabric, it doesn’t matter whether:

· The traffic is between two endpoints in the same Endpoint Group (EPG) or across different EPGs.

· The traffic is within the same bridge domain or across different bridge domains.

None of these details are as significant as answering this one question: Is the traffic switched or routed?

What Does It Mean?

1.??? Switched/Bridged Traffic:

o?? This happens when:

§? Endpoints are in the same subnet.

o?? The traffic remains Layer 2 (L2) and does not require routing.

o?? It is forwarded based on MAC addresses and remains within the bridge domain.

2.??? Routed Traffic:

o?? This happens when:

§? Endpoints are in different subnets.

o?? The traffic requires Layer 3 (L3) routing to reach its destination.

o?? It is forwarded based on IP addresses, and routing occurs across bridge domains or subnets.

Layer 2 Forwarding Flowchart

Let’s go through how ACI handles Layer 2 forwarding. In this section, we’ll use a flowchart to explain the process, and then we’ll look at how ACI handles ARP and Layer 3 traffic in future topics. These flowcharts help us understand how ACI manages Layer 2 traffic, ARP requests, and Layer 3 traffic.

Let’s start with the Layer 2 forwarding flowchart. The basic rule is simple:

· If Endpoint A and Endpoint B are in the same subnet, it’s Layer 2 traffic.

· If they are in different subnets, it’s Layer 3 traffic.

So, ACI first checks if the traffic is Layer 2 or Layer 3. This is done by looking at the destination MAC address.

Here’s how it works:

1.??? If the destination MAC address is the ACI MAC address (the MAC address of the default gateway on the ACI leaf switch), ACI knows that Endpoint A is trying to reach a different subnet. In this case, it will send the traffic to the default gateway, which means it's Layer 3 traffic.

2.??? If the destination MAC address is not the ACI leaf’s MAC address, ACI understands that the two endpoints are in the same subnet. This is Layer 2 traffic, so ACI will forward the traffic based on the MAC address, not the IP address.

o?? If the traffic is Layer 2, ACI checks the destination MAC address.

o?? If the MAC address is learned locally on the same ACI leaf, the traffic is forwarded directly to the local port.

o?? If the MAC address is not local, ACI forwards the traffic to the egress leaf or the leaf where the endpoint is located.

But what happens if ACI doesn’t know the destination MAC address? This can happen if the MAC address hasn’t been learned yet. In this case, ACI uses the L2 Unknown Unicast option.

· If L2 Unknown Unicast is set to FLOOD, the packet will be flooded across the entire bridge domain until it reaches the correct destination.

· Alternatively, ACI can use a hardware proxy for L2 Unknown Unicast. Here’s how it works:

o?? If the leaf doesn’t know where the MAC address is, it forwards the traffic to the spine switch.

o?? The spine switch checks its COOP database to see if it knows where the MAC address is. If it does, the spine sends the traffic to the correct egress leaf (like Leaf 2 in our example).

o?? If the spine switch doesn’t have the MAC address in its COOP database, the traffic will be dropped.

Example Scenrios for ACI Layer 2 Forwarding

Let's look at how ACI handles different Layer 2 forwarding scenarios.

1.??? Case 1: Destination MAC is known to the ingress leaf

o?? If the destination MAC address is known to the ingress leaf (the leaf switch receiving the packet) and the endpoint is connected to the same leaf, the packet will be forwarded directly to the correct port.

o?? For example, if Endpoint 1 is connected to Ethernet 1/1 and Endpoint 2 is connected to Ethernet 1/20 on Leaf 1, and both MAC addresses are in the Endpoint Table of Leaf 1, the leaf will forward the packet directly to Ethernet 1/20.

2.??? Case 2: Destination MAC is known, but the endpoint is on a remote leaf

o?? If the destination MAC address is known to the ingress leaf, but the endpoint is connected to a different leaf (for example, Leaf 2), the packet will be forwarded to the remote leaf.

o?? For instance, if Leaf 1 knows the MAC address of Endpoint 1 but Endpoint 2 is connected to Leaf 2, Leaf 1 will forward the packet to Leaf 2 using a VXLAN tunnel (from Leaf 1 to Leaf 2) to reach Endpoint 2.

3.??? Case 3: Destination MAC is not known to the ingress leaf

o?? If the destination MAC address is not known to the ingress leaf (meaning the leaf doesn’t know where the endpoint is), ACI will handle it in two ways: flooding or using hardware proxy mode.

o?? Flooding:

§? If flooding is enabled, the packet will be sent to all endpoints in the bridge domain. For example, if Leaf 1 doesn’t know where Endpoint 2’s MAC address is, it will flood the packet across the entire bridge domain.

§? When flooding happens, the packet is encapsulated with a multicast IP address (called a group IP Outer or GIPO), which is assigned to this bridge domain, such as 226.1.1.1.

o?? Hardware Proxy Mode:

§? If hardware proxy mode is configured, the packet will be sent to the spine switch. Leaf 1 will ask the spine where Endpoint 2 is. The packet will be encapsulated and sent towards the spine using an anycast address.

§? If the spine has Endpoint 2’s MAC address in its COOP database, it will forward the packet to the destination leaf (for example, Leaf 2).

§? However, if the spine doesn’t have the MAC address (for example, if Endpoint 2 is inactive), the spine will drop the packet. as seen below

These are the main ways ACI handles Layer 2 forwarding. Depending on whether the destination MAC address is known locally, remotely, or unknown, ACI will choose the best method to make sure the traffic reaches its destination.

Let’s review the Layer 2 forwarding flow chart again

When a packet reaches the leaf switch, the first thing the leaf switch needs to do is decide whether the traffic is Layer 2 or Layer 3.

· Layer 3 Traffic: If the destination MAC address matches the ACI MAC address (the default gateway MAC), then it’s Layer 3 traffic.

· Layer 2 Traffic: If the destination MAC address does not match the ACI MAC address, then it's Layer 2 traffic.

For Layer 2 traffic, the leaf switch checks if it knows the destination MAC address:

· If the leaf knows the MAC address, it forwards the packet either to a local port (if the endpoint is local) or to a remote port on another leaf (if the endpoint is remote).

· If the leaf doesn’t know the MAC address, it will look at the bridge domain L2 Unknown Unicast configuration to decide what to do. There are two options for this:

1.??? Flooding:

§? If flooding is enabled, the packet will be sent to all endpoints in the bridge domain. The packet is sent to a multicast group IP Outer address (GIPO), allowing it to be flooded across the Bridge domain until it reaches the correct destination.

2.??? Hardware Proxy:

§? If flooding is not enabled, the leaf will use a hardware proxy. It will ask the spine switch to check its COOP database (where MAC addresses are stored).

§? If the spine knows where the MAC address is, it will forward the packet to the correct leaf.

§? If the spine doesn’t know the MAC address (for example, if the endpoint is silent), it will drop the packet.

That's it.

See You in the next one,

Mohammed Khalefa

7.Feb.2025

Next Article:

PART6 : Soon