ARP Flooding in ACI Fabric

In this article, lets see ARP traffic over the ACI Fabric.

By default, ARP flooding is disabled within a bridge domain.? Since switches in the ACI fabric learn an endpoint’s IP address and the leaf the endpoint is behind, ARP traffic does not need to be flooded over the fabric. Instead the ARP broadcast can be sent directly to the destination leaf switch.

Currently, IP address 5672-B are in the same subnet as 5672-A.

Now lets ping our bridge domain gateway from 5672-B. This will allow your endpoint IP and MAC address to get learned by leaf 104

Lets verify it from Leaf 104. Highlighted below confirms 10.1.80.3 address being learned.

Once the leaf learns about the endpoint it should send an update via COOP to the spine switch. Lets confirm it from spine 201

Indeed it is being learnt. However, leaf switch 101 should not learn about the endpoint

Here's what we see:

1. Leaf switch 104 learned local endpoint 10.1.80.3
2. Leaf switch 104 sent an update to spine via COOP about endpoint 10.1.80.3
3. Leaf switch 101 does not know about the endpoint 10.1.80.3 yet.

We will now send an ARP broadcast from 5672-A (10.1.80.2) to 5672-B (10.1.80.3).? Currently only leaf switch 104 and the spine switches know about the 5672-B (10.1.80.3) endpoint.? Leaf switch 101 does not know about the 5672-B (10.1.80.3) endpoint.

What will happen if the ingress switch (leaf 101) does not know where the endpoint is? Lets find out

On 5672-B, running ethanalyzer to capture ARP broadcast traffic sources from 5672-A

In the output of the show command above the ARP broadcast is being sourced from 5672-A’s MAC and IP address (this is important to take note of because the source MAC will change when doing an ARP in a different scenario)

Because the spine switch and egress leaf switch know where the 10.1.80.3 endpoint resides, the ARP broadcast is sent directly to 5672-B and not flooded over the ACI fabric.

Lets see if Leaf 101 knows the destination endpoint.

It indeed knows. Now that the ingress leaf switch 101 knows about the 10.1.80.3 endpoint, it will send an ARP broadcast directly to egress leaf switch 104.? Here is how ACI fabric will handle an ARP broadcast if it ingresses into leaf 101 and is attempting to retrieve 10.1.80.3’s MAC address:

1. ARP Broadcast is sent from 5672-A, looking for 10.1.80.3's MAC
2. Leaf 101 does know where the 10.1.80.3 endpoint is (via Spine) which is directly connected to 104
3. ARP broadcast is sent directly to endpoint since leaf 104 knows which interface the endpoint is behind.

Lets clear our ARP entries on 5672-A and send another broadcast to 10.1.80.3 endpoint to find out.

We should see an identical ARP broadcast that is being sourced from 5672-A’s MAC and IP address.

ACI can reduce ARP flooding.

• If ARP flooding is disabled, the ARP packet is L3 looked up for target IP in VRF
• ARP behaves like a L3 unicast packet
• If an endpoint is connected at L2 (i.e. that never sends traffic to the default gateway) never sent an ARP packet, the directed ARP feature cannot work.
• ARP flooding is required when you need gratuitous ARPS to update hosts ARP caches or router ARP caches. This is the case when an IP address may have a different MAC (e.g. clustering of failover of load balancers and firewalls)

ARP Flooding - ON

ARP flooding optimization (to remove ARP flooding) are great, but there are many situations when ARP flooding should be kept on.

Taking into account diverse workloads types the recommendation is to keep ARP flooding on for these reasons:

• NIC teaming
• Firewall devices failover
• Floating IP addresses in clusters

That's all there is to it! I hope this example has been useful to you. In our next article lets understand unicast routing in Bridge domain.