Cisco ACI Traffic Forwarding Simplified | Part III

This article is part Three of a series dedicated to exploring how ACI manages various types of traffic, including Layer 2, Layer 3, and ARP traffic.

Previous Articles :

Part 1 : https://www.dhirubhai.net/pulse/cisco-aci-forwarding-simplified-part-i-mohammed-khalefa-xvkwf/

Part 2 : https://www.dhirubhai.net/pulse/cisco-aci-traffic-forwarding-simplified-part-ii-mohammed-khalefa-ayndf/

COOP and Identity to Location mapping

Overview

The COOP (Council of Oracle Protocol) running in the ACI fabric is a critical component for maintaining accurate and consistent endpoint information across the fabric. It provides efficient communication and mapping between endpoints and their locations. Let’s break down its key functions:

What the COOP Database Does

The COOP database, managed by the spine switches, is responsible for maintaining a centralized and synchronized record of endpoint identities and their corresponding locations across the ACI fabric.

Identity-to-Location Mapping

·? Identity: This includes the details of the endpoints:

o?? Endpoint A with MAC A and IP A.

o?? Endpoint B with MAC B and IP B.

o?? Endpoint C with MAC C and IP C.

· Location: Each endpoint is mapped to the leaf switch it is connected to:

o?? Endpoint A is located behind Leaf 1.

o?? Endpoint B is located behind Leaf 2.

o?? Endpoint C is located behind Leaf 3.

This ensures that all spine switches maintain a consistent copy of the identity-to-location mapping, enabling efficient forwarding and endpoint learning across the fabric.

How Information Is Sent to the Spines

Each leaf switch communicates its local endpoint information to the spine switches (or Oracles) using a protocol known as Zero Message Queue (ZMQ).

COOP Data Components

>> MAC-to-VTEP Mapping

One of the primary functions of the COOP database is to maintain the mapping between:

· MAC addresses of endpoints and their corresponding VTEP addresses (Virtual Tunnel Endpoints).

Example:

· The MAC address of Endpoint A is mapped to the VTEP address of Leaf 1.

· The MAC address of Endpoint B is mapped to the VTEP address of Leaf 2.

>> IP-to-VTEP Mapping

While MAC-to-VTEP mapping is the primary focus, the COOP database can also store IP-to-VTEP mappings in certain scenarios:

· If unicast routing is enabled within the fabric.

· If routed traffic is received by a leaf, allowing the leaf to learn the endpoint’s IP address.

Example:

· The IP address of Endpoint A is mapped to the VTEP address of Leaf 1.

· This additional mapping ensures that routed traffic can also be efficiently forwarded.

Endpoint Identity and Location Learning

This section explains the process of endpoint communication in ACI, step by step:

1. The Setup

·??????? The setup consists of three ACI leaf switches: Leaf 1, Leaf 2, and Leaf 3.

·??????? Endpoint A (with MAC A and IP A) needs to communicate with Endpoint C.

2. Initial Steps

· Endpoint A sends a packet.

· When the packet is sent, the ingress leaf (in this case, Leaf 1) processes the packet first.

· The ingress leaf examines the packet's source MAC address and source IP address.

o?? Leaf 1 learns the MAC address and IP address of Endpoint A from the packet itself, using the data plane.

· After learning this information, Leaf 1 reports it to one of the spine switches, which maintains the COOP database.

o?? Example: Leaf 1 informs Spine 1: "I have learned a new endpoint: Endpoint A with MAC A and IP A."

3. Packet Transmission

· If Leaf 1 already knows that the destination, Endpoint C, is behind Leaf 3:

o?? Leaf 1 creates a VXLAN tunnel to send the packet across the ACI fabric.

o?? The packet travels from Leaf 1 to Leaf 3 through this VXLAN tunnel.

4. Role of the Spine

· When Leaf 1 reports the new endpoint information to Spine 1, the spine updates its COOP database.

· Spine 1 synchronizes this updated information with other spines (e.g., Spine 2) to ensure consistency.

o?? This synchronization occurs through control traffic between the spines and is separate from the data plane.

5. Egress Leaf Processing

· When the packet reaches Leaf 3 (the egress leaf), it processes the packet:

o?? Leaf 3 may learn the source MAC address or the source IP address from the inner VXLAN packet after decapsulating the VXLAN header.

6. Completion

· After processing, Leaf 3 forwards the packet to Endpoint C.

· At this stage, the communication process is complete.

Summary

This step-by-step explanation highlights how ACI ensures seamless endpoint communication by leveraging VXLAN tunnels, the COOP database, and the coordinated roles of leaf and spine switches.

>> Now let’s discuss another crucial aspect of endpoint identity and location in Cisco ACI:

Bridge Domain Configuration for Unicast Routing

· Under each bridge domain, there’s an option called unicast routing.

o?? Activated: Routing is enabled on the fabric, and the system will also enable IP learning for endpoints.

o?? Deactivated: The leaf will not learn IP addresses for the endpoints, limiting its knowledge to MAC addresses only.

When Unicast Routing is Enabled

The leaf can learn both the MAC address and the IP address of endpoints under specific conditions. Here’s how it works:

For Local Endpoints:

The leaf learns IPv4 or IPv6 addresses when:

1.??? An ARP/GARP is received from the endpoint.

2.??? Routed traffic is received, indicating communication between endpoints in different subnets.

Example:

· Suppose we have Leaf A with a local endpoint, Endpoint A, connected to it with:

o?? MAC Address: MAC A

o?? IP Address: IP A

· The learning process is as follows:

1.??? Leaf A naturally learns the MAC address of Endpoint A as soon as it connects.

2.??? If unicast routing is enabled:

§? When Endpoint A sends an ARP/GARP, Leaf A also learns its IP address.

§? If routed traffic is sent (e.g., communication between subnets), Leaf A learns the IP address through the data plane.

In these cases, the leaf will store both the MAC address and the IP address of the endpoint.

For Remote Endpoints:

Now consider Leaf B, which has Endpoint B connected to it. If Endpoint A (connected to Leaf A) needs to communicate with Endpoint B (connected to Leaf B):

· Leaf B learns about Endpoint A as a remote endpoint.

· Depending on the type of traffic received, Leaf B learns different information:

o?? Bridged Traffic: If Endpoint A and Endpoint B are in the same subnet, Leaf B learns only the MAC address of Endpoint A.

o?? Routed Traffic: If Endpoint A and Endpoint B are in different subnets, Leaf B learns only the IP address of Endpoint A.

See You in the next one,

Mohammed Khalefa

01.Feb.2025

Next Article :

PART4 : https://www.dhirubhai.net/pulse/cisco-aci-traffic-forwarding-simplified-part-iv-mohammed-khalefa-05anf/