ACI Multi-Site Overview || Part#1

Overview

Cisco Application-Centric Infrastructure (ACI) Multi-Site is an architectural approach that enables you to interconnect separate Cisco ACI fabrics with independent Cisco Application Policy Infrastructure Controller (APIC) cluster domains. It provides a simple way to manage the different Cisco ACI fabrics with a complete fault and change domain separation.

Another important component in the Multi-Site design is the Inter-Site Network (ISN), The Intersite Network (ISN) facilitates the connection between the sites, so you should know its components, requirements, and details of the ISN control plane operations.

While exploring the communication in Cisco ACI Multi-Site, you should know how unicast and broadcast, unknown unicast, and multicast (BUM) traffic is handled across sites.

To be able to achieve this design of Multi-Site, there are some functional components:

• Cisco Nexus Dashboard Orchestrator (NDO): This component is the intersite policy manager. It is an orchestrator that is positioned on the top of the Cisco APICs in different sites, providing a single-pane management. You deploy the Cisco Nexus Dashboard Orchestrator as an application in Cisco Nexus Dashboard.
• Inter-site Control Plane: Endpoint reachability information is exchanged between sites using a Multiprotocol Border Gateway Protocol (MP-BGP) Ethernet VPN (EVPN) connection, which is established between the spine nodes that are deployed in separate fabrics that are managed by the same instance of Cisco NDO.
• Inter-site Data Plane: All communication between endpoints connected to different sites is achieved by establishing site-to-site Virtual Extensible LAN (VXLAN) tunnels across a generic IP network that interconnects the sites.

As a centralized management pane, Cisco ACI NDO is used when defining the intersite policies that you can deploy across separate Cisco ACI fabrics, while each site has a separate Cisco APIC cluster domain with associated configuration and policies.

The MP-BGP EVPN control plane between sites enables the exchange of MAC and IP address information for the endpoints that communicate across sites, while VXLAN data plane is used to allow intersite Layer 2 and Layer 3 communication.

So now after we know the main functional components of Cisco ACI Multi-Site, here is Important question, which is:

How policy information is applied on the traffic and communication exchanged between fabrics?

The use of site-to-site VXLAN encapsulation greatly simplifies the configuration and functions required for the intersite IP network. It also allows network and policy information (metadata) to be carried across sites.

As shown in the below figure:

The VXLAN Network Identifier (VNID) identifies the below network information:

• Bridge domain for layer-2 communication.
• Or the VRF instance for Layer-3 communication of the endpoint sourcing the traffic.

While the Class-ID* is the unique identifier of the source Endpoint Group (EPG), CLass-ID is the "pcTag*".

And because of a complete separate and independent APIC domain and fabric are deployed at each site, a translation function (referred as name-space normailization) must be applied before the traffic is forwarded inside the receiving site, to ensure that receiving values are same as the locally significant values EPG, bridge domain, and VRF instance are used.

The Cisco ACI Multi-Site facilitates end-to-end policy definition and enforcement using the Cisco ACI NDO to to ease of this translation between different site.

This function is applied to ensure that a Cisco ACI NDO can orchestrate not only a brand-new ACI fabric but also existing ACI fabrics that may be already using overlapping VNIDs and class IDs in each site. The translation logic ensures that locally significant values for the VNID and class ID of each site, which identify the same bridge domain, VRF instance, and source endpoint group (EPG), can still be used within each site without worrying about the conflict with existing IDs in each site.

How this translation function works?

In the below example, the traffic within the same EPG across two sites. When the traffic is across two EPGs, NDO defines a contract between those two EPGs.

If one EPG is exist only in one site, the cisco NDO create shadow service EPG is created in the other site that doesn't has this EPG, in order to enable that site to perform the EPG Class-ID translation and apply the needed policy/contract between those EPGs, EPG in site-1 and the other EPG in site-2.

When the policy is created on Cisco Nexus Dashboard Orchestrator stating that “EP1 EPG” must communicate with “EP2 EPG,” the Nexus Dashboard Orchestrator receives from each APIC controller the specific identifiers (pcTag, L2VNI, L3VNI) assigned to the local and shadow objects, and instructs the APIC controllers to program proper translation rules in the local spines.

The end result is that the configured policy can then correctly be applied on the leaf node before sending the traffic to the destination endpoint.

Resources:

Cisco Application Centric Infrastructure - Cisco ACI Multi-Site Architecture White Paper - Cisco

To know more about Class-ID or pcTag check this link:

Class ID / pcTag-1

Class ID / pcTag-2

In next article will go through the ISN and its requirments and control plane