Deploying Virtual ACI Multi-Site Lab

ACI MultiSite design is the architecture interconnecting multiple APIC cluster domains with their associated pods. A MultiSite design could also be called a Multi-Fabric design, because it interconnects separate regions (fabrics) each deployed as either a single pod or multiple pods (a Multi-Pod design).

The need for complete isolation (both at the network and tenant change-domain levels) across separate Cisco ACI networks led to the Cisco ACI Multi-Site architecture, introduced in Cisco ACI Release 3.0(1).

First article discussed how to install the Application Centric Infrastructure Simulator on VMware ESXi. This second article will help install a virtual ACI Multi-Site Lab and understand some concepts and configurations on MSO like Schema, Template, Tenant, L2 / L3 Stretching and some inter-site policies. 

The Virtual Multi-Site LAB Architecture :

• Two ACI fabrics built with ACI Simulator
• One APIC cluster domain in each fabric with one spine and two leafs nodes
• One ACI Multi-Site Orchestrator used to manage the two fabrics and to define inter-site policies.

Scenario 1 : Installing Single Node Orchestrator in VMware ESX

Scenario 2 : Multisite Orchestrator GUI Walkthrough

Scenario 3 : Adding Sites to MSO

Scenario 4 : Creating Tenant, Schema and Templates for MSIT

Scenario 5 : Adding Application Profile, EPG, BD, VRF to Schema Templates

Scenario 6 : Creating policy inside the templates and deploying to sites

---------------------------------------------------------------------------------------------------------------

Scenario 1 : Installing Single Node Orchestrator in VMware ESX

Caution: Single node installations are supported for testing purposes only. Production Multi-Site deployments require a 3-node Orchestrator cluster.

Step 1 : Download the Open Virtualization Appliance (OVA) file.

• Download the ACI Multi-Site Image (ESX Only) file (esx-msc-<version>.ova) for the release.

Link below : https://software.cisco.com/download/home/285968390/type

• Click ACI Multi-Site Software and Choose a Cisco ACI Multi-Site Orchestrator release version.

In my lab i’m using 3.1(1g) release.

Step 2 :  Deployed the OVA directly in ESX

• Confirm the virtual machine allocated the required Disk, CPU and Memory.
• Power on the virtual machine.

Step 3 : Complete the MSO initial Setup

• Log in to the VM using the console
• Username = root , Password = cisco
• It will be required to change the password immediately (root enforced). This should be a complex password.

• Type nmtui for to configure networking basics :

Caution : No routing could be done between the spines and ISN so it is important to put Eth0 of MSO and APICs in the same subnet. Do not forget we are in a full virtual lab, the ACI Simulator includes simulated switches, so you cannot validate a data path.

• Click on OK, go back deactivate and reactive this connection
• Set System Hostname

• Click and Exit the Network Manager TUI
• Ping one APIC to validate connectivity between MSO and the ACI Simulator

• Log in to the VM now using SSH

• Move into the deployment scripts directory

# cd /opt/cisco/msc/builds/<build_number>/prod-standalone

•  Run the initialization script

#./msc_cfg_init.py

• Run the deployment script.

# ./msc_deploy.py

•  Wait Deployment complete

Step 4 : Log in ACI Multi-Site Orchestrator GUI

• The default login is admin and the default password is We1come2msc!
• When you first log in, you will be prompted to change the password.

Scenario 2 : Multisite Orchestrator GUI Walk-through

Functionality of each Multi-Site Orchestrator GUI page is described in the following link

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/aci_multi-site/sw/3x/configuration/cisco-aci-multi-site-configuration-guide-3x/m-gui-overview.html

Scenario 3 : Adding sites to MSO

• GO to Infrastructure > Sites

• Click the Add Site button to add each site one by one

1. Name: the site name.
2. APIC Controller URL: add APIC Controller URL.
3.  Username: the site username.
4.  Password: the unique site password for access.
5.  APIC Site ID : Each Site must have a unique Site ID

• Click save if all inputs are OK, a warning message will confirm Site is successfully connected to the MSO
• If the inputs are not good a fail message will appear

•  Add second site

• Go to Operation > Sites Firmware
• Check Overview of all Sites Firmeware

• Logon an APIC and check the topology, you can see now Inter-Pod Network

Scenario 4 : Creating Tenant, Schema and Template in MSO

• Schema: Profile including the site-configuration objects that will be pushed to sites. Schemas are the containers for single or multiple templates that are used for defining the policies.
• Templates : are the framework for defining and deploying the policies to the sites. Child of a schema, a template contains configuration-objects (App Profile, EPG, VRF, Contract, BD, Filter, External EPG) that are shared between sites or site-specific.
• Tenant : A secure and exclusive virtual computing environment. In Cisco ACI, a tenant is a unit of isolation from a policy perspective, but it does not represent a private network

Use Case :

1. A VRF ‘VRF_stretched’ containing three Bridge Domains. This VRF will be stretched beetween DC1 and DC2
2. Bridge Domain ‘BD1_Stretched’ will be pushed to both sites
3. Bride Domain ‘BD1_DC1’ will be pushed to DC1 only
4. Bride Domain ‘BD1_DC2’ will be pushed to DC2 only

For this we need to create at least 3 different Templates :

1. One Template for the stretched BD / EPG that will be pushed to both site
2. One Template for each BD that will be pushed to one site only
3. Create a Schema containing the three Templates

Step 1 : Creating Tenant

• Go to Application Management > Tenant

• Click on Add Tenant

• Choose associated sites and save
• Do Same for each Tenant that will be deployed on each site only

• Lets verify on APIC for each site

You can see Tenants are deployed => Tenant_DC_ALL on all sites, Tenant_DC1 on site DC1 and Tenant_DC2 on site DC2

Below on APIC you can see thos object are created on MSO

Step 2 : Creating Schema and Template

• Go to Application Management > Schemas

• and Click Add Schema

• Go to Application Management > Schemas choose schema created in last step and create Template

• Default is Template 1 => give a name and select Tenant will be assocciated to this Template
• Associate Tenant to Schema Template

Template is created with a list of objects (APP, VRF, BD, etc ...) by default with options to activate for this Tenant or not

• Create Template_DC1, Template_DC2 and Template_DC_ALL and associate each with Tenant_DC_ALL

Step 3 :Associate Site(s) and Template(s)

• One Template for each BD that will be pushed to one Site only

• One Template for the stretched BD / EPG that will be pushed to both sites

• Verify complete schema configuration

Scenario 5 : Adding Application Profile, EPG, BD, VRF, vzAny to Schema Templates

• Go to Application Management > Schemas>Template_DC_ALL

• Add Application Profile

• Add VRF (you can choose to activate L3 Multicast, VzAny.....)

• Add BDs

• Add Subnet to BD

• Add EPGs

• Associate EPGs to BD and choose some options related

• Deploy configuration to sites

• Now create EPG_DC1 / BD1_DC1 and EPG_DC2 / BD1_DC2 that will only respectevely deployed on DC1 and DC2

• Deploy for each site
• Do same for EPG_DC2 / BD1_DC2  

• Checks on APICs

Objects created in MSO is tagged. On APIC DC1 we can see 2 Tenants created on MSO : Tenant_DC1 and Tenant_DC_ALL (stretched on the 2 sites)

On APIC DC2 we can see 2 Tenants created on MSO : Tenant_DC2 and Tenant_DC_ALL (stretched on the 2 sites)

• Below all objects created in MSO and deployed on APIC DC2

vzAny Multi-Site

• Step 1 : Create Filter

Step 2: Create Contract

• Contract scope could be VRF, Tenant or Global

Step 3: Configure vzAny to Consume/Provide the Contract

Step 4: Deploy the vzAny contract

Step 5: Checks on APIC

• Contract deployed

• Go to Tenant_DC_ALL > Networking>VRF_Stretched

Scenario 6 : Creating policies and deploying to sites

DHCP Relay Policy

• Go to Application Management > Policies

• Choose Create DHCP Policy
• Add Name of the Policy
• Associate it to the Tenant Tenant_DC_ALL

• Add Provider => select the EPG where located the DHCP Server

• Select Application EPG (server located inside ACI) and give the IP address of the server

• Click Save

• Popup shows POlicy successfully created

• DCHP Server EPG is tagged

• Activate DHCP Policy on BDs and choosethe relay Policy

• Deploy and lets Verify on APIC

• Policy is created on each APIC
• Below Policy attached to BDs

Limitations

The ACI Simulator includes simulated switches, so you cannot validate a data path. It is for this reason that the Infra part has not been configured in this lab. In Production you will have to do this. The goal here is to allow you to have a fairly realistic lab that can allow you to familiarize yourself with ACI Multi-Site, design, configuration and interaction with APIC

References :

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739609.html

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/aci_multi-site/sw/2x/installation/Cisco-ACI-Multi-Site-Installation-Upgrade-Guide-221.html

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/aci_multi-site/sw/3x/configuration/cisco-aci-multi-site-configuration-guide-3x/m-gui-overview.html