Deploy Cisco VXLAN EVPN Multi-Site with EVE-NG & VMware ESXi (Part II)

The main functional component of the EVPN Multi-Site architecture is the border gateway (BGW). BGWs separate the fabric-side (site-internal fabric) from the network that interconnects the sites (site-external DCI) and mask the site-internal VTEPs.

There?are?no Control Plane or Underlay (VTEP, Point-2-Point, Loopback etc) relationship requirements between sites. Only BGW IP Addressing must be Unique and Aligned between Sites.

Intra-Site VXLAN tunnel terminates at a Border Gateway which, from the same VTEP, reinitiates a new VXLAN tunnel toward the remote sites (Inter-Sites).

2 possible deployment models :

• Anycast Border Gateways
• vPC Border Gateways

?Each model meets a specific need. Example for the integration of a legacy network to a VXLAN fabric, a pair of BGWs in VPC model is deployed rather than Anycast model.

2 BGW function enablement in the VXLAN EVPN fabric:

• BGWs on Leaf node (Border Gateway Leaf)
• BGWs on Spine node (Border Gateway Spine

This lab is built with BGWs on Leaf node + Anycast model. eBGP is used to provide reachability between the BGWs.

FOR BUM

• Option to use multicast or ingress replication inside the fabric/site and ingress replication across different fabrics/sites. One site can run PIM ASM Multicast to replicate the BUM traffic inside the same VXLAN domain regardless of what the replication mode used on other sites
• PIM BiDir is not supported for fabric underlay multicast replication with VXLAN Multi-site.
• PIM is not supported on Multi-site VXLAN DCI links.
• Prior to NX-OS 10.2(2)F only ingress replication was supported between DCI peers across the core. Beginning with Cisco NX-OS Release 10.2(2)F both ingress replication and multicast are supported between DCI peers across the core.
• ?The BUM traffic is carried across the Multi-Site overlay using Ingress Replication

In this lab I use Ingress Replication in the Inter-Site area along with Multicast in the Intra-Site area in order to manage BUM Traffic. ?So in BGW both multicast routing and unicast replication methods are enabled per L2 VNI and in interface NVE.

Topology

Scenario 1: Underlay between BGW and Spines

• Configure interfaces
• Configure IGP
• Configure Multicast

Step 1: Activate all features on BGWs

Step 2: Configure interfaces into routed ports and enable jumbo frames:

Validation

At this stage, L1 connectivity done between BGWs and DCI-CORE

Step 3: Configure interface L3 Point-to-Point

Validation

At this stage L3 connectivity is done between BGW – SPINE and BGW and DCI-CORE

Step 4: Configure site-internal underlay and loopback interfaces

• OSPF is used to provide reachability between the intrasite VTEP (leaf), spine, and BGW.
• Loopback0 for router ID and BGP peering. ?
• Loopback1 for the individual VTEP (PIP) used for BUM in Anycast BGW
• Loopback100 interface as the EVPN Multi-Site source interface (anycast and virtual IP VTEP). The VIP is for communications between the Border Gateways in different Sites as well as for communications between the Border Gateways and Leaf’s within a Site.

Note :

• All these 3 loopback must be advertised to the site-internal underlay as well as to the site-external underlay.
• In this labs I use only one BGW peer site, in Anycast design mode (not VPC), this loopback100 IP must be configured on all site BGWs, In real situation should have at least 2 BGWs per site for redundancy.

Validation

At this stage the BGW have established OSPF sessions with spines and all loopback advertised and reachable.

Step 5: Configure Site-internal Underlay Multicast

Note: The ip pim sparse-mode setting is needed only for site-internal multicast-based BUM replication.

At this stage the BGW have established PIM sessions with the spines. Loopback254 ?shared by spines as anycast PIM RP

10.10.10.254 for Anycast RP of Site-1 spines and 20.10.10.254 for Anycast RP of Site-2 spine

Scenario 2: Configure Site-internal Overlay MP-BGP EVPN

Note: For configurations between Spines and other leafs check Part I. Below concerns only Spine – BGW

Step 1 : Configure the BGP overlay for the EVPN address family

Spines are the Route Reflector for L2VPN EVPN address families

Validation

At this stage the BGW have established BGP L2VPN EVPN sessions with the spines. EVPN adsress-family is active.

• BGW-S1 (Lo0?: 10.10.10.100) peering with Spine-1001 ( Lo0 10.10.10.101) and Spine-1002 (10.10.10.102)
• BGW-S2 (Lo0?: 20.10.10.100) peering with Spine-2001 ( Lo0 20.10.10.101)

Scenario 3: Configure Multisite underlay between BGW and DCI-CORE

Step 1: Configure L3 interfaces

?Note: The ip pim sparse-mode setting is not needed because site-external BUM replication always uses ingress replication

Step 2: Configure eBGP between BGW and DCI-CORE

Activate the BGP IPv4 unicast global address family (VRF default) to redistribute the required loopback and physical interface IP addresses within BGP. So border gateways at the two sites can communicate with each other.

Validation

At this stage IPv4 BGP Neighborships are functional, each BGWs receives the Loopback routes for each other this is mandatory for multisite Overlay

Step 3: Configure Overlay between BGW-S1 and BGW-S2

Multi-Site Overlay Peering BGP L2VPN EVPN neighborships

Overlay eBGP L2VPN EVPN neighborships are built directly between the 2 Border Gateways' Loopback IP address.?

peer-type fabric-external

• Enables Next-Hop Rewrite for Multi-Site
• Defines Site External BGP neighbors for EVPN exchange

rewrite-evpn-rt-asn

• Rewrites Route-Target Auto information to simplify MAC-VRF and IP-VRF configuration
• Normalizes outgoing Route-Targets AS number to match remote AS number
• Uses BGP configured Neighbors Remote AS

Note: Site-external EVPN peering is always considered to use eBGP with the next hop the remote site BGWs.

eBGP L2VPN EVPN address family between sites will advertises MAC and IP addresses between sites. Border gateways should see each other directly as next-hop address, This is required to create VXLAN tunnel between border gateways.

• evpn multisite fabric-tracking? : EVPN Multi-Site interface tracking for the site-internal underlay. This command is mandatory to enable the Multi-Site virtual IP address on the BGW. At least one of the physical interfaces that are configured with fabric tracking must be up to enable the Multi-Site BGW function (keeping the virtual IP VTEP address active).
• evpn multisite dci-tracking : EVPN Multi-Site interface tracking is used for the site-external underlay. This command is mandatory to enable the Multi-Site virtual IP address on the BGW. At least one of the physical interfaces that are configured with DCI tracking must be up to enable the Multi-Site BGW function.
• Should configure first “evpn multisite border-gateway site-id” to activate evpn multisite fabric-tracking and evpn multisite dci-tracking?

Validation

At this stage the BGW of the 2 sites have established BGP L2VPN EVPN sessions each other. Multisite EVPN adsress-family is active. Overlay neighborship between BGW is built directly to remote site loopback

Scenario 4: Configure VXLAN L2/L3VNI Intersite

?Step 1?: Configure overlay VRF and VNI-Segment

?Step 2: ?Configure Vlans and VXLAN VNI

Layer3 VNI associated interface vlan?does not have an ip address used as layer3 VNI to route Inter-VNI traffic

Validation

Step 3: Configure Network Virtualization Endpoint (NVE) interface

Note: Site-external BUM replication uses ingress replication. Site-internal BUM replication can use multicast (PIM ASM) or ingress replication.

?Ingress Replication is used for Inter-Site BUM traffic for all VNI while Intra-Site BUM traffic?uses?Multicast

Validation

Note : Leaf-102 and Leaf-103 are powered off to save CPU and RAM. they are not part of this part II.

?Same for Site-2 BGW-S2 ( peering with Internal VTEP + Site-2 BGW-S2 PIP and VIP)

The virtual IP address on the BGW is used for all data-plane communication leaving the site and between sites when the EVPN Multi-Site extension is used to reach a remote site. The single virtual IP address is used both within the site to reach an exit point and between the sites, with the BGWs always using the virtual IP address to communicate with each other

?BGW?PIP address is used to handle BUM traffic between BGWs at different sites, because EVPN Multi-Site architecture always uses ingress replication for this process.

If the BGW is providing external connectivity with VRF-lite next to the EVPN Multi-Site deployment, routing prefixes that are learned from the external Layer 3 devices are advertised inside the VXLAN fabric with the PIP address as the next-hop address.

Please read Cisco white-paper for more detail and additionnal use case for when BGW PIP address is used

Scenario 5: Hosts reachability between Site-1 and Site-2

Use case 1 : L2 Intersite? ??

Between

• Site-1 Host-1?192.168.10.10 VLAN 10? L2-VNI 10010? ?
• Site-2 Host-121? 192.168.10.50 VLAN 10? L2-VNI 10010?

Site-1 : Leaf-101

From Site-1 Leaf-101 Next-hop for 0050.7966.6815 is the loopback100 (Multisite) of the Site-1 BGW

Site-2 : Leaf-201

Use case 2 : L3 Intersite???

Between

• Site-1 Host-1?192.168.10.10 VLAN 10? L2-VNI 10010? ?
• Site-2 Host-122 192.168.30.50 VLAN 30? L2-VNI 30030?

VRF Leaking is done through External connectivity in Site-1. For more detail check lab Part I.

Author:?S. Oumar NDIAYE CCIE #63716 – Cisco Champion 2023.

References?: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/92x/vxlan-92x/configuration/guide/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-92x.html

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-739942.html