Deploy Cisco VXLAN EVPN Multi-Site with EVE-NG & VMware ESXi Part I
Multi-Site Advantages?
Lab Objective
Topology
Multi-Site Lab Architecture
Lab Traffic flow
Part I : VXLAN EVPN in Fabric 1
1.???? Intra-subnet traffic L2 VNI same VRF same Fabric
2.???? Inter-subnet traffic L3 VNI same VRF => Routing via Fabric
3.???? Inter-subnet traffic L3 VNI different VRF => Routing via External Network
4.???? EVPN VXLAN L3 VNI to External Network
Part II : VXLAN EVPN MULTI-SITE
?1.???? Intra-subnet traffic L2 VNI same VRF Multi-Site
2.???? Inter-subnet traffic L3 VNI same VRF Multi-Site => Routing via Fabric
Terminology
Scenario 1 : Underlay
Step 1: Activate all features on Spines and Leafs
Step 2: Configure interfaces into routed ports and enable jumbo frames:
From Cisco: If the fabric only contains Cisco Nexus 9000 and 7000 series switches, then the MTU should be set to 9216.
An MTU of 9216 bytes on each interface on the path between the VTEPs accommodates maximum server MTU + VXLAN overhead. Most datacenter server NICs support up to 9000 bytes. So, no fragmentation is needed for VXLAN traffic.?
Validation
Step 3: Configure VPC on Leaf-102 and Leaf-103
Configure VPC between Leaf-102 – Leaf-103 and Host-VPC
Validation VPC Configuration
Step 4: Configure interface L3 Point-to-Point
/31 network - An OSPF or IS-IS point-to-point numbered network is only between two switch (interfaces), and there is no need for a broadcast or network address. So, a /31 network will suffice for this network. Neighbors on this network establish adjacency and there is no designated router (DR) for the network.
Note : IP Unnumbered for VXLAN underlay is supported starting with Cisco NX-OS Release 7.0(3)I7(2).
IP addressing :
o?? Fabric 1 Uplinks Point-to-Point: 10.30.1.0/24
o?? Fabric 2 Uplinks Point-to-Point: 10.30.2.0/24
Validation
Step 5: Configure underlay IGP and loopback interfaces
The underlay network is responsible for providing connectivity between VTEPs. As a best practice, use a simple IGP (OSPF or IS-IS) for underlay reachability between VTEPs with iBGP for overlay information exchange. In this lab I use OSPF.
P2P links is used once only two switches are directly connected, it can avoid a Designated Router/Backup Designated Router (DR/BDR) election
o??OSPF area ID: 0.0.0.0
o?? OSPF VRF name: UNDERLAY??????
o?? Routing Loopback0 BGP/OSPF/PIM:
§? Fabric 1: ?10.10.10.0/24
§? Fabric 2 : 20.10.10.0/24
Do same for others Leafs
Configure loopback secondary address for VPC VTEP Leaf-102 & Leaf-103
Configure OSPF in Fabric 2
Validation
Fabric 2
Step 6: Configure Underlay? Multicast? PIM-ASM & PIM Anycast RP
Control plane will be configured with BGP EVPN, the VXLAN fabric still requires a technology for Broadcast/Unknown unicast/Multicast (BUM) traffic to be forwarded.
?Two different methods to forward BUM traffic in a VXLAN network :
In this lab I configure Underlay Multicast PIM-ASM + Anycast RP
Each VXLAN segment is mapped to an IP multicast group in the transport IP network (underlay) and each VTEP participating in the VXLAN segment will independently join the multicast group as an IP host through IGMP
o?? Loopback254 Anycast RP Spines :
§? Fabric 1: 10.10.10.254/24
§? Fabric 2: 20.10.10.254/24
o?? Fabric 1 Multicast group : ?
§? Fabric 1: 239.1.0.0/16
§? Fabric 2: 239.2.0.0/16
Anycast RP ensures redundancy and load sharing between the two Rendezvous-Points. To use Anycast RP, multiple spines serving as RPs will share the same IP address (the Anycast RP address). Meanwhile, each RP has its unique IP address added in the RP set for RPs to sync information with respect to sources between all spines which act as RPs.
Fabric 2
Validation
Scenario 2: Configure Overlay MP-BGP EVPN
To overcome the limitations of the flood-and-learn VXLAN as defined in RFC 7348, organizations can use Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network (MP-BGP EVPN) as the control plane for VXLAN.
MP-BGP EVPN is designed to distribute network layer reachability information (NLRI) for the network. A unique feature of EVPN NLRI is that it includes both the Layer-2 and Layer-3 reachability information for end hosts that reside in the EVPN VXLAN overlay network. it advertises both MAC and IP addresses of EVPN VXLAN end hosts.
Configure the BGP overlay for the EVPN address family
Same config for Spine 1002 just put the correct the router -id
Same config for all leafs just put the correct the router-id
Validation
Scenario 3 : VXLAN BGP EVPN L2 and L3 VNI
With the EVPN VXLAN fabric built, we can now run configuration required to perform L2 bridging (L2VNI) and L3 routing (L3VNI) from our end hosts.
Use case 1 : VXLAN with BGP EVPN L2 VNI
Hosts inside the same VLAN should communicate with each other
Step 1?: Create VRF and configure VNI
Fabric 1: For All Leaf Fabric
Beginning with Cisco NX-OS Release 9.2(1), auto derived Route-Target for 4-byte ASN is supported.
No need to do RD and RT configuration under VRF :
Just do this
Step 2: ?Configure Vlans and VXLAN VNI
Step 3: Configure Network Virtualization Endpoint (NVE) interface
In order to reconfigure a TCAM region, use the?hardware access-list tcam region <feature_name> <feature_size>?command in the configuration terminal. Once you have changed the regions to be the intended sizes, you must reload the device.
ARP suppression reduces ARP broadcast traffic by leveraging the BGP EVPN control plane information. ARP suppression is enabled on a per-Layer 2 VNI basis. In this way, for all known endpoints, ARP requests are sent only between the endpoint and the local edge device/VTEP.
NOTE: Starting NXOS 9.2.(x), RD/RT values for L2 VNIs are generated?by default. Can be overwritten with user-defined configuration.
The?rd auto?and?route-target?commands are automatically configured unless you want to use them to override the?import?or?export?options.
?Below commands are?not required :
领英推荐
Step 4: Configure SVI anycast gateway
Not mandatory for L2 Bridging
Step 5: Configure interfaces for hosts
Step 5 : Configure Virtual PC Simulator (VPCS) as hosts
VPCS is a program that runs within Windows or Linux and implements limited network functionality. Specifically it lets you configure the IP address, next hop gateway, and a few other parameters of up to 9 virtual PCs, and it lets you run pings and traceroutes.
Example?: Host-3 => VLAN 30, 192.168.30.10/24? Anycast GW 192.168.30.254
Validation
Check VNI Status in Leaf-101
Check NVE Status on Leaf-102
For VPC? Leaf-102 x Leaf-103 we can see NVE1?: 10.20.20.23 which corresponds to the loopback1 secondary address.
Checks local and remote mac address
Check BGP L2VPN EVPN table for VLAN 10 VXLAN L2 VNI 10010 on Leaf-101
Check BGP L2VPN EVPN? full table on Leaf-104
VXLAN bridging flow for hosts on the same leaf or accross leafs. We have reliability between hosts within VNI.
Use case 2 : VXLAN with BGP EVPN L3 VNI
Inter-subnet traffic L3 VNI same VRF => Routing via Fabric
Routing between host same VRF
Step 1: Configure overlay VRF VLAN and configure VN-Segment
Step 2 : Create server facing SVI and enable distributed anycast-gateway.
Step 3 : Attach VRFs to BGP
Route-map HOST-SVI local routes (static, direct) are not imported into VPNv4 table by default – explicit redistribution is required.
Step 4 : Configure Core-facing SVI for VXLAN routing
This SVI with no ip address, is used to forward packets between VTEPs for inter-subnet communication
When two endpoints, located in different L2VNIs (different subnets), the VTEPs will encapsulate and forward the routed traffic over the L3VNI
ip forward command enables the VTEP to forward the VXLAN de-capsulated packet destined to its router IP to the SUP/CPU.
Step 5: Configure Network Virtualization Endpoint (NVE) interface
Complete configuration overlay for L3VNI to NVE
Check L2VPN BGP EVPN peering
Check VNI / VNE? Status in Leaf-101
RIB and MAC table on VTEP 101 for VRF TENANT-GREEN
Check mac address oh host 192.168.10.20 behind VPC Leaf-102 LEAF-103 learn by LEAF-101
EVPN routes learn againt L3VNI (type 2 and type 5)
Host in VLAN 10 ping Host in VLAN 10 and VLAN 20, we have reliability in VRF TENANT-GREEN but not able to ping host in VRF TENANT-BLUE. Reliability between VRF will be our next use case.
Use case 3 : VXLAN BGP EVPN External Connectivity – Routing between different VRF
External connectivity allows the movement of Layer 2 and Layer 3 traffic between an EVPN VXLAN network and an external network. It also enables the EVPN VXLAN network to exchange routes with the externally connected network. Routes within an EVPN VXLAN network are already shared between all the VTEPs or leaf switches.
Inter-subnet traffic L3 VNI different VRF => Routing via External Network
?And
Leaf-104 is border leaf peering eBGP with Egde router EXT-NET. The VRF leaking will occur in the Edge router wich inject a default route to each VRF and the Border Leaf 104 advertise the SVI to the Edge.
Step 1: Configure Sub-interfaces on border leaf-104 and Edge Router
Step 2: Configure eBGP peering between on border leaf-104 and Edge Router
Aggregate host routes into one BGP update in order to avoid advertise all all host routes
Validation :
Peering between border leaf Leaf-104 and Edge Router EXT-NET
Border Leaf-104 receives the BGP Update message from EXT-NET in VRF TENANT-GREEN and attachs RT 65000:100000
BGP table entry under L2VPN EVPN AFI. RT 65100:100000 Extended community and encapsulation type-8 which means the VXLAN encapsulation
Leaf-101 receives the BGP Update, it imports routing update from Leaf-104 (10.10.10.4) based on RT 65100:100000 which is configured under its vrf context TENANT-GREEN. It creates an L3VNI entry for the network 0.0.0.0/0.
Check ip routing table 0.0.0.0/0 for VRF TENANT-BLUE L3VNI 300000 et TENANT-GREEN with L3VNI 100000 and VXLAN ENCAP.
Data Plane is correct and we have IP connectivity between Hosts in VLAN10 - 192.168.10.10/24?? VLAN20 - 192.168.20.10 ?VRF TENANT-GREEN connected to LEAF-101 and host in VLAN30 192.168.30.10 VRF TENANT-BLUE connected to VTEP VPC LEAF-102 LEAF-103. VRF leaking is done on the Edge Router.
Use case 4 : VXLAN BGP EVPN External Connectivity
Objective is to have IP Connectivity between Hosts in VXLAN EVPN Fabric and External Network.
We keep eBGP peering between border leaf-104 and Edge router? EXT-NET.
A Loopback10 : 192.168.40.1/24 is configured on the Edge and advertised to the border leaf Leaf-104; All Hosts in the Fabric should have IP connectivity with this loopback.
Validation
Route advertised by EXT-NET is received from the border Leaf-104
Border Leaf-104 has received BGP Update from EXT-NET AS 65300 and advertises it to the Spines with RT 65100:100000 in VRF TENANT-GREEN. Same for VRF TENANT-BLUE with RT 65000:300000
Leaf-101 receives the BGP Update, it import routing update from Leaf-104 (10.10.10.4) based on RT 65100:100000 which is configured under its vrf context TENANT-GREEN. It creates an L3VNI entry for the network 192.168.40.0/24 VXLAN ENCAP.
Check ip routing table 192.168.40.0/24 for VRF TENANT-BLUE L3VNI 300000 et TENANT-GREEN with L3VNI 100000 and VXLAN ENCAP
IP connectivity between hosts in VXLAN fabric and External Network.
End of the first part. The next one is the implementation of Multi-Site
Author:?S. Oumar NDIAYE CCIE #63716 – Cisco Champion 2023.
https://nwktimes.blogspot.com/search?q=vxlan Very nice blog of Toni Pasanen
This lab with NXOS 9000v uses a lot of CPU and memory, to setup it you would need the necessary resources
System Engineer Intern @ Cisco | CCNP DC | DCACI
3 个月Thanks for the awesome material!
ACI and Network architect at Societe Générale ( Looking for new position as a network engineer SP/DC or enterprise)
8 个月Great write up Oumar
Network InfraStructure Specialist
11 个月If we want to setup a simple VXLAN (without evpn) , & if we don't use MultiCast : How VTEPs Detect EachOther's IP Address ? How does VTEP-1 detect IP address of VTEP-2? Connection between VTEPs are L3-Based Tunnel , so VTEPs can NOT use legacy Layer2-ARP-Broadcast ! In "Ingress Replication" Solution , When VTEP-1 Wants To Send An ARP-Packet , Does Works As Legacy Layer2-SW & BroadCast ARP To All Switches (including VTEP & none-including VTEP) ? If Yes , The Only Thing Preventing MAC-Overlap Between Other VLANs Is VNI In VXLAN-Packet. Would you please leave your comment to clarify this topic ?
Network InfraStructure Specialist
11 个月Excellent ! The Article was about VXLAN EVPN. Is there any Simple VXLAN (without EVPN) Sample Scenario ? I Know The Only Difference Between VXLAN & VXLAN EVPAN Is Engaging BGP In Control Plane. I Guess You Will Say This Scenario Covers Simple VXLAN Too, So Would You Please Specify In Which Section Of Article Simple VXLAN Ends & VXLAN-EVPN Starts ? Thank You In Advance...