Installation Notes on Deploying SD-WAN On-Premises with Enterprise Certificates

After deploying SD-WAN a couple of times and running into a few issues I decided to write this post. Configuring SD-WAN isn't terrible (most of the time...), especially when you use the Cisco hosted command and control components...at that point you're just working to onboard your edges and build out your templates. However, when you throw in the on-premise and enterprise certificate pieces it can become a bit daunting. My hope here is to give other engineers configuring this solution with these requirements a little bit of a leg up on some of the finer points and hopefully help them avoid some of the same pitfalls I've come across.

Also as a caveat, this is not meant to be a comprehensive article on the foundational components of SD-WAN; there is a bit of assumed knowledge you should have to really leverage the information here. At the end of this post I've provided a few resource links that I highly recommend for getting up-to-speed if you're not already. Additionally, these notes are based on SD-WAN v20.7.1 and IOS-XE v17.7.1a.....your mileage may vary on other versions. Let's get started!

Step 1a) Plan your deployment!

Whether your building a bird house or rolling out a next-generation software defined WAN technology, proper planning is a key component.

!!!!! Some items to think about before implementation.

?	How many sites will you have?? 
    --> What will the site-id hierarchy look like?

?	What type and how many transports will each of the sites use?? 
    --> Internet, MPLS, LTE, etc
    --> Who are the POCs? 
    --> Do you have all information to configure them?

?	How many service VPNs are needed? 
    --> Are all of these absolutely necessary?
    --> What numbering scheme for VPNs will be used?

?	What types of devices will you be using (Edges and Virtual environment)?
    --> Edge Hardware and Server models will be in use?
    --> Are these devices listed in Cisco's Smartnet site?

?	Where will your On-Prem SD-WAN virtual components live?
    --> (vManage, vBond, vSmart) be located?   
    --> What firewalls or other access control are in place?

?	What server will be the CA for the deployment? 
    --> Make sure you are able to acquire the Root CA certificate.   
 
?   What IP addresses will you use for the servers?

?   What DNS and NTP resources will you use?

?   What Organization Name, Site IDs, and System-IPs will you use?

        

Below are basic guidelines for the server hardware requirements for hosting the vManage, vBond, and vSmart on premises. These parameters are provided based on Cisco's Community Post on "SD-WAN Controller Setup Guide (On-Prem, Non Cloud-Managed)" the source reference is listed at the end of this post. Obviously, you’ll want to double-check these based on the current version and scale of your deployment. Note:?SSDs are strongly recommended / required for storage especially in production environments. Multiple physical servers are recommended for resiliency.

!! Basic guidelines for the server hardware requirements for hosting the vManage, !! vBond, and vSmart on premises. These parameters are provided based on Cisco's !! Community Post on "SD-WAN Controller Setup Guide (On-Prem, Non Cloud-Managed)" 

vManage
? 2 CPU
? 32GB Memory
? Disk 1: 30GB
? Disk 2: 100GB (≥ 500GB in production, based on scale)

vSmart
? 2 CPU
? 4 GB Memory
? Disk 1: 10.5GB (Approx. image size)

vBond
? 4 CPU
? 2GB Memory
? Disk 1: 10.5GB (Approx. image size)

VMWare ESXI v6.0+ instance for managing VMs and VM networks:
? One virtual network and switch for VM management and
? One virtual network and switch for SD-WAN
? Datastore(s) configured with at minimum 500GB
? Workstation with network access to servers and controllers.?
        

Step 1b) Plan your deployment software versions.

Everything should be vetted against the solution compatibility matrix.

Visit Cisco’s SD-WAN compatibility matrix website and confirm your versions: (https://www.cisco.com/c/en/us/td/docs/routers/sdwan/release/notes/compatibility-and-server-recommendations.html)?You need to make sure that your software versions are compatible between the command-and-control components (vManage, vBond, and vSmart) and the device version that you plan to use as your Edges. I’ve noticed that some features may work with versions that are not listed but other features may require a specific version; you’ve been warned.?You’ll notice that the software version numbers differ between the SD-WAN (virtual) components and Physical Edge components. SD-WAN versions are 19.X or 20.X….cEdges are IOS-XE version 16.X or 17.X. Also, you’ll want to make sure that you’re deploying what Cisco considers an “extended maintenance” release for the Edges as well as controllers. Standard Maintenance releases have much shorter support lifetimes whereas Extended Maintenance releases have longer support with more scheduled rebuilds / bug fixes, it is recommended that customers stay on Extended Maintenance releases wherever possible. Check these sites for recommendations –

?Step 2) Download the OVFs files from software.cisco.com.

( https://software.cisco.com/download/home/286320995/type )

You’ll download three (3) OVFs from Cisco.?One (1) for vManage, one (1) for vSmart, and one (1) for vBond. The vBond image is actually the same image used for the virtual Edge deployment.

·???????viptela-vmanage-20.7.1.1-genericx86-64.ova (vManage)

·???????viptela-smart-20.3.4-genericx86-64.ova (vSmart)

·???????viptela-edge-20.3.4-genericx86-64.ova (vBond)

Again, pay close attention to both compatibility constraints as well as trying to use an Extended Maintenance release when selecting a version to use. Case in point, I actually needed to run the 20.7 release in my setup because the customer was using VMware ESXi 7.0; it was the only release at the time that officially supported it.

Step 3) Deploy your OVFs to the virtual environment.

The actual deployment of the OVFs takes almost no time at all.?When you deploy them make sure to not power on the vManage immediately; you’ll need to add an additional hard drive (500GB or so depending on the scale of the deployment) to be used for the database. This designation is selected upon first boot.?You also indicate the vManage persona.?This selection will determine if the vManage will function as a centralized (Compute+Data) or distributed/clustered deployment. A vManage cluster creation is beyond the scope of this document, however, a reference is provided at the end of this document for further configuration if necessary.??You’ll also need to change the default password upon the initial logon to all VMs. The default username/password is (U: admin / P: admin). Initial configuration can be completed through ESXi or vCenter VM console server.?Once you have a reachable IP address on the device you can SSH into the device and copy/paste your configuration scripts in.

Step 4) Configure the basic configuration for the vManage, vSmarts, and vBond.

NOTE: (( bracketed )) items should be identified/changed prior to configuration deployment.

!!!!!!!!!!! vManage Base Configuration. 

config t

system
?host-name? ? ?       (( vManage-1))
?system-ip? ? ?       ((1.1.1.1))
?site-id? ? ?        ?((100101))
?sp-organization-name? (("SDWAN - 04192022"))
?organization-name?   (("SDWAN - 04192022"))
?vbond? ? ? ?         ((10.10.10.101))

vpn 0
?interface eth0
?ip address ((10.10.10.100/24))
?no ipv6 dhcp-client
?tunnel-interface
?no allow-service dhcp
?allow-service dns
?allow-service icmp
?allow-service sshd
?allow-service netconf
?no allow-service ntp
?no allow-service stun
?allow-service https
?no shutdown

!!! Next hop for tunnel interface?
ip route 0.0.0.0/0? ((10.10.10.1))

commit
        

!!! vManage Notes:?Once the initial configuration is setup on vManage (it may take a few minutes for internal setup processes to complete) you’ll be able to login to the GUI.?If you can’t reach it, make sure the vManage Eth0 interface is in the proper virtual port-group and that you have IP connectivity.

!!!!!!!!!!! vSmart Base Configuration. 

config t

system
?host-name? ? ? ? ? ? ((vSmart-1))
?system-ip? ? ? ? ? ? ((1.1.1.4))
?site-id? ? ? ? ? ? ? ((100101))
?organization-name? ? (("SDWAN - 04192022"))
?vbond                ((10.10.10.101))
?
vpn 0
?interface eth0
?ip address ((10.10.10.104/24))
?no ipv6 dhcp-client
?tunnel-interface
? no allow-service dhcp
? allow-service dns
? allow-service icmp
? allow-service sshd
? allow-service netconf
? no allow-service ntp
? no allow-service stun
? allow-service https
? no shutdown

!!! Next hop for tunnel interface?
ip route 0.0.0.0/0? ((10.10.10.1))

commit
        

!!! vSmart Notes:?Once the initial configuration is setup on vSmart it should become available almost immediately.?If you can’t reach it, make sure the vSmart Eth0 interface is in the proper virtual port-group and that you have IP connectivity from your workstation.

!!!!!!!!!!! vBond Base Configuration.


config t


system
?host-name? ? ? ? ? ? ? ((vBond-1))
?system-ip? ? ? ? ? ? ? ((1.1.1.6))
?site-id? ? ? ? ? ? ? ? ((100101))
?organization-name? ? ? (("SDWAN - 04192022"))
?vbond? ? ? ? ? ? ? ? ? ((10.10.10.101)) local


vpn 0
?interface ge0/0
?ip address ((10.10.10.101/24))
?no ipv6 dhcp-client
?no shutdown
?
!!! Next hop for tunnel interface?
ip route 0.0.0.0/0? ((10.10.10.1))

commit
?        

!!! vBond Notes:?Once the initial configuration is setup on vBond it should become available immediately.?If you can’t reach it, make sure the vBond ge0/0 interface is in the proper virtual port-group and that you have IP connectivity. Please note that this interface is different (in name) from the other VMs that you’ll setup. If you’re trying to configure eth0, this will NOT work. Also note that there is no tunnel interface associated to this device initially. Once the vBond is up and online you can create the tunnel interface for a more secure control-plane connection. Lastly, note the “local” parameter used after the vbond <ip> identification. This keyword is critical and what differentiates the VM as a vBond versus an virtual Edge VM.

Step 5a) Initial vManage setup – Add vBonds and vSmarts, then set Organization Name.

Before you do this, you’ll need to get your Enterprise Server RootCA certificate and have that on-hand. This will be something you obtain from the administrator of your Enterprise PKI infrastructure and is typically just a small text file. After installation to the vManage this file can be located in /usr/share/viptela/ on the vManage….accessed through CLI after the “vshell” command.

?Once all server images are deployed, the first thing you’ll want to do is confirm IP connectivity. Login to vManage, vBonds, and vSmarts and make sure they are up and running.?Next you'll want to add you vBonds and vSmarts to your vManage. This can be accomplished in Configuration > Devices – Select Controllers. then +Add Controller pull-down menu and choose your respective controller (vbond or vsmart); add all controllers. Once this is completed go and define the vManage Organization (Org) name. The organization name is a critical step in the setup and has direct ramifications for successfully generating certificate signing requests (CSRs) for both the vManage and other SD-WAN servers.

Before you set your Org name in vManage you should know that the Org name will need to be globally unique within Cisco’s database of organization names even in an On-Prem installation.?This said, there’s nothing to stop you from setting something that’s NOT globally unique when you change the setting in vManage. You’ll only find this out later when you look to define your “controller-profile” on Cisco's PNP site and find that that "already exists" Eventually, this will cause issues with getting a “provisioning file” and zero-touch-provisioning if you plan to use it. If you have no plans to use these features, you may be able to get by but best practice dictates that you should aim to make your defined organization name globally unique. Note: Once the control connections are up and running, the organization name bar is not editable; the option to edit the Org name disappears.?If you need to change this after the fact you will likely need to delete any associated vBonds and vSmarts from vManage prior to the setting being editable again.

I’ve also confirmed from Cisco that there is no way to official way to confirm if an Org name is already in use, however, you are able to use the same Smartnet licensing portal and define your On-Prem vBond “controller” profile before you define your Org. During this process you are asked to specify your Org name and the database is checked for duplication. If the org name is already there you will get an error message that states, “The name already exists”.?Cisco’s Smartnet licensing portal is located at – https://software.cisco.com/#pnp-controllerProfiles. Under controller profiles, select +Add Profile – For controller type select VBOND from the pull-down menu and then Next. Fill in the information. If you're able to click Next and successfully create the controller profile you should be in good shape with your Org name.

Once you’ve confirmed that you’re Org name is unique, navigate within vManage to Administration > Settings from the vManage main dashboard. The first setting at the top of the page will be Organization Name. The name is case-sensitive and must match exactly wherever it is used, including any characters or spaces. Once you've added the Org name you should also add the vBond (IP or FQDN) information just below Org name in the same Settings area.

Step 5b) Initial vManage setup – Change the deployment to Enterprise Certificates and add the Root CA.

Navigate within vManage to Administration > Settings from the vManage main dashboard. Look for the “Controller Certificate Authentication” setting…click Edit and choose the “Enterprise Root Certificate” radio button. Click Save.?

At this point you’ll be prompted to provide some information. You’ll need to input the Enterprise RootCA certificate. You’ll also want to check the “Set CSR Properties” box at the bottom.

When you check the “Set CSR Properties” box you’ll get the following inputs

!!! “Set CSR Properties” Notes:?

Enter the information here carefully. The info may need to be duplicated by the individual that will be when submitting the CSRs for CA signing. The domain name you choose is important and seemingly limited….let me explain. The domain name you use will be used automatically in the common-name (CN) field of the certificate signing request (CSR) you’ll generate shortly for the vManage, vSmart, and vBond. RFC 5280 defines the maximum length of the CN to be 64 characters (including periods).?When you generate your CSRs for your vManage, vBond, and vSmart the common-name parameter is automatically generated by vManage.?The default format that it uses is a generic hostname with your domain name appended and looks something like this - vmanage-4aa5dacd-cd30-422d-9ffa-66656b61c1bf-1.<domain-name>. This fact essentially uses up 46 characters of the CN field before you even add the domain name and only leaves 18 characters (including periods) for the domain name.?If you attempt to use a domain name that exceeds this your CSR generation will fail.

An additional note to add here is about the “Secondary Organization Name”. Note that this optional feature allows you to configure a secondary organizational unit when configuring the certificates. If specified, this setting is applied to all controllers and edge devices.

?Step 6) Generate CSRs for vManage, vBond, and vSmarts

At this point you’ll generate the CSRs for your management and control infrastructure. This can be accomplished on the Configuration > Certificates page, select Controllers at the top of the screen then right-click the “…” next to each server to generate a CSR. Note: Make sure that you ONLY generate a single CSR for each server. If you click Generate CSR more that one time you will invalidate the previously created CSR. If your CSR generation fails, check your domain-name length (see note in the previous step).?After you have your CSRs generated you can confirm the properties by using your preferred X.509 decode utility.

Step 7) Install your signed certificates.

This can be accomplished on the Configuration > Certificates page. Click Install certificate in the top right and add the certificate. It will be automatically matched to the correct server and installed if all settings match. Complete this task for all signed certificates.

Step 8) Prepare to onboard Edges. (Add edge list to vManage and install RootCA to Edge devices)

In order to add or on-board devices you’ll need to perform two tasks. First, you’ll need to make sure that the vManage has a list of the Edges, Serial numbers, and Certificates that it’ll expect to see connections from. Secondly, you’ll need to make sure you copy and install the RootCA you’re using to the individual Edges.

For the list that you need to provide to the vManage you’ve got two options:

1) Provision a vBond “controller-profile” in Cisco’s Network Plug and Play ( https://software.cisco.com/)?website, assign your devices to it, then download the associated Provisioning file that’s generated; this file is then uploaded to vManage. Note, this is a required step for Zero Touch Provisioning (ZTP) (which I will not be covering here) you’ll also need your RootCA certificate as well as your unique organization name.

2) Alternatively, you can generate a comma separated variable text file and upload that to program your list of edges into the vManage. The format of this text file looks like so:

!!!! Format for the CSV file for manual edge definition on vManage

C8300-2N2S-6T-FLM293123AB, ,040945701489475869A1,FLM293123AA
C8300-2N2S-6T-FLM293123AC, ,040945701489475868A1,FLM293123AC
C8300-2N2S-6T-FLM293123AD, ,040945701489475867A1,FLM293123AD
        

Make sure you check the box to push this list to the controllers as you upload it. Otherwise, you’ll need to do that after the fact. In Configuration > Certificates you’ll note that “Send to Controllers” is highlighted in Red.

To copy the RootCA certificate to the edges you can either use your favorite file transfer application or USB. Then use the following command to install the certificate (Cisco Edge):

!!! IOS-XE cEdge CLI command uninstalls existing root certificates
request platform software sdwan root-cert-chain uninstall


!!! IOS-XE cEdge CLI command installs the uploaded enterprise root ca certificate
!!! The enterprise CA is named "ca-cert.cer" and was copied to bootflash.
request platform software sdwan root-cert-chain install bootflash:ca-cert.cer        

Step 9) Bootstrap your Edges. We’re almost there!!!!

Perform a basic configuration on your edge devices otherwise known as bootstrapping them.

Remember the goal here is to provide the Edge with a few pieces of critical information in order for it to find the vBond, authenticate, then join the SD-WAN overlay.?Those pieces include:

? Interface and IP address that will be used to reach the vBond
? Default route for the interface
? The vBond IP or FQDN
? A DNS server
? A NTP server
? SD-WAN “Org” name (Make sure this is EXACTLY as input to vManage)
? SD-WAN system-ip
? SD-WAN site-id
? SD-WAN overlay-id
? SD-WAN tunnel interface and physical interface association.
        

Below is a basic script for cEdge bootstrap that has worked fine for most of my purposes. NOTE: (( bracketed )) items should be identified/changed prior to configuration deployment.

!!! Enter “config” mode in SD-WAN XE cod
config-t
?
system
?system-ip? ? ? ? ? ? ((1.1.1.10))
?overlay-id? ? ? ? ? ?((1))? ? ? ? ??
?site-id? ? ? ? ? ? ? ((100501))? ? ? ? ? ?
?port-offset? ? ? ? ? 0
?control-session-pps? 300
?admin-tech-on-failure
?sp-organization-name (("DEFAULT – 1234567")) ?
?organization-name? ? (("DEFAULT – 1234567”))
?port-hop
?no track-transport
?no track-default-gateway
?console-baud-rate? ? 9600
?vbond ((vbond-1234567.viptela.net)) port 12346


!!! Set DNS and default route info.
ip name-server ((10.10.10.51))
ip name-server ((10.10.10.52))
 
!!! This route should be reachable from your active inteface.
ip route 0.0.0.0 0.0.0.0 ((10.10.10.1))


!!!!! Configure your initial VPN0 interfaces


interface ((TenGigabitEthernet0/0/0))
?no shutdown
?arp timeout 1200
?ip address ((10.10.10.5 255.255.255.0))
?ip redirects
?ip mtu? ?1500
?mtu 1500
?negotiation auto


interface Tunnel1
?no shutdown
?no ipv6 redirects
?no ip redirects
?ip unnumbered ((TenGigabitEthernet0/0/0))
?ipv6 unnumbered ((TenGigabitEthernet0/0/0))
?tunnel source ((TenGigabitEthernet0/0/0))
?tunnel mode sdwan
exit


!!!!! Configure your “transport” side connectivity

sdwan
?interface ((TenGigabitEthernet0/0/0))
? tunnel-interface
? encapsulation ipsec weight 1
? no border
? color biz-internet
? no last-resort-circuit
? no low-bandwidth-link
? no vbond-as-stun-server
? vmanage-connection-preference 5
? port-hop
? carrier? ? ? ? ? ? ? ? ? ? ? default
? nat-refresh-interval? ? ? ? ?5
? hello-interval? ? ? ? ? ? ? ?1000
? hello-tolerance? ? ? ? ? ? ? 12
? no allow-service all
? no allow-service bgp
? allow-service dhcp
? allow-service dns
? allow-service icmp
? no allow-service sshd
? no allow-service netconf
? no allow-service ntp
? no allow-service ospf
? no allow-service stun
?exit


!!!!!! Set OMP Parameters (default parameters)

omp
?no shutdown
?send-path-limit 4
?ecmp-limit? ? ? 4
?graceful-restart
?timers
? holdtime? ? ? ? ? ? ? 60
? advertisement-interval 1
? graceful-restart-timer 43200
? eor-timer? ? ? ? ? ? ?300
? exit


!!!!!! Save Changes.

Commit
        

Once you apply this configuration and connect the device it should try to begin reaching out to vBond.

Some troubleshooting commands you should be familiar on the cEdge, if you are not already, are “show sdwan control connections” and “show sdwan control connection-history”. These will give you a very good idea of any sort of issues that you may be encountering if the Edges are not successfully connecting.?One of the most common issues you may see as an output is the DTLS Connection Failure (DCONFAIL) error. This typically indicates that there is some kind of issue with the device reaching out and connecting and packets are either being filtered or dropped.

?

Step 10) Final Steps - Your Edges are connected!!!!

At this point hopefully you have some SD-WAN Edges connecting to your vManage for further configuration. Next steps you’ll want to look at are setting up licensing, creating feature and device templates then associating them to your devices.??Nice work!

References:

?Cisco Community Post: SD-WAN Controller Setup Guide (On-Prem, Non Cloud-Managed) https://community.cisco.com/t5/networking-documents/sd-wan-controller-setup-guide-on-prem-non-cloud-managed/ta-p/3921360

Cisco SD-WAN Controller Certificates and Authorized Serial Number File Prescriptive Deployment Guide. https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-controller-cert-deploy-guide.html

?Cisco SD-WAN Getting Started Guide (Chapter: Certificate Management): https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/manage-certificates.html

Troubleshoot SD-WAN Control Connections: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214509-troubleshoot-control-connections.html

Cisco SD-WAN vManage Cluster Creation and Troubleshooting White Paper: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/white-paper-c11-741440.html

Cisco SD-WAN: On-Prem Controller Deployment on ESXi with Manual Certificates. (YouTube): https://www.youtube.com/watch?v=CQruni5x8Vk

!!!! Version History

May 2nd, 2022 - Version 1.0 - Initial Release