Cisco APIC M1/M2/M3/L1/L2/L3 to M4/L4 Cluster Migration

Welcome to this extensive exploration of APIC M1/M2/M3/L1/L2/L3 to M4/L4 Cluster Migration. This document has been carefully prepared for network engineers, system administrators, and IT professionals who are tasked with managing or migrating Cisco Application Policy Infrastructure Controllers (APIC). Whether you're embarking on this journey for the first time or looking to refine your existing process, this guide is designed to guide you through every critical step of the migration, from pre-migration preparation to post-migration validation.

My goal is to provide you with the knowledge, strategies, and best practices to ensure a smooth, efficient, and successful migration. By the end of this guide, you will have learned:

• The nuances of upgrading your ACI Fabric: Understand the why and how of upgrading to the latest (and most stable) code versions to ensure optimal performance and security of your infrastructure.
• Detailed steps for APIC server installation and initialization: We break down the initial setup process, including physical server connections, basic configuration settings, and accessing your APIC's GUI or CLI for management.
• BIOS and CIMC firmware upgrade procedures: Critical insights into upgrading your server's BIOS and CIMC firmware to ensure your hardware is running smoothly and is compatible with the latest software requirements.
• Navigating the OS Upgrade/Downgrade Path: Master the process of upgrading or downgrading your APIC OS and align your infrastructure with the required versions for a seamless migration.
• Troubleshooting Common Challenges: Arm yourself with the knowledge to anticipate, identify, and resolve potential problems.

Get started with the following content available from Cisco to help you plan your APIC OS upgrade/downgrade

Why You Shouldn't Fear Upgrading Your ACI Fabric-the Handbook - BRKDCN-2910

Cisco ACI Upgrade Checklist

You can find everything about APIC in one place. Please visit the following link

Cisco Application Policy Infrastructure Controller (APIC)

If you are planning to migrate the APIC M2 to M4/L4, the first step in this migration is to upgrade the existing ACI Fabric to the latest (stable) code version and to upgrade/downgrade new APICs.

Let's Suppose

• The existing APIC hardware is an APIC M2 Cluster; The existing ACI Fabric is running APIC OS is 5.2(8)
• The new APIC hardware is the APIC M4/L4 Cluster ; the new APIC OS will be 5.3(1d)
• Leaf/Spine; Cisco NX-OS 15.3(1)
• 4.3.2.230207 CIMC HUU ISO (recommended*) for UCS C220/C240 M5 (APIC-L3/M3) and UCS C225 M6 (APIC-L4/M4)

NOTE: The new APIC M4/L4 will come with the 6.0(X) version by default, in my case it was the 6.0(2h) version.

APIC Server Installation & Basic Initialization (CIMC):

You've already received your new APIC M3/M4/L4, the next step is to rack-mount the C-Series Server (APIC) it and complete the basic initialization,

Which includes but is not limited to

• Set up username/password
• Set up IP address
• Enabling GUI/KVM etc

Before proceeding to Step 1, let's understand the physical connectivity of the APIC M4/L4. Refer to the following figure

Fabric Links: Connect these links with Leaf or IPN (L3 APIC Etc)

APIC OOB: To access the APIC GUI/Cli, connect to this link for APIC management.

CIMC Platform Management: Connect to this link for CIMC management, to access the CIMC GUI/Cli, you can also do reverse Telnet APIC (connect host)

Step 1: Access the CIMC ; Complete the basic initialization

The Cisco APIC M3/L3/M4/L4 Server operates on a Cisco Integrated Management Controller (CIMC) platform. You can make an initial connection to the CIMC platform using one of these methods:

• Use a KVM cable (Cisco PID N20-BKVM) to connect a keyboard and monitor to the KVM connector on the front panel of the server
• Connect a USB keyboard and VGA monitor to the corresponding connectors on the rear panel of the server

Open the Cisco IMC Configuration Utility:

• Press and hold the front panel power button for four seconds to boot the server
• During bootup, press F8 when prompted to open the Cisco IMC Configuration Utility

NOTE: The first time that you enter the Cisco IMC Configuration Utility, you are prompted to change the default password. The default password is password. The Strong Password feature is enabled.

Step 2: Set the NIC mode to choose which ports to use to access Cisco IMC for server management. Cisco IMC for APIC appliance only supports dedicated mode:

Dedicated - The dedicated management port is used to access Cisco IMC. You must select the None NIC redundancy setting in the following step.

Step 3: Set the NIC redundancy to none, which is the only supported option for dedicated NIC mode:

None - The Ethernet ports operate independently and do not fail over if there is a problem. This setting can be used only with the Dedicated NIC mode.

Step 4: Choose whether to enable DHCP for dynamic network settings, or to enter static network settings

NOTE: During Staging, I'll use Static network settings.

• Add the Cisco IMC IP address, Gateway and Preferred DNS server address

After entering the details, press F10 to save, then press ESC to exit.

Step 5: Enable the SOL

Enable SOL, through GUI

Enable SOL, through CLI

Server# scope sol
Server /sol # set enabled yes
Server /sol *# set baud-rate 115200
Server /sol *# commit
Server /sol *#
Server /sol # show
Enabled Baud Rate(bps)  
------- --------------- 
yes     115200    
Server /sol #        

NOTE: After basic configuration, You'll be able to access the CIMC through GUI/CLI.

Upgrade BIOS and CIMC Firmware from 4.2(2g) to 4.3(x):

Running Version: 4.2(2g)

Target Version: 4.3(2.230207)

See; Upgrading the CIM Software

Step 1: Download the target CIMC Software (ucs-c225m6-huu-4.3.2.230207.iso)

Download the target CIMC 4.3(x) version

Step 2: Map the ISO image (ucs-c225m6-huu-4.3.2.230207.iso)

Step 3: Power cycle the CIMC server and Click Confirm

Step 4: Press F6 to enter the boot menu

You will be asked to enter the password, if you don't change the default BIOS password, then just use the default 'password'.

Step 5: Please select the boot device? Choose the ISO image (ucs-c225m6-huu-4.3.2.230207.iso), mapped earlier.

Step 6: Accept the License

Step 7: Click Update & Activate

Step 8: Wait for Update to be Complete

Step 9: Verify update went successful

APIC OS Upgrade/Downgrade:

According to my situation, I'm going to upgrade ACI Fabric (Prod) i.e APIC M2 from version 5.2(8) to 5.3(1d), but also downgrade the new APIC i.e M4/L4 from version 6.0(2h) to 5.3(1d).

Basic Initialization of APIC M4/L4 [APIC OS 6.0(2h)]

Let's say you are doing a staging of APIC M4/L4. You only have access to APIC and CIMC through their respective Mgmt IPs

Step 1: Access the CIMC KVM and you'll be able to see APIC is in it's initialization state, you'll need to configure the following (APIC Initialization) before accessing the APIC via GUI/CLI

Step 2: Add password, OOB mgmt ip and gateway address Etc

admin user configuration ...
  Enter the password for admin [None]:
  Reenter the password for admin [None]:

Out-of-band management configuration:
 Enter the IP Address [192.168.0.1/24]:
  Enter the IP Address of default gateway [192.168.0.254]:

Would you like to edit the configuration? (y/n) [n]:        

Step 3: Access the APIC via GUI (https://192.168.0.1)

Step 4: How are your APIC controller connected to your ACI Fabric?

Choose Directly attached to leaf switch, Click Next

Step 5: Enter the following details

Cluster size, choose 1, if You'll choose 3, You'll be asked to provide Serial # of other APICs (that you want to include in the APIC cluster)

Step 6: Enter the CIMC password and click Validate

After validation successful, click Save and then click Next

Step 7: Verify details and click Deploy

Step 7: Go to KVM, place cursor on Console, then select SOL (from drop-down list), now you can access APIC CLI through KVM

NOTE: APIC OS 5.3(x) or earlier, you can get access to SOL (via CIMC KVM) without first initializing APIC (via GUI), but specifically for 6.0(X), you must first initialize the APIC via GUI, then you will get APIC CLI access via KVM/SOL/Telnet/SSH etc

After successful initialization (APIC), you will see that SOL is accessible via CIMC KVM.

Existing ACI Fabric Upgrade:

Simply follow; Cisco APIC Installation and APIC Upgrade and Downgrade Guide

NOTE: don't forget to follow Cisco ACI Upgrade Checklist

APIC M2 to M4/L4 Cluster Migration:

Simply follow; Cisco APIC M1/M2/M3/L1/L2/L3 to M4/L4 Cluster Migration, Release 5.3(1d)

Simply follow;. Cisco APIC M1/M2/M3/L1/L2/L3 to M4/L4 Cluster Migration, Release 6.0(2)

Challenges/Issues/Blockers:

Migrating from APIC M2 to M4/L4 clusters is a complex process that, while meticulously planned, can encounter unforeseen challenges. Some of these challenges include

• Accidental Deletion of Boot Images: When updating the BIOS and CIMC firmware on the APIC M4, there's a risk of accidentally deleting critical boot images. This error can halt the migration process and require a reinstallation of the APIC OS required images
• Prerequisites for Installation: It is critical to ensure that all prerequisites are met for the APIC OS installation on the C-Series server. Failure to meet any prerequisite can delay or interrupt the installation process, For more information have a look Cisco ACI Upgrade Checklist
• Compatibility issues: It is critical to ensure that all hardware and software components are fully compatible with new APIC releases. Incompatibilities can result in system instability or performance degradation, For more information have a look Cisco APIC Release Notes
• Firmware and Software Synchronization: Keeping the firmware versions of physical components in sync with the software versions of the APIC OS and ACI fabric can be challenging. Mismatches can result in unexpected behavior or loss of functionality, For more information have a look Cisco ACI Upgrade Matrix
• Data Backup and Loss: The risk of data loss during migration is a significant concern. It is imperative that comprehensive backups are in place and verified before the migration process begins.
• Training and knowledge gaps: Familiarizing the technical team with new hardware and software features can be a challenge. Ensuring that the team is adequately trained and prepared for the migration is critical to a smooth transition.
• Hardware failures: Hardware issues, such as failures in the new APIC M4/L4 servers, can occur unexpectedly. Quick access to Cisco TAC or warranty services is essential to resolve these issues quickly.

By recognizing and preparing for these challenges, you can increase the resiliency and efficiency of the migration process. Proactive planning, thorough testing, and contingency strategies are key to overcoming these obstacles.

I personally encountered the problem of accidentally deleting the boot image during the upgrade process of updating the BIOS & CIMC firmware, here is how to install the APIC OS on C-Series server for the first time.

APIC OS Installation on the C-Series Server:

Before starting the APIC OS installation on the C-Series server, you must first make sure that the http server is up and running because Cisco recommends that you map the ISO image using the http server for APIC OS 5.X or later.

NOTE: Installing IIS services on Windows Server was not difficult for me, although I am not a Windows administrator/expert.

http Server; The Cisco APIC ISO must be available on an HTTP server reachable from the APIC-M4/L4 Server CIMC management interface and the OOB management interface.

Obtain the relevant Cisco APIC .iso image from Cisco.com and copy the .iso image to the HTTP server.

How to setup Windows Server as http Server ?

NOTE: After you successfully install IIS services on Windows server, you need to make sure that you've added ISO in the file extension (http server), you also need to copy the APIC OS and NX-OS images into the http server folder as shown in the figure below.

Step 1: Go to Server Manager

Step 2: Click, add roles and features, Click Next

Step 3: Select the Server & click Next, then select Web Server (IIS) i.e Web Server, FTP Server, Management Tools then click Next, click Next again and then click Install

NOTE: I already installed the IIS so You can see it's showing Installed

After a successful installation, you'll see IIS in Server Manager as follows

Step 4: Click Manageability, then click Go to IIS, then click IIS Manager

Step 5: You will be able to see the MIME types in the http feature section, click it and then click Open Feature, click Add and .iso file type, Stop/Start the service

Step 6: Go to (C:) drive, go to inetpub (folder), then go to wwwroot (folder), copy the ISO images that you wanted to upload to the APIC

i.e aci-apic-dk9.5.3.1d.iso, aci-apic-dk9.6.0.2h.iso, aci-n9000-dk9.15.3.1d.bin

NOTE: We have fulfilled the prerequisite in order to start the upgrade / downgrade of the APIC

How to Install APIC OS?

I follow the Installing the Cisco APIC Software Using CIMC Virtual Media

Procedure:

Step?1: Access the vKVM console:

1. Open the Cisco Integrated Management Controller (CIMC) GUI for the controller.
2. For an APIC-M1, M2, M3, L1, L2, or L3 server, from the CIMC GUI, choose Server > Summary > Launch KVM, then select either Java based KVM or HTML based KVM to access the KVM console.We recommend using the Java based KVM option whenever possible, because it is a more reliable option for larger-sized files.
3. For an APIC-M4/L4 server, from the CIMC GUI, choose Server > Summary > Launch vKVM to access the HTTP-based vKVM console.

Step?2: Access the Serial over LAN (SOL) console:

From a terminal window, log in to the CIMC console:

# ssh admin@cimc_ip
Where cimc_ip is the CIMC IP address. For example:
# ssh [email protected]
[email protected]'s password: 
CIMC_APIC-01#        

Change the scope to virtual media

CIMC_APIC-01# scope vmedia
CIMC_APIC-01 /vmedia #        

Map the .iso image to the HTTP server:

CIMC_APIC-01 /vmedia # map-www volume_name https://http_server_ip_and_path iso_file_name        

Where:

• volume_name is the name of the volume.
• http_server_ip_and_path is the IP address of the HTTP server and the path to the .iso file location.
• iso_filename is the name of the .iso file.

NOTE: copy/paste following string and change your values i.e file name, IIS ip and iso file name.

map-www APIC6.0_iso https://192.168.0.99/ aci-apic-dk9.6.0.2h.iso

Check the mapping status

CIMC_APIC-01 /vmedia # show mappings detail        

The Map-Status should be shown as OK.

Step?3: Access the Serial over LAN (SOL)/KVM console, powercycle the APIC server to access the boot menu (press F6).

Choose the boot Image You just mapped i.e APIC6.0_iso

NOTE: APIC installation will take a bit of time, so wait with patience and as soon as it is done, you'll be able to see APIC in initialization state.

PRETTY STRAIGHTFORWARD? Try it You'll come to know ...

• APIC L4, received without APIC OS installed
• I followed the same procedure but something wrong with APIC L4 hardware, after detailed testing/verification session with Cisco TAC, we then created the RMA and received new APIC L4, I followed the same procedure and successfully installed the APIC OS then.

Reference:

Cisco APIC Installation and ACI Upgrade and Downgrade Guide

Cisco ACI Upgrade Checklist

Cisco APIC Release Notes

Release Notes for Cisco Nexus 9000 Series Switches in ACI Mode

Cisco ACI Upgrade Matrix

Pre-Upgrade Validation Script

First of all, good luck with your upcoming upgrade/downgrade and APIC migrations, if you have any questions/suggestions feel free to reach out to me.

FINAL NOTE:

If you're looking for support/services for your upcoming APIC OS upgrade/downgrade or APIC migration from M1/M2/M3/L1/L2/L3 to APIC M4/L4, don't hesitate to contact Me.