Troubleshoot Cisco Application Infrastructure (Initial Fabric Setup)
"Learning by doing" is a great way to move forward in your career. In this article, I would like to share my experience regarding Cisco ACI Troubleshooting Guide, specifically focusing on Section #1 of the Initial Fabric Setup: Troubleshoot ACI Fabric Discovery Initial Fabric Setup.
Scenario:
Recently, I worked on a project for one of my customers. I was responsible for delivering a Cisco ACI deployment from scratch to go live. This involved setting up the equipment, staging, and building the entire fabric from the ground up (Green field deployment).
During the setup, I encountered an issue where the Seed Leaf switch was failing to register with the APIC controller (Leaf switch stuck in inactive state). This posed a significant challenge to the initial fabric discovery process. Resolving this issue required a thorough investigation into connectivity checks, configuration verifications, and potential misconfigurations or hardware issues. In this article, I will discuss all these steps and challenges to share my experience and help you address similar issues when delivering a project from scratch.
Cisco ACI Hardware
Cisco ACI Software
To begin the ACI Fabric deployment (Green field) from scratch, you first need to set up the CIMC/APIC Controller. For more detailed guidance on this topic, you can review my article titled "Cisco APIC M1/M2/M3/L1/L2/L3 to M4/L4 Cluster Migration."
NOTE:
APIC M4 by default comes with ACI 6.0x software version, if You're required to deploy fabric with 5.3x version, You need to downgrade the ACI fabric which also address in same article discussed above in following section
Let's build the ACI Fabric from Scratch
Pre-requisites
Optionally but recommended
Let's assume you have already racked and stacked the equipment, powered it up, and now it's time to begin building the ACI fabric, starting with Seed Leaf Initialization.
How ACI Fabric Discovery Works?
When the Controller Type is?Physical:
a.?????? APICs connect in-band to Leafs via Virtual Interface Cards (VICs)
(1)???? Auto assign it an IP address via DHCP
(2)???? Manually commission the leaf with unique Node ID
b.?????? APIC discovers a Leaf via LLDP
(1)???? Auto assign it an IP address via DHCP
(2)???? Manually commission the leaf with unique Node ID
c.??????? APIC then learns about all Spines through LLDP from the Leaf
(1)???? Assigns the Spines IP address via DHCP
(2)???? Manually commission the Spines with unique Node IDs
d.?????? APIC then learns about the rest of the Leaf via DHCP through the Spines
(1)???? Auto-assign an IP and manually commission them with unique Node IDs
e.?????? During automation, IP addressing & IS-IS routing is auto-configured
(1)???? VRF overlay-1 is used for VXLAN transport
(2)???? Use the command show ip route vrf overlay-1 from CLI on Leafs and Spines for verification
Register Seed Leaf Switch
Step #1 Register the seed leaf switch. Follow the fabric discovery procedures and stages outlined below to complete the initialization process.
Normally, LF101 should register and then proceed to discover the first available Spine Switch, which will also be registered. Subsequently, all available leaf and spine switches are discovered and registered, followed by the discovery and preparation of all APICs to build the APIC cluster.
However, things don’t always go as expected...
Let's start troubleshooting Inactive Leaf Switch
Beginning in ACI 4.2, a new CLI command is available on fabric nodes to assist in the diagnosis of common discovery issues. The following sections will cover the checks performed and provide additional validation commands to assist in troubleshooting failures.
When I ran the command (show discoveryissues) on LF101, the following issues were found:
LF101# show discoveryissues
=====================================================
Check 1 Platform Type
=====================================================
Test01 Retrieving Node Role ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? [Info] Current node role: LEAF
? ? ? [Info] Please check CH09 DHCP status section for configured node role
=====================================================
Check 2 FPGA/BIOS in sync test
=====================================================
Test01 FPGA version check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? [Info] No issues found for FPGA versions
Test02 BIOS version check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?FAILED
? ? ? [Warn] BIOS version mismatch for sys/ch/supslot-1/sup/running
ExpectedVersion: v01.09(10/08/2023)
CurrentVersion: v01.08(05/31/2022)
=====================================================
Check 3 HW Modules Check
=====================================================
Test01 Fans status check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] All fans status is ok
Test02 Power Supply status check? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? [Info] All PSUs status is ok
Test03 Fan Tray status check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] All FanTrays status is ok
Test04 Line Card status check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] All LineCard status is ok
=====================================================
Check 4 Node Version
=====================================================
Test01 Check Current Version ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] Node current running version is : n9000-15.3(1d)
=====================================================
Check 5 System State
=====================================================
Test01 Check System State ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Warn] Top System State is : out-of-service
? ? ? [Info] Node upgrade is in notscheduled state
=====================================================
Check 6 Updated LLDP Adjacencies
=====================================================
Port: eth1/53
? ?Test02 Adjacency Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? ? ? ? ?[Error] Error occurred while parsing through LLDP TLVs
Port: eth1/54
? ?Test02 Adjacency Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? ? ? ? ?[Error] Error occurred while parsing through LLDP TLVs
Port: eth1/1
? ?Test02 Wiring Issues Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? ? ? ? ?[Info] No Wiring Issues detected
? ?Test03 Port Types Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? ? ? ? ?[Info] No issues with port type, type is:leaf
? ?Test04 Port Mode Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? ? ? ? ?[Info] No issues with port mode, type is:trunk
? ?Test02 Adjacency Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? ? ? ? ?[Info] Adjacency detected with APIC
=====================================================
Check 7 BootStrap Status
=====================================================
Test01 Check Bootstrap/L3Out config download ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Warn] BootStrap/L3OutConfig URL not found
? ? ? [Info] Ignore this if this node is not an IPN attached device
=====================================================
Check 8 Infra VLAN Check
=====================================================
Test01 Check if infra VLAN is received ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] Infra VLAN received is : 3967
Test02 Check if infra VLAN is deployed ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] Infra VLAN deployed successfully
=====================================================
Check 9 DHCP Status
=====================================================
Test01 Check Node Id ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] Node Id received is : 101
Test02 Check Node Name ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? [Info] Node name received is : LF-101
Test03 Check TEP IP ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] TEP IP received is : 10.0.176.64
Test04 Check Configured Node Role ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? [Info] Configured Node Role received is : LEAF
=====================================================
Check 10 IS-IS Adj Info
=====================================================
Test01 check IS-IS adjacencies ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Warn] No IS-IS adjacencies found
? ? ? [Info] Ignore this if it is a APIC attached leaf
=====================================================
Check 11 Reachability to APIC
=====================================================
Test01 Ping check to APIC ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? [Info] Ping to APIC IP 10.0.0.1 from 10.0.176.64 successful
=====================================================
Check 12 BootScript Status
=====================================================
Test01 Check BootScript download ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Error] Cannot retrieve bootScript download status
Let's begin addressing the FAILED issues identified in the LF101 output above.
Check02 — FPGA/EPLD/BIOS out of sync
The FPGA, EPLD and BIOS versions could affect the leaf node's ability to bring up the modules as expected. If these are too far out of date, the interfaces of the switch could fail to come up. The user can validate the running and expected versions of FPGA, EPLD, and BIOS with the following moquery commands
LF101# moquery -c firmwareCardRunning
=====================================================
Check 2 FPGA/BIOS in sync test
=====================================================
Test01 FPGA version check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? [Info] No issues found for FPGA versions
Test02 BIOS version check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?FAILED
? ? ? [Warn] BIOS version mismatch for sys/ch/supslot-1/sup/running
ExpectedVersion: v01.09(10/08/2023)
CurrentVersion: v01.08(05/31/2022)
To address the mismatch BIOS version, we have two options available
OPTION #1
If the Leaf or Spine switch is already registered with the APIC, upgrading the switch software to the latest required version will resolve the BIOS version mismatch. However, in my case, the Seed Leaf was not registered, so this option was not applicable.
OPTION #2
As per Troubleshooting Cisco Application Centric Infrastructure, Second Edition, you can follow the below procedure to fix it.
I proceeded with Option #2 to resolve the issue, but unfortunately, it did not work.
NOTE:
In my case, the switch output showed version v01.09 as the current version, but when running the show discoveryissues command, it indicated a version mismatch.
biosVer : v01.09(10/08/2023)
Check01 — System state
When the leaf has been allocated a Node ID and registered to the fabric, it will begin to
download its bootstrap and then transition to an in-service state
=====================================================
Test01 Check System State ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Warn] Top System State is : out-of-service
? ? ? [Info] Node upgrade is in notscheduled state? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
NOTE:
In my case, the switch output showed as (out-of-service), According to the Cisco ACI Troubleshooting Guide, this status will transition to 'in-service' once the leaf is successfully registered with the APIC.
Check 6 Updated LLDP Adjacencies
=====================================================
Check 6 Updated LLDP Adjacencies
=====================================================
Port: eth1/53
? ?Test02 Adjacency Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? ? ? ? ?[Error] Error occurred while parsing through LLDP TLVs
Port: eth1/54
? ?Test02 Adjacency Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? ? ? ? ?[Error] Error occurred while parsing through LLDP TLVs
Port: eth1/1
领英推荐
? ?Test02 Wiring Issues Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? ? ? ? ?[Info] No Wiring Issues detected
? ?Test03 Port Types Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? ? ? ? ?[Info] No issues with port type, type is:leaf
? ?Test04 Port Mode Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?PASSED
? ? ? ? ? ? ?[Info] No issues with port mode, type is:trunk
? ?Test02 Adjacency Check ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? PASSED
? ? ? ? ? ? ?[Info] Adjacency detected with APIC
NOTE:
In this check, I observed that Port: eth1/1 did not indicate any wiring issues, which suggests that the physical connection between APIC and LF101 is intact.
Check 7 BootStrap Status
=====================================================
Check 7 BootStrap Status
=====================================================
Test01 Check Bootstrap/L3Out config download ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Warn] BootStrap/L3OutConfig URL not found
? ? ? [Info] Ignore this if this node is not an IPN attached device
NOTE:
In my case, the result for Check 7 was FAILED, but it did not impact the overall process.
Check 10 IS-IS Adj Info
=====================================================
Check 10 IS-IS Adj Info
=====================================================
Test01 check IS-IS adjacencies ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Warn] No IS-IS adjacencies found
? ? ? [Info] Ignore this if it is a APIC attached leaf
NOTE:
In my case, the result for Check 10 was FAILED, but it did not impact the overall process.
Test01 Check BootScript download
=====================================================
Test01 Check BootScript download ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FAILED
? ? ? [Error] Cannot retrieve bootScript download status
NOTE:
In my case, the result for Check 01 was FAILED. This issue will not be resolved until the Leaf is registered with the APIC and successfully downloads the boot script.
In addition to the steps outlined in the Cisco ACI Troubleshooting Guide, I also used the following commands to display different outputs to further troubleshoot the issue.
APIC Commands
APIC1# cat /data/data_admin/sam_exported.config
APIC1# cat /etc/sysconfig/network-scripts/ifcfg-bond0.3967
APIC1# show lldp
APIC1# acidiag verifyapic
APIC1# acidiag run lldptool out eth2-1
APIC1# acidiag run lldptool in eth2-1
APIC1# ps aux | grep dhcp
LF101 Commands
LF101# show lldp neighbors
LF101# show lldp traffic
LF101# show processes |grep lldp
LF101# moquery -c topSystem
LF101# acidiag avread
LF101# moquery -c lldpInst
LF101# show vlan encap-id 3967
LF101# moquery -c lldpIf -f 'lldp.If.wiringIssues!=""'
LF101# moquery -c firmwareCardRunning
LF101# show inventory
LF101# moquery -c pconsBootStrap
LF101# tcpdump -ni kpm_inb port 67 or 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on kpm_inb, link-type EN10MB (Ethernet), capture size 262144 bytes
NOTE: I wasn't able to get output for above command.
Let's attempt to resolve the problem, even though there is currently no clear visibility into the exact issue.
Cisco ACI Software Upgrade
APIC Cluster
If you're using ACI version 6.0(x) or later, the procedure for managing the APIC cluster differs from the process used in earlier ACI versions.
STEP #1
Erase the configuration on the APICs by connecting to the KVM console and entering the following commands:
APIC1 # acidiag touch clean
APIC1 # acidiag touch setup
APIC1 # acidiag reboot
NOTE:
The acidiag touch command alone is not sufficient for this procedure, as it does not trigger the APIC to run the startup script. It is recommended to use acidiag touch setup instead, which ensures the APIC is brought up with the initial setup process.
STEP #2
APIC #1 is ready to be initialized. Access the vKVM console and enter the following details to initialize APIC #1. At this stage, you will only need to provide the following information in the CLI:
admin user configuration ...
Enter the password for admin [None]:
Enter the password for admin [None]:
Out-of-band management configuration ...
Enter the IP Address [192.168.0.1/24]: 192.168.0.1/24
Enter the IP Address of the default gateway [192.168.0.254/24]: 192.168.0.254/24
Would you like to edit the configuration (y/n) [n]: n
System pre-configured successfully.
Use: https://192.168.0.1 to complete the bootstrapping
STEP #3
Access the APIC #1 via GUI (https://192.168.0.1)
STEP #4
The first step is to enter the Connection Type information. On the Connection Type screen, select the type of connection between the APIC and the fabric.
The options are:
Click NEXT
STEP #5
The second step is entering the Cluster Details. Enter the fabric-level details in the Cluster Details screen.
a.?????? Fabric Name: Enter a name for the fabric.
b.?????? Cluster Size: The default cluster size displayed is "3", which is the recommended minimum cluster size. You can modify this value, based on your cluster size. The supported values are 1, 3, 4, 5, 6, 7, 8, and 9.
c.?????? GiPo Pool: Enter the IP address used for fabric multicast. The default address is 225.0.0.0/15. The range is from 225.0.0.0/15 to 231.254.0.0/15. The prefixlen must be 15 (128k IP addresses).
(1).???????? You cannot change this value after you have completed the configuration. Having to modify this value requires a wipe of the fabric.
d.?????? Pod ID: (applicable only for directly connected APICs (virtual and physical)) the pod ID is displayed. If this is your first APIC, "1" is auto-populated. Subsequent APICs of the cluster can be associated with any pod number.
e.?????? For a remotely-attached APICs, pod is 0.
f.?????? TEP Pool: (applicable only for directly connected APICs (ESXi virtual APIC and physical APIC)) enter the subnet of addresses used for internal fabric communication. The size of the subnet used will impact the scale of your pod.
(1).???????? You can not change this value after you have completed the configuration. Having to modify this value requires a wipe of the fabric.
f.?????? Infrastructure VLAN: Enter the VLAN ID for fabric connectivity (infra VLAN). This VLAN ID should be allocated solely to ACI, and not used by any other legacy device(s) in your network. Default value is 3914. The range is from 0 to 4093.
(1).???????? You can not change this value after you have completed the configuration. Having to modify this value requires a wipe of the fabric.
Click NEXT
STEP #6
The third step is entering the?Controller Registration?details. Click?Add Controller?to add the first?APIC?(of the cluster). Enter the following details:
When the Controller Type is?Physical:
a.?????? CIMC Details pane
(1)???? IP Address: The CIMC IP address. Only for the first?Cisco APIC, this IP address is auto-populated. When you add more controllers to the cluster, you need to enter the CIMC IP addresses.
(2)???? Username: The username to access the CIMC. The username is auto-populated (for the first controller and subsequent controllers).
(3)???? Password: Enter the password to access CIMC. For the first controller, the password is auto-populated. For the subsequent controllers, enter the password.
(4)???? Click?Validate.?Validation success?is displayed on successful authentication.
If the CIMC is unreachable from the?Cisco APIC?out of band management IP address due to the CIMC NIC mode settings, change the NIC mode or enter JSON strings to perform the bootstrap.
b.?????? General pane
(1)???? Name: Enter a name for the controller.
(2)???? Controller ID: If it is the first controller of the cluster, "1" is auto-populated. If it is the second controller, "2" is auto-populated, and so on (increasing order).
(3)???? Pod ID: (applicable only for a directly-connected?APIC) the pod ID is auto-populated for?APIC?1 of the cluster. For subsequent controllers of the cluster, enter a value. The range is from 1 to 128.
(4)???? Serial Number: The serial number is auto-populated (for?APICs 1 to N, where N is the cluster size) after CIMC validation.
APIC?1 verifies the reachability of the CIMC IP addreses and also captures the serial number of the new?APICs.
c.?????? Out of Band Network pane
(1)???? IPv4 Address: For?APIC?1, the address is auto-populated. For subsequent?APICs, enter the IP address (as defined during the deployment).
(2)???? IPv4 Gateway: For?APIC?1, the gateway address is auto-populated. For subsequent?APICs, enter the IP address (as defined during the deployment).
If you have enabled IPv6 addresses for OOB management earlier (step 5), enter the IPv6 address and gateway.
d.?????? Infra L3 Network pane (this pane is displayed only if the?Connection Type?that you chose earlier is remotely attached through a Layer 3 network).
(1)???? IPv4 Address: Enter the infra network IP address.
(2)???? IPv4 Gateway: Enter the infra network IP address of the gateway.
(3)???? VLAN: Enter a VLAN ID.
On the?Controller Registration?screen, after you have entered and saved the first?APIC?details, click?Add Controller?to add another?APIC?to the cluster.
STEP #7
e.?????? Click Next
(1)???? The?Next?button is disabled until all the controllers for a cluster are added. This is defined by the value you have entered for?Cluster Size?in the?Cluster Details?screen.
(2)???? You can use the?Back?button to navigate to an earlier screen. After adding an?APIC, click?Edit Details?to edit the information for an?APIC. Except the first?APIC, you can delete the other controllers, if required, by clicking the delete icon.
e.?????? In the?Summary?screen, review the updates, and click?Deploy.
g.?????? The?Cluster Status?page is displayed, which shows the current status of the cluster formation. Wait for a few minutes after which you will be automatically redirected to the standard?Cisco APIC?GUI.
NOTE:
The APIC cluster will not fully form until the Leaf/Spine registration is complete. Once registered, APIC #2 and #3 will sync with APIC #1 and automatically join the cluster.
Cisco ACI Software Upgrade (Nexus 9000 Switch via CLI)
STEP #1
Download the software file (aci-n9000-dk9.16.0.7e-cs_64.bin) and copy it to a USB drive. Connect the USB directly to the Nexus 9000 switch.
Ensure the USB is properly connected and contains the correct image for installation.
STEP #2
Log in to the Leaf or Spine switch and reload it.
STEP #3
Type Ctrl + c to enter the (loader >) mode
STEP #4
Type dir to verify the ACI software image (usb) on the Nexus 9K Switch
STEP #5
Type boot <image name> to boot from the image you intended to upgrade
STEP #6
Check and verify the current boot variable settings for this node by using the following command
cat /mnt/cfg/0/boot/grub/menu.lst.local
cat /mnt/cfg/1/boot/grub/menu.lst.local
Set the node's boot variables to the intended or desired firmware version by ensuring the firmware image is located in /bootflash. Use the following command:
clear-bootvars
setup-bootvars.sh <firmware image>
i.e: "aci-n9000-dk9.16.0.7e-cs_64.bin"
Delete bootflash/auto-s
none# clear-bootvars.sh
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done
none# setup-bootvars.sh aci-n9000-dk9.16.0.7e-cs_64.bin
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done
Now, erase the configuration and reload the Nexus 9000 switch by using the following commands:
STEP #7
After the switch comes back online following the reload, run the show version command to verify if the software has been updated and check for the correct version, such as aci-n9000-dk9.15.3.1d.bin or aci-n9000-dk9.16.0.7e-cs_64.bin.
NOTE:
The output in the screenshots reflects the installation of aci-n9000-dk9.15.3.1d.bin, not aci-n9000-dk9.16.0.7e-cs_64.bin. So, please don't be confused or consider it an error; the output is correct for the version being installed.
NOTE:
Ideally, Cisco Nexus 9000 switches should be upgraded via the APIC rather than individually. However, due to an issue with Seed Leaf registration, I had to upgrade the Leaf switch (Seed Leaf) directly through the CLI as the only available option.
Seed Leaf Registration:
Now that we have upgraded the APIC Controller to the 6.0(7e) software version, along with the compatible software version on Leaf101, we can proceed to repeat the Seed Leaf registration process.
STEP #1
Register the leaf switch as explained earlier in the article
STEP #2
After registration, if the Seed Leaf (LF101) status is still inactive, disconnect all connections from the Leaf switch except for the connection to APIC#1. Then, run the following command on the Leaf switch:
LF101 # acidiag touch clean
LF101 # reload
Once the Leaf switch comes online, it will register with the APIC controller without any issues. Leaf switch LF101 will automatically register with APIC #1.
STEP #4
Connect the Leaf switch to the Spine switches and other devices as needed.
Ensure to review and follow the appropriate hardware installation guide for both Leaf and Spine switches to ensure proper setup and connectivity.
Key take away from this guide is ...
STEP #5
Register the Leaf and Spine switches until the APIC cluster is fully formed and the fabric initialization is completed.
Here are some possible options I tried while troubleshooting the Seed Leaf Switch registration issue, where the Seed Leaf was stuck in an inactive state after registration and didn't transition to the Active state:
a.?????? Verified APIC/Leaf/Spine/Cable-SFPs compatibility
b.?????? Verified the physical connections by following the connectivity table.
c.??????As per the Cisco Nexus 9000 ACI-Mode Switches Release Notes for Release 15.3(1), the Cisco Nexus N180YC-FX3 can run aci-n9000-dk9.15.3.1d.bin, which is supposed to be compatible with aci-apic-dk9.5.3.1d.iso, but the N9K FX3 series Leaf switches were unable to register with the APIC.
(3)??????APIC/N9K Software- 5.3(1d) download
d.??????Changed physical ports on the APIC Controller, switching from Eth 2/2 to Eth 2/1, and vice versa.
(1)??????Tried 10GE SFP/DAC instead of the 25GE DAC cable.
(a)??????SFP-10/25G-CSR-S
(b)??????SFP-H10GB-CU5M=
(c)??????SFP-H25G-CU5M=
Reference:
Best of luck with troubleshooting the Seed Leaf registration issue (stuck inactive). If you have any questions or need assistance, feel free to reach out to me!
FINAL NOTE:
If you're seeking support or services for your ACI projects—whether greenfield or brownfield, planning/designing/migrating/implementing ACI fabric, or training (including DCACI/DCACIA Etc), feel free to reach out to me.
Principal System Engineer at Boehringer Ingelheim | CCIE Data Center| Cisco ENCOR| SPSP
4 个月Insightful
Network Infrastructure Engineer @ US Navy | Active DoD SECRET cleared | 2x CCNP | CCNA | AWS Solution Architect
4 个月Hammad T. Thanks for sharing! Any guide or resources for deploying Virtual APIC?
3*CCIE (R&S, SP, DC) / AZ-900 / DevOps
4 个月Very helpful