The Impact of FC-Ports without a Link
FC ports that are not used and do not have a link should be deactivated, as these significantly extend the runtime of a series of commands and operations (e.g. LPM).
(Note: our LPAR tool is used in some examples, but the corresponding commands on the HMC or the virtual I / O server are always shown!)
Two 4-port FC adapters are in use on one of our virtual I / O servers (ms26-vio1):
$ lpar lsslot ms26-vio1 DRC_NAME DRC_INDEX IOPOOL DESCRIPTION U78D3.001.XXXXXXX-P1-C49 21040015 none PCIe3 x8 SAS RAID Internal Adapter 6Gb U78D3.001.XXXXXXX-P1-C7 2103001C none PCIe3 4-Port 16Gb FC Adapter U78D3.001.XXXXXXX-P1-C2 21010021 none PCIe3 4-Port 16Gb FC Adapter $ (HMC: lshwres -r io --rsubtype slot -m ms26 --filter lpar_names=ms26-vio1)
However, only 2 ports of the 8 ports are cabled:
$ vios lsnports ms26-vio1 NAME PHYSLOC FABRIC TPORTS APORTS SWWPNS AWWPNS fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 $ (VIOS: lsnports)
When working with the virtual I / O server, it is noticeable, that some of the commands have an unexpectedly long runtime and sometimes hang for a long time. Some example commands are given below, along with the measured runtime:
(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats … Error opening device: /dev/fscsi1 errno: 00000045 Error opening device: /dev/fscsi2 errno: 00000045 Error opening device: /dev/fscsi3 errno: 00000045 Error opening device: /dev/fscsi5 errno: 00000045 Error opening device: /dev/fscsi6 errno: 00000045 Error opening device: /dev/fscsi7 errno: 00000045 real 1m13.56s user 0m0.03s sys 0m0.10s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time lsnports name physloc fabric tports aports swwpns awwpns fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 real 0m11.61s user 0m0.01s sys 0m0.00s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1 Error opening device: /dev/fscsi1 errno: 00000045 real 0m11.31s user 0m0.01s sys 0m0.01s (4)padmin@ms26-vio1:/home/padmin>
LPM operations also take significantly longer, since all FC ports are examined when searching for suitable FC ports for the necessary NPIV mappings. This can lead to delays in the range of minutes before the migration is finally started.
In order to avoid these unnecessarily long runtimes, FC ports that are not wired should not be activated. The fscsi device has the attribute autoconfig, with the possible values defined and available. By default, the value available is used, which means that the kernel configures and activates the device, even if it has no link, which leads to the waiting times shown above. If the autoconfig attribute is set to defined, the fscsi device is not activated, it then remains in the defined state.
The following example shows how to reconfigure the fscsi1 device:
$ vios chdev ms26-vio1 fscsi1 autoconfig=defined $ (VIOS: chdev -dev fscsi1 -attr autoconfig=defined) $ $ vios rmdev ms26-vio1 fscsi1 $ (VIOS: rmdev -dev fscsi1 –ucfg) $ $ vios lsdev ms26-vio1 fscsi1 NAME STATUS PHYSLOC PARENT DESCRIPTION fscsi1 Defined U78D3.001.XXXXXXX-P1-C2-T2 fcs1 FC SCSI I/O Controller Protocol Device $ (VIOS: lsdev -dev fscsi1) $ $ vios lsattr ms26-vio1 fscsi1 ATTRIBUTE VALUE DESCRIPTION USER_SETTABLE attach none How this adapter is CONNECTED False autoconfig defined Configuration State True dyntrk yes Dynamic Tracking of FC Devices True+ fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True+ scsi_id Adapter SCSI ID False sw_fc_class 3 FC Class for Fabric True $ (VIOS: lsdev -dev fscsi1 –attr) $
With the autoconfig=defined attribute, the fscsi device remains defined even when the cfgmgr is run!
If one repeats the runtime measurement of the commands above, one can see that the runtime of the commands has already measurably improved:
(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats … Error opening device: /dev/fscsi1 errno: 00000005 Error opening device: /dev/fscsi2 errno: 00000045 Error opening device: /dev/fscsi3 errno: 00000045 Error opening device: /dev/fscsi5 errno: 00000045 Error opening device: /dev/fscsi6 errno: 00000045 Error opening device: /dev/fscsi7 errno: 00000045 real 1m1.02s user 0m0.04s sys 0m0.10s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time lsnports name physloc fabric tports aports swwpns awwpns fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 real 0m9.70s user 0m0.00s sys 0m0.01s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1 Error opening device: /dev/fscsi1 errno: 00000005 real 0m0.00s user 0m0.02s sys 0m0.00s (4)padmin@ms26-vio1:/home/padmin>
The running time of the netstat command was shortened by 12 seconds, the lsnports command was about 2 seconds faster.
We now also set the autoconfig attribute to defined for all other unused FC ports:
$ for fscsi in fscsi2 fscsi3 fscsi5 fscsi6 fscsi7 > do > vios chdev ms26-vio1 $fscsi autoconfig=defined > vios rmdev ms26-vio1 $fscsi > done $
Now we repeat the runtime measurement of the commands again:
(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats … Error opening device: /dev/fscsi1 errno: 00000005 Error opening device: /dev/fscsi2 errno: 00000005 Error opening device: /dev/fscsi3 errno: 00000005 Error opening device: /dev/fscsi5 errno: 00000005 Error opening device: /dev/fscsi6 errno: 00000005 Error opening device: /dev/fscsi7 errno: 00000005 real 0m0.81s user 0m0.03s sys 0m0.10s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time lsnports name physloc fabric tports aports swwpns awwpns fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 real 0m0.00s user 0m0.01s sys 0m0.01s (0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1 Error opening device: /dev/fscsi1 errno: 00000005 real 0m0.04s user 0m0.00s sys 0m0.00s (4)padmin@ms26-vio1:/home/padmin>
The netstat command now takes less than 1 second, the lsnports command only 0.1 seconds.
It is therefore worthwhile to set the autoconfig attribute for unused FC ports to defined!
A german version of the article is available here: https://powercampus.de/die-auswirkungen-von-fc-ports-ohne-link
Sr. Systems Engineer
4 年Thanks for the excellent Article. I tested this on our VIOs and was not able to display the autoconfig attribute. Is this some limitation with the type of adapters that we use? lsdev -dev fscsi1 name????????????status?????description fscsi1??????????Defined????FC SCSI I/O Controller Protocol Device lsdev -dev fscsi1 -attr attribute?????value????description?????????????????????????????????????user_settable attach????????none?????How this adapter is CONNECTED???????????????????False dyntrk????????yes??????Dynamic Tracking of FC Devices??????????????????True+ fc_err_recov??fast_fail FC Fabric Event Error RECOVERY Policy???????????True+ io_thrshld_tmr no???????FC special error recovery threshold timer enable True+ scsi_id?????????????????Adapter SCSI ID?????????????????????????????????False sw_fc_class???3????????FC Class for Fabric?????????????????????????????True ioslevel 3.1.1.25 Below is the adapter type and the microcode level fcs0 210313?PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729 fcs1 210313?PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729 fcs2 210313?PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729 fcs3 210313?PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729
Lead Service Architect | IBM Champion 2020 - 2025
4 年Thanks for great article Armin! I tested this on my dev VIOS ; but the results wasnt like expected, am I doing something wrong here: padmin@power-d02-vio1:/home/padmin # ioslevel????????????????? 3.1.2.10 padmin@power-d02-vio1:/home/padmin #?lsdev -type adapter |grep fcs?????? fcs0??????Available?PCIe3 10Gb 4-Port FCoE Adapter (df1060e214101004) fcs1??????Available?PCIe3 10Gb 4-Port FCoE Adapter (df1060e214101004) fcs2??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs3??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs4??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs5??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) padmin@power-d02-vio1:/home/padmin # lsnports name??????physloc????????????fabric tports aports swwpns?awwpns fcs2??????U78C9.001.WZS0177-P1-C8-T1?????1??64??42?2048??1978 fcs3??????U78C9.001.WZS0177-P1-C8-T2?????1??64??42?2048??1978 padmin@power-d02-vio1:/home/padmin #????? padmin@power-d02-vio1:/home/padmin #? padmin@power-d02-vio1:/home/padmin # chdev -dev fscsi0 -attr autoconfig=defined fscsi0 changed padmin@power-d02-vio1:/home/padmin # rmdev -dev fcs0 -ucfg fcs0 Defined padmin@power-d02-vio1:/home/padmin #??????????? padmin@power-d02-vio1:/home/padmin # chdev -dev fscsi1 -attr autoconfig=defined fscsi1 changed padmin@power-d02-vio1:/home/padmin # rmdev -dev fcs1 -ucfg fcs1 Defined padmin@power-d02-vio1:/home/padmin #??????????? padmin@power-d02-vio1:/home/padmin # chdev -dev fscsi4 -attr autoconfig=defined fscsi4 changed padmin@power-d02-vio1:/home/padmin # rmdev -dev fcs4 -ucfg fcs4 Defined padmin@power-d02-vio1:/home/padmin #??????????? padmin@power-d02-vio1:/home/padmin # chdev -dev fscsi5 -attr autoconfig=defined fscsi5 changed padmin@power-d02-vio1:/home/padmin # rmdev -dev fcs5 -ucfg fcs5 Defined padmin@power-d02-vio1:/home/padmin #?lsdev -type adapter |grep fcs fcs0??????Defined??PCIe3 10Gb 4-Port FCoE Adapter (df1060e214101004) fcs1??????Defined??PCIe3 10Gb 4-Port FCoE Adapter (df1060e214101004) fcs2??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs3??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs4??????Defined??PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs5??????Defined??PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) padmin@power-d02-vio1:/home/padmin # cfgdev padmin@power-d02-vio1:/home/padmin #?lsdev -type adapter |grep fcs fcs0??????Available?PCIe3 10Gb 4-Port FCoE Adapter (df1060e214101004) fcs1??????Available?PCIe3 10Gb 4-Port FCoE Adapter (df1060e214101004) fcs2??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs3??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs4??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) fcs5??????Available?PCIe2 4-Port 8Gb FC Adapter (df1000f114100104) e.g eventough autoconfig is set to defined...those adapters come back on Available state after running cfgdev
manager of myself : in progress :-)
4 年Thank you Armin, excellent article, I tweeted it: https://twitter.com/phhermes/status/1333013659736567808
IBM Power Systems with AIX and Storage Specialist at CFAO TECHNOLOGIE ALGERIA
4 年Very interesting thanks for the share
Senior UNIX Systems Engineer and Security Focal Point for Unix presso Avaloq
4 年Very interesting