Preparing VIOS Update. VSCSI

In the last articles we automated network switch-over using #Ansible:

• Shared Ethernet Adapter
• SR-IOV with VNICs
• Network Interface Backup

Now it is time to start with storage configuration. Generally speaking there are two types of virtualized storage configuration in PowerVM:

• VSCSI-connected storage
• NPIV-connected storage

Both of them have their own advantages and disadvantages.

NPIV usually performs better. You have better performance and more options to configure storage connection. On the other side you must install vendor-provided drivers into each NPIV-connected LPAR. If you have some funny drivers from funny vendors (I will not name them), you must check before upgrade if the vendor supports the newest versions of AIX / IBM i / Linux. Sometimes the drivers don't integrate very well with the operating systems and you must follow some complex procedures to upgrade your operating system or worse - use VSCSI for the operating system itself (rootvg) and NPIV for data.

With NPIV you have a wide range of possible storage drivers to automate. Commands for PowerPath differ from commands for Hitachi's HDLM.

VSCSI is more generalized protocol and you don't have any problems with drivers. It is supported by AIX, IBM i and all Linux distributions on IBM Power. On the other hand, it is not so performant as NPIV and you move the control of the storage to the Virtual I/O Server.

With VSCSI you have several types how you can virtualize your disk storage:

• one to one
• one to many
• many to many

You can get one disk device from your storage and map it to the target LPAR (one to one).

You can get one disk device from your storage, divide it into logical volumes and map each logical volume to another LPAR (one to many).

You can get many disk devices from your storage, create one storage pool from all of them and then attach parts of the pool to LPARs (many to many).

On AIX LPAR we have two VSCSI devices. Each device represents a connection to Virtual I/O Server's vhost device:

You can check corresponding Virtual I/O Servers and their vhost-devices using kdb (Kernel Debugger).

For each disk using VSCSI you can check the status of the paths using lspath command:

If you have dual VIOS configuration and you've lost one VIOS, the path to the VIOS will be missing. When the VIOS comes back, the path will be enabled again. You can tune how fast AIX finds out that the path is missing or online again. More about VSCSI tuning you can read in the Redbook "IBM PowerVM Best Practices".

As for us now it is important, that we can switch off a path to prevent writes through VIOS which is being updated. We can do it using chpath command on AIX.

When we are done with the update, we can enable the path again.

Now we have all the pieces of the information and ready to automate. We switch off VSCSI paths on all AIX client LPARs connected to VIOS, we want to update, using Ansible.

I use the same dynamic inventory as I used last time when automating NIB switch over.

Same way as last time I define which VIOS I want to update in the variable (line 6). I use one additional collection community.general (line 4), because I want to search through Ansible facts using json_query filter.

Because it is a very small example playbook for AIX, we start with gathering device information from AIX hosts. If we don't have AIX LPARs on the managed systems, the rest of the playbook will be skipped.

After we gathered device information we select only VSCSI-connected disks (line 19) using json_query filter into new variable named "vscsi" (line 17). We do it only for AIX LPARs and if we gathered information about devices (lines 20-23).

In the next step we run lspath command (line 26) for each VSCSI disks we've found (line 28 - loop over vscsi variable and lines 31-33 - only if vscsi has data). The output of the command is saved into the variable "lspath_output" (line 27) for future analysis.

Because I don't like the standard title of a loop item, I changed it to more useful for me (lines 29-30).

In line 34 we tell Ansible that we don't change anything executing lspath. By default every command run by shell, command or raw Ansible modules increments the number of changes made on the target system. By executing lspath we don't change anything, we just get more information.

Additionally to paths we need VIOS connection information from kdb.

In the next step we parse the output of kdb. The regular expression in line 45 is suitable for my configuration. I have only two VIO servers and only two VSCSI devices. That's why I simply make an array of four values:

• first VSCSI device
• corresponding VIO server
• second VSCSI device
• corresponding VIO server

If you have more VSCSI devices, you may need to update the regex in line 45.

To make our decision easier in the next step, we build a complex structure in this step. This structure contain disk name, paths to the disk and VIOS connection information.

We loop over all lspath_output.results (line 52). Each item in this loop represents output of lspath command we run earlier.

The output itself is in item.stdout_lines as array of strings.

The disk name is in item.item.name. Yes, it is not typo. It is called really so.

We add two fields with the names of VSCSI devices (vio_info[0][0] and vio_info[0][2]) and with the values of the corresponding VIOS (vio_info[0][1] and vio_info[0][3]).

The resulting structure will look like:

ok: [lpar] => {
? ? "paths": [
? ? ? ? {
? ? ? ? ? ? "disk": "hdisk0",
? ? ? ? ? ? "paths": [
? ? ? ? ? ? ? ? "vscsi0",
? ? ? ? ? ? ? ? "vscsi1"
? ? ? ? ? ? ],
? ? ? ? ? ? "vscsi0": "vio2",
? ? ? ? ? ? "vscsi1": "vio1"
? ? ? ? }
? ? ]
}        

As you see the variable paths is an array, because we can have many VSCSI disks. Because we didn't use or define it earlier, we provide a default value to the variable using default() filter in line 51.

After we found all the information we can switch off paths through our VIOS. First of all we loop over all paths we've found earlier (line 62). The paths are in subelements "paths" of the variable "paths". Yes, as you see, I can choose the right variable names to create as much confusion as possible.

Not every LPAR has VSCSI disks and as such - the variable "paths". That's why we use default() filter again. Using the filter we ensure that the command will be executed only on LPARs with VSCSI disks.

Why don't we use "when" as earlier in the playbook?

Ansible evaluates first the expression in loop (line 62) and then compares it to the expression in when (lines 65-66). Because we use a subelement of the variable, which may be undefined, in line 62, Ansible will throw an exception for each host which doesn't have the variable. Even if we add "when: paths is defined". For all hosts without VSCSI disks the playbook will end earlier as we may wish and with an error. If you implement it as a part of a bigger playbook, it might not always be the best strategy. That's why it is easier to make a default empty array and iterate over it.

After we joined the information using subelements filter, we have item.0 as our original structure and item.1 as paths subelement from our original structure.

We iterate over all our path information and if the path connected to VIOS, which we want to update - "item.0[item.1]", we execute chpath to switch off the path.

Don't forget to enable all the paths after you updated the VIOS!

I hope it helps you in your automation journey!

Have fun with Ansible!

Andrey