Running PerfPMR Scripts: Configuration and Network

In part one we looked at different ways to run the PerfPMR diagnostic utility as a whole, focusing on some important flags that aid in the proper execution of PerfPMR. Some of these flags help capture important systemic events like program starts and stops, while others allow us to customize PerfPMR output. Now we’re taking the concept of PerfPMR customization to a deeper level. I’ll show you how to execute only the parts of PerfPMR you’ll need in any given situation. I'll also explain how to add your own scripts so they're executed along with the rest of PerfPMR.

In part one I noted that the majority of the executable files that are created when installing PerfPMR have a .sh suffix appended to their names. Once you install PerfPMR, do a long listing in your installation directory and you’ll see these *.sh files. The files that are so appended can be run independently of the rest of PerfPMR. While PerfPMR addresses dozens of informational categories, very often in your own diagnostic efforts, you don’t need all of the voluminous data a full run of PerfPMR provides. In fact, it's very likely you'll focus on one issue at a time. I know that's typical in my own performance practice. Maybe you want only networking data or only storage data. Perhaps you're interested in AIX kernel traces, or you'd just like to look at locking activity.

Thankfully, the PerfPMR developers make it easy to drill down to perform a specific task. With this in mind, let's get into executing individual scripts with PerfPMR. Mind you, this is far from a complete description, but what follows should give you an idea of PerfPMR’s flexibility.

Configuration

All of my customers – whether they have one AIX system or thousands – keep some sort of record of the configuration of those systems. Generally, these records are compiled into a concise operational “run book” that is easily referenced by everyone in an IT organization. In short, configuration is a big deal, and PerfPMR is a big help in this regard, because it allows you to get detailed configuration information with just a few easy commands.

One of the scripts that's part of the PerfPMR whole is called config.sh. When PerfPMR runs in its default mode, config.sh also runs, producing roughly 50 files, each of which zeroes in on a different configuration aspect of your system. One of these files is called config.sum (sum being short for summary). Along with a prtconf and the output from an lparstat -i command, the config.sum file provides comprehensive information about the way your system is put together. Add or subtract whatever other commands – or files generated by the config.sh script – you wish, but config.sum can serve as the basis for a thorough record of configuration.

Generating the config.sum file is easy. As root, preferably in the /perfdata directory you’ve already created, run this command:

???????????perfpmr.sh -x config.sh

Then let PerfPMR chug and take note of the messages it leaves in your terminal session. Start by picking out the config.sum file and FTP it to your central repository of site information. You’ll have a complete run book in no time.

But before we leave config.sh, let’s take a look at some of the files it produces:

tunables.sum – As a performance specialist, it’s my job to understand how CPU, memory, networking and storage tunables are set in each of my customers' systems. Were I to generate all of the files I needed to do this, I'd have to run separate commands to list each of the VMO, SCHEDO, IOO, NO and NFSO settings, and then compile all of these lists into one file. Instead, PerfPMR does this automatically. The config.sh script generates a tunables.sum file in two parts. It first issues the appropriate tunable command (e.g., vmo) with the -Fa flag. This is useful for current tunable settings, but now we get to the detailed stuff. We have a list of each tunable setting with its current, default and boot time settings, as well as the acceptable range of values allowed for each tunable. These lists are generated in each tuning category with the -FL flags. I always put the -FL lists in my run books so I have every tunable setting at my fingertips should a system run into performance difficulties.

mem_details_dir – This directory contains a number of files that tell you how memory is used in your system. There are files written by the svmon command that provide detailed process, segment and user data. mem_details_dir also includes a file called memdetails.out, which tells you where every page of memory in your system is allocated, including kernel heap, file and text data. The contents of this directory are often all you’ll need to debug many memory issues, so be sure to refer to memdetails.out in any diagnostic undertaking.

instfix.out – This file lists every APAR that's applied in your system. It's very handy for troubleshooting AIX upgrade difficulties.

lssrad.out – This file lists your system's Scheduler Resource Allocation Domain identifiers (SRADs); these are logical groupings of CPUs and memory. lssrad.out is an essential starting point when tracking down poor thread affinity and orphan memory.

lsrset.out – This file lists your system's resource sets. RSETs let you build an “LPAR-within-an-LPAR” and isolate different workloads in your system. They also come in handy when you have workloads that compete with one another for CPU time and memory. Speaking of resource sets, you’ll need the information in the mempools.out and vmpools.out files that the config.sh script generates to build them effectively. (I recently devoted an article to RSETS. Read it here.)

https://www.ibmsystemsmag.com/aix/administrator/performance/Build_RSET/

proctree.out – This file lists the threads spawned by every process in your system. When used in conjunction with TPROF data, it's very useful for debugging problems with individual threads.

These are just some of the files generated by the config.sh script. I recommend paging through every file config.sh gives you. You might take an hour or so on each system, but it's time well spent. Look at it this way: Running individual commands to get all of this configuration data in one place would likely require far more effort than it would to simply dive into everything you find in config.sh.

As with the perfpmr.sh master script that runs PerfPMR as a whole, config.sh also has a number of flags that allow you to tailor its output according to your needs. Like perfpmr.sh, you can page through the config.sh file to see the available flags; they’re at the top of the file. For example, you may not want kdb information files generated. In this case, you might invoke the config.sh script in this manner:

???????????perfpmr.sh -x config.sh -k

Many other flags are available to omit different kinds of configuration data. As a general rule, I like all of it, but how much you grab is up to you.

Network Diagnostics

Now we’ll learn how to run some of the network scripts. PerfPMR bundles AIX diagnostic commands into convenient wrappers, so there's no need to wade through the man pages for each searching for the right flags (though they're still available if needed). So in no particular order, let’s have a look:

iptrace.sh – Most of my clients are surprised when I tell them AIX comes with a built-in packet sniffer, but iptrace has many of the functions of those hardware devices. This utility captures every packet that transits your network adapters and provides a complete breakdown of its character by displaying data like the packet’s size, window size, sequence number, source, destination, etc. Take caution though: iptrace even displays the ASCII contents of those packets, so only use it according to your corporate security rules.

Here’s how to invoke this powerful network utility:

???????????perfpmr.sh -x iptrace.sh 5

This syntax says to run iptrace for 5 seconds. When done, you’ll have a few files, one of which will be called iptrace.raw. Convert this binary file to ASCII with the ipreport utility like this:

???????????ipreport iptrace.raw > iptrace.formatted

Now page through the iptrace.formatted file. Looks a lot like the output from a packet-sniffer costing thousands of dollars, doesn’t it? The iptrace.sh script is essential if you want to capture all of the details of every packet that transits a network interface. You’ll need information like the sequence number for any group of packets when you’re tuning for the best TCP send/receive size, or each packet’s size when you’re determining if implementing Jumbo Frames will speed up your backups and restores. And yes, you can run the iptrace.sh script with the flags that are listed at the top of the iptrace.sh file.

tcpdump.sh – This utility captures all aspects of a network conversation between systems. Use the tcpdump.sh script when you just want to zero in on this data; any errors in these conversations will be readily apparent in your output. Run the tcpdump.sh script like this:

???????????perfpmr.sh -x tcpdump.sh 5

Again, append a number to the end of your syntax to tell tcpdump how long to run. After the script finishes, you’ll have several files, one of which will be called tcpdump.raw. tcpdump output also needs to be converted to ASCII, so format it with this command:

???????????tcpdump -venr tcpdump.raw > tcpdump.formatted

Paging through your tcpdump.formatted file, you’ll see the IP addresses of both your local and remote systems, the MAC address for each, and a breakdown of the conversation that happened between them. The first thing I do with tcpdump data is to search on the word “error.” You’ll be surprised at how often this simple search winds up pinpointing the cause of a network problem. And as usual, there are flags you can use with the tcpdump.sh script; they’re at the top of the file.

netstat.sh – This script invokes the netstat utility, applying many of its more common flags in the process. To get a high-level view of network activity in your system, instruct PerfPMR to zero out the statistics on your network adapters and then run the netstat.sh script for however many seconds you’d like. (Note: This network diagnostic utility is generally run for minutes, not seconds like iptrace.sh and tcpdump.sh.) Reset your stats and invoke the netstat.sh script like this:

???????????perfpmr.sh -x netstat.sh -r ; perfpmr.sh -x netstat.sh 120

You’ll be left with a file called netstat.int – the .int suffix is short for “interval.” (And do make sure it’s okay with management to reset those stats!)

Taken as a trio, the output from the iptrace.sh, tcpdump.sh and netstat.sh scripts give you a complete view of network activity in an AIX system. Concatenating the syntax of these scripts plus their formatting instructions makes the process a snap.

Next month we’ll look at PerfPMR's system profilers and tracing commands. Then to conclude the series, I'll tell you how to add your own scripts to a PerfPMR run and tune based on its output.