Teaching Troubleshooting Skills

Teaching Troubleshooting Skills – D M Goldstein, April 2023

I was recently asked to answer on a Support Slack forum where a participant asked how to train staff to troubleshoot a technical issue. There were several good responses, pointing out that it depends on your product. Training by observing others troubleshoot was recommended. Another mentioned that her team must master being able to “confidently answer all of the how-to questions,” and then move on to understanding why the module functions as it does, identifying what's intended vs not intended. Test environments are a must, and support agents should be encouraged to poke around and intentionally try to cause errors. Yet another said he had a game called “If, then what?” to encourage agents to “try all options.”

My response

This is a good question, and the previous comments are a good start.

For "Support 101", there are the basic questions (from my PRP article https://www.dhirubhai.net/pulse/problem-resolution-process-miles-goldstein/):

• What is happening or not happening?
• What is the expected behavior?
• What, if anything, was recently changed, and when?

My paper on Support Readiness covers a few training concerns (https://www.dhirubhai.net/pulse/support-readiness-miles-goldstein):

• Functions, features, and benefits: how would you describe to a novice what this feature does? Why would a customer need this feature? And how would a customer invoke this feature?
• Ecosystem overview: how does this feature fit into the mix? What are its upstream and downstream dependencies? What other participants (programs, functions, configuration parameters) are necessary for this feature to work end-to-end?
• Known design compromises: what short-cuts did we decide to take and why? What are the implications of those design decisions? What manual work will be required to maintain this?
• Troubleshooting: where did we hide the bodies? How do we collect logs or other evidence of this feature working and failing? What steps are we enabled (and expected) to take when troubleshooting this feature?
• Who you gonna call: what is the escalation path for this? Who are the trial customers, if any? Who else throughout the field is trained and enabled in this feature?

But to answer the larger question, "troubleshooting" is a skill that takes time to develop. Specific product knowledge is required as a baseline, and checklists (from Engineering?) help, but those run out as "how-to" morphs into "break/fix". It requires critical thinking skills, which not everybody has, and the ability to separate "signal" from "noise". That takes intelligence and experience. This is often accomplished by having senior folks teach the junior ones.

Take the problem, "My car won't start." You must understand how a car’s ignition works, which is technical training. But then you also must be able to think linearly through the problem and eliminate possibilities. That's the "critical thinking" part. (This example is for an internal-combustion gasoline-powered engine.)

• Is the key turning? Or, if it's a keyless fob, do I have the right one, does it have a good battery, and is it close enough to the sensor?
• Do I hear the engine cranking? If not, do any of the lights or gauges work? If not, then check the battery and/or try to jump start.
• If the battery seems OK, then do I have enough gas?
• Have I possibly flooded the engine? (More common on older cars than newer ones)
• and so on... (e.g. spark plugs, belts, obstructions, temperature, etc.)

From a software tech support standpoint, those questions might be: does the user have adequate permissions for what they are trying to do; is what they're trying to do even possible; did this ever work and, if so, what has changed; can I reproduce it; if I follow the logical flow of information from their desktop to the servers and back, where is the most likely point of failure that could cause this (app, database, network, etc.)? Being able to think this through is at the heart of troubleshooting.

When hiring "senior" staff, you can ask them experiential questions about tough problems and how they've solved them. Or you can have a technical task as part of the hiring process and have them walk you through how they solved it. For more junior staff, you might come up with some benign logic puzzle to see how they think. It doesn't have to be a technical one - just something to determine whether they have the innate skill to step out and look at the larger picture. The "nine dots" comes to mind, but there must be lots of other ones (ref: https://conferences.inf.ed.ac.uk/cogsci2001/pdf-files/0489.pdf).

So how can we teach this?

While writing this I saw a meme on Facebook (so we know we cannot trust the attribution) that “Education is not the learning of facts but the training of the mind to think. (Albert Einstein)” Regardless of whether the quote and attribution are authentic, I agree that one must learn to think. Ever since I was a child, I’ve believed that “knowledge” is the memorization of facts, “intelligence” is the ability to draw conclusions to accurately fill in the gaps between those facts, and “wisdom” is the ability to extrapolate beyond the facts and gaps to gain new understanding. For the purposes of this article, to troubleshoot one must at least have intelligence and the ability to think, taking the facts presented and filling in the gaps to determine the root cause of a problem.

In “Tales from Support” (https://www.dhirubhai.net/pulse/tales-from-support-miles-goldstein) I have a short section, “John teaches me critical thinking skills,” in which a coworker has me repeat how a particular utility works a couple of times until the light dawns on me that “read errors” and “write errors” are not the same thing. (I won’t go into it here; feel free to read my other articles.) That basic methodology works for teaching both children and adults. “Daddy, why does [choose a phenomenon] happen?” “Well, why do you think it happens?” You may then go on to explore the child’s theory and help them realize which parts make sense and which do not. It’s the same with adults. “Boss, why can’t I get this to work?” “Well, what have you tried so far, and what were your results?” It’s almost like a caricature of a psychiatrist asking, “What do you think it means?” By having them talk through the problem you help develop their thought processes and analytical skills. “If A equals B and B equals C then…?” “Oh! A must equal C!”

Of course, you must first provide enough technical knowledge of the situation for the other party to start their diagnostic journey. A five-year-old is unlikely to have the background necessary for my “car won’t start” example above, but somebody who has been driving for at least a couple of years should know the basics.

Overthinking it

One natural phenomenon I’ve observed is that people who are new to troubleshooting some problems tend to overthink them. I like to start with “Occam’s Razor”, which Wikipedia says is “sometimes inaccurately paraphrased” as, "The simplest explanation is usually the best one" (ref: https://en.wikipedia.org/wiki/Occam's_razor). If, for example, a door won’t open, then the most obvious problems are that it is either locked, blocked, or stuck. Those obvious problems should be investigated before going off on obscure tangents. Another problem is the previously mentioned “nine dots” challenge, where the participant assumes that there are rules or barriers which are not there. For the inoperable door, don’t assume that there is no other way into or out of the room; look for another door with access to get around the problematic one, and then you can learn more about why it won’t open. And don’t assume that somebody actually tried using a key you have; check whether it works.

That “second door” action brings up another point. Can you achieve the desired behavior through a different path? If so, what is it about the problematic path that prevents it from working? And is your presumption that they are yielding different results accurate? I had a friend in college who was completely baffled by something he observed. From outside the dormitory, he could look up to another friend’s window and see that the room was dark. When he went in and upstairs to the door, there was obviously a light behind it. But when he went back outside it was still dark. It took him a while to figure out that there was a light turned on; one that was focusing its light on the door. The room behind it appeared dark through the window, but from outside the door you could see the light. Mystery solved: both observed behaviors were correct and consistent, even though they appeared contradictory.

Finally: check the basics. Yes, it could sound rude to start by asking, “Is it plugged in? Is it turned on?” But without verifying the basics you may be assuming something is a much larger problem than it really is. My 11-year-old daughter is a master of Legos. My father-in-law had started a project, which she then picked up. At one point a section she built did not properly connect to a base he had built. She went back through the instructions looking at the section he built. It turns out that one set of pieces was basically backwards. She reset those and was able to easily move on to finish the project. I have had several customer situations like this - some basic item they had done was responsible for the problem they were observing; when I identified it the answer was, “Oh, that couldn’t be the problem,” but it was.

Where to go from here (aka “The Answer”)

To troubleshoot a problem, one must:

• have technical knowledge of the item being analyzed
• be able to visualize what should be happening from end to end
• have the ability to create a theory and test/verify it
• not be bound by false assumptions or perceived non-existent rules
• not be afraid to challenge assumptions and start with “the basics”

To get there, it is useful to have:

• a senior teammate walk them through their thought processes to hone them
• a lab environment where they can experiment by “breaking” things
• checklists or runbooks for common problems to guide them in collecting useful data
• documentation of expected behavior and (if relevant) product architecture or design docs
• time and an environment to observe, experiment, and learn
• an inquiring mind; you can’t diagnose many problems without curiosity

I’m going to sum this up by coining a new eXpression: “The Three eXes: eXposure, eXperimentation, and eXperience”. I hope this eXplains how to develop eXcellent trouble-shooting skills that can eXceed your eXpectations.