Waiting for internet access, and how to do better

This is a story of not-great customer experience, and it’s not over yet (but maybe today!)  Customer experience is our window into how things are going inside an organization. I’ll suggest that cognitive science gives us a new way to interpret what we see through that window, a way in which VPs, CIOs, and CEOs might think differently about computer systems and the business processes they support. The ideas are simple, but not at all easy. Eventually, this could lead to a better customer experience. Surprisingly, this could also save a lot of money.

My wife and I just moved to Tehachapi, a couple hours north of LA — let’s call that address B. A few weeks ago, at our previous place, address A, I called AT&T to get new internet access and a land line set up, so that when we arrived at address B, we’d be ready to go.  All of the sales and service folks I interacted with were decent, polite, and hard-working, and I very much appreciated their efforts. This really isn’t a story about the people on the front lines. In the end, it’s really a story for executives at most large organizations, not just at AT&T.   

We arrived at address B Friday night. Today is Tuesday, and I’m still using a personal hot spot with a weak cellular signal to get intermittent, slow internet access. Here’s what has happened so far.   

To get the new service for B, I had to open a new account, although I’ve been an AT&T customer for more than 30 years. We have maybe three or four other accounts with AT&T, and they have nothing to do with each other. I’m not aware of a way to consolidate them to one account.

To get the new account, AT&T had to check my credit, although I’ve been paying my bill automatically for maybe 15 years, on each of the three or four other accounts we have with AT&T.

So that anyone can access my credit, I have to unfreeze my credit with the credit bureau (this process isn’t a great customer experience either, but that would be another story). To minimize that effort, I asked which credit bureau would be used, but the person handling my request couldn’t tell me, since that’s determined randomly, so I had to unfreeze my accounts with all three of the credit bureaus and call back in.

When I called back in, I had to start from the beginning, rather than where we left off.

I often had to enter an account number to the automated system, but then I had to repeat it and maybe a pin number to whichever person I spoke to. This seemed to happen whenever I got transferred to another person, which happened often.

If we got disconnected, at least some of the time I had to start over, rather than being called back. If I was on hold, often the phone was silent, so I wasn’t sure whether something was still in progress.

Whenever I called in, I got a voice recognition system, which tried to narrow down my request to something that could be handled automatically. The system often didn’t recognize what I was saying, particularly if I was using airpods instead of speaking directly into the phone. There often aren’t numeric options for the menu items so that I could choose something by typing a number, rather than making multiple attempts to be understood. 

My general sense is that it takes one or two more attempts to get to a human than I think it should, which makes me feel impatient.

About ten days ago I got an email confirming that my equipment was to be sent, which was good, but it didn’t say whether it was going to be sent to address A or address B. I called to verify that it was going to address B.

The tracking number for UPS wasn’t valid yet, perhaps because the equipment hadn’t yet been given to UPS, so UPS couldn’t tell me. I would need to get the information from AT&T’s systems.

I had to talk to four people to get an answer, as the various systems used by the customer service folks didn’t have the information. After hunting for a long time, the last person finally confirmed that the equipment was in fact going to address B. I was relieved.

Saturday morning, my first morning at address B, I looked for the equipment. Not finding it, I tried the UPS tracking number again, which worked this time, and confirmed that the equipment was at address A.

Before I called service, I decided to check my phone line, and plugged my phone into all the jacks in the house. All were silent, but I don’t have a filter for the digital signal, so wasn’t sure what to expect.

I called in about the equipment delivery, and the person confirmed the error, and explained that she would transfer me to sales to correct the problem. The sales person then reconfirmed the problem.

I also asked about the phone line, and she said that silence is what would be expected, but that the line wasn’t live anyway, because the original visit earlier in the week couldn’t be completed without inside access while the owner was present. Although I had originally been told the line could be turned on without inside access, there was apparently a problem in this case.

I had not been notified that there was a problem with the first visit, nor that I would need to schedule another visit. I was told that a note should have been left at the door, but I hadn’t seen one (although movers had been here, so the note might have been lost). I asked why there was not also a notification sent to my email, since the whole point of the plan was to get access ready before we could arrive to read a note. I was told that the physical note is the only notification.

I was told that we could solve both problems at the same time, which was good news. When the technician returned to fix the line, they could also bring the equipment.  

The only remaining problem was that the return visit could not be scheduled right then, because someone else was currently updating my record, and the record was locked against any other updates. That someone was neither of the two people I had talked to so far on the current call and my previous calls were many days in the past. I asked to speak to a supervisor.

Before I was transferred to the supervisor, I asked if there was someone I could speak to about the whole experience I’ve had. I was given a phone number. When I tried it later, I was told it’s no longer in service.

The supervisor explained in more detail what she was attempting to do, which involved sending messages to the person who was in the record, and one or two people in that person’s management chain, none of whom responded (it was Saturday, so that’s not too surprising). There is apparently no way to override the lock on my record, so the supervisor had to just keep trying. She called me back twice to give me an update, and volunteered to keep trying after her shift.

The next morning, Sunday, on her next shift, she called me back to say the record was open, and she had scheduled the visit for the next open slot, on Tuesday (today), between 8AM to 6PM. The local office apparently can’t give more narrow slots. 

I told her how much I appreciated her efforts to go the extra mile.

At the end of many calls to AT&T, the person will sign off by saying something like “Thank you for being the most important part of what we do.” That expresses a good sentiment, but it isn’t matched by what actually happens.

Now let’s talk about what I believe are the underlying problems and opportunities here. This example is for AT&T, but I think the situation is similar at many large organizations.

For AT&T, none of this might be thought to be very important, since the real money is the $200+ that my family’s wireless bill runs per month, plus other smaller bills for our other accounts. The thought could be that I might be annoyed, but that I won’t bother shifting everything to a new carrier. That has been true — I haven’t bothered — so far.

In suggesting how to get to a better customer experience, it might seem that we’re talking about spending more money than the way we are currently doing things. I suggest we could spend less money, not even counting the loss from frustrated customers.

I’ve worked in computer systems, mostly for big organizations, for almost 40 years. I recently finished a doctorate in mathematical behavioral science.

The problem that started me on the path to that PhD is why big organizations have such consistent problems writing big computer systems.

In the past twenty years there has been increasing interest in “agile” methods for writing software systems. By now, folks have been attempting to be agile long enough that we have come to see that results are uneven. Sometimes little seems to change.

I suggest that an important missing element is to better understand where we’re starting from, rather than just focusing on the new thing we’re moving to. Rather than go into the cognitive science behind that suggestion, I’ll just dive into specifics.

A common metaphor is that a software system is a building. Under the Building frame, we expect to spend a lot of time designing the building (“architecture”) before we stick a shovel in the ground (“typing code”).

It takes a lot of bricks and concrete and steel to build a building, and one brick ("line of code") is a lot like another.

We also expect to spend a great deal of money building the building ("capital expense"), and a much smaller amount of money in each subsequent year maintaining the building ("operating expense").

Metaphors and frames are tools of our thinking. Like any tool, they aren’t right or wrong, they are just more or less useful. The Building frame is very useful when we are building a physical building. But I suggest that it is not very useful when we are writing software.

When we are writing software, we should consciously set the Building frame aside, and use other tools.

When building a real building, we are talking about physics and fixed structure — the foundation and framework are rigid. If we build a four story building, and later decide that the third floor is not serving our purposes, we can’t just take it out and put in a new third floor.

But with software, we can replace that third floor (layer), if we’re careful. The point of software is that it is changeable.

Also, a line of code is not a brick. We can’t just replace one line of code with another random line of code; the system will break.

Now, we tell ourselves, “I know that software isn’t *really* a building — that’s just a metaphor.” For the executives who are reading this (and any of us with gray hair) — while you have been advancing in your careers, the cognitive science has also moved.

Over the past few decades we have come to understand that metaphors and frames are deeply held in our brains, and they affect our thinking in ways we are not always conscious of.

It takes real effort to bring frames and metaphors to the surface, and make them conscious tools of our thinking. In particular, it takes a lot of work to set them aside when they are not serving us well.

Another metaphor is that building a system is running a project. The Project frame is concerned with budgets, schedules, resources, and tasks.

The Project frame is essential in a large organization — it’s the best way we know of to make changes in how an organization works.

But the Project frame doesn’t serve us as well when the problem is writing and modifying software.

We end up having to make a business case to write or change the software. A business case requires a return on investment. Making the business case is itself a sizable effort, and tends to encourage that changes are bunched together as a package.

An easy way to think of such a bundle of changes is as a new system, as a new Building. The Building and Project frames reinforce one another, such that we get relatively infrequent, big, expensive, tightly supervised projects.

In between, we have long periods where the only changes are those necessary to “keep the lights on” (Building frame thinking again).

The last metaphor is that writing software is production. This metaphor comes from economics, and introduces the More frame. More is more — more inputs gives us more outputs.

We each build the More frame as infants, even before we start talking — as water pours into the glass, we see the glass fill up. As sand pours onto the pile, we see the pile get bigger.

So without thinking much about it, we believe that if we spend more money and add more people, we’ll get more working software.

The More frame is useful across a wide variety of domains, but it is not useful everywhere. If we’re talking about a structured system, the More frame often doesn’t work. Consider a mosquito — a structured system.

Suppose we were making a 1950s B monster movie about an experiment gone bad, with mosquitos the size of dogs — wouldn’t that be scary? Well, only in the movies.

In the real world, a mosquito the size of a dog couldn’t fly. It probably couldn’t even breathe, because an insect’s respiratory system couldn’t get oxygen to all that volume. In this case, the More frame leads us to a failed system.

The More frame also works poorly when applied to the structured system of writing software. Each line of code is not just another brick in the wall of a building. Each line has to work with all the other lines, and each line does a different thing — that’s why it’s there.

Over the decades, what we’ve observed over and over again is that good big systems start from good small systems — “two guys in a garage”. The small system takes time — years — to build.

Only very gradually does the system get to a size where we can start to add more people productively. When we blow the mosquito up in weeks or months, it either fails outright, or is weak and fragile. By fragile, we mean that it’s hard to change safely. Systems that are hard to change safely end up changing very slowly, if at all. We think “if it ain’t broke, don’t fix it.” That thinking goes along with the Building frame, where we’re stuck with the steel and concrete we laid down decades ago.

After a number of years, in frustration, we toss the entire system and rewrite it, as another Building. It has a pretty blueprint and graphics, and promises new features. But it’s still a Building.

Because we are desperate to move away from the problems of the old system, and because we’re a big organization, we throw money and people at it (More), supported by lots of people supervising (Project).

We get another weak and fragile system, if we get anything at all (sometimes we just fail). This is by far the most expensive, disruptive, and unsatisfactory way to write software, and it starts with thinking about software using the Building, Project, and More (BPM) frames. 

To do better, I suggest we don’t need the next flashy technology, the next cloud or micro service. It might sound odd to hear that from someone who has spent decades in tech. But the underlying problems are not technical.

It’s that the BPM frames are very blunt tools when applied to writing software. They get results, the way a hammer can drive a screw into wood. It will work, sort of, but the results will be poor — expensive, slow, and fragile.

A more useful alternative metaphor is that a system is a product. Under the Product frame, we are constantly updating the software to respond to market demands and internal pressures. The product is never finished, so making it simple to change safely is central to what we do. The Product frame matches better to the inherent attributes of software — with effort and discipline, software can be changed frequently, at low risk and low cost.

Another crucial frame is the Software frame — a distillation of all we’ve learned about writing software in very small increments, released into production in days or hours, where all processes are automated — infrastructure, testing, deployment, and backout.

This thread is addressed to executives of organizations who aren’t Google, Microsoft, Apple, Facebook, Amazon. Those organizations were started and are led by people who have absorbed the Software frame through their fingertips, after years at a keyboard.

For everyone else, the BPM frames are the default way of thinking about our business. Putting these frames aside when the problem is writing software is a simple idea, but it is not easy. The BPM frames permeate our thinking.

To understand the challenge, consider the Christmas tree. Decorated evergreen trees only became associated with Christmas in the last few hundred years, in northern Europe, far from Jerusalem. These days, it is difficult to think of Christmas without a tree coming to mind.

A Christmas tree is a conceptual blend — it brings two or more ideas together through some common association, regardless of their logical relationship. We do this all the time, often without being conscious of it.

A popular blend when thinking about software these days is the idea of “scale”. Scale is a blend of the Software and More frames — it suggests that a useful and important goal of a big organization is doing everything that a small organization can do, but just more so.

Scale (the More frame) is important in some areas of IT, for example, when we are trying to ramp up to handle more volume, more customers. But writing software doesn’t scale, any more than mosquitos scale.

It is very hard to let go of the BPM frames, of the idea that a big organization should be able to do anything a smaller organization can do, only more so.

The BPM frames aren’t just ideas. They are embedded in our compensation systems, our accounting systems, and the minds of our stakeholders and boards of directors. Letting go of the BPM frames for writing software is a simple thought, but is very difficult to accomplish.

I suggest that the BPM frames are part of what leads to the kind of customer experience in the first part of this story. 

I have no doubt that within AT&T there are many people who have wanted to fix each of the problems I described. Few of these are likely to be major changes, but even small changes can be overwhelming to accomplish when our only tools for thinking about them are the BPM frames, and the fragile systems produced under the BPM frames.

It’s possible to do better, and save money at the same time. But doing better starts in the minds of our executives. It starts with a willingness to face the BPM frames, and then deliberately put them aside, when the problem is writing software.