Software and Saxophones: you can never go home again

This morning I was banging away at some code documentation on the train, listening to an instant mix provided by my old friend, Google Play. I love that instant mix feature, because it does a pretty good job of picking out songs similar to the one that kicked off the mix, but more importantly - gawd bless it! - it regularly reintroduces me to songs in my library that I've almost totally forgotten about.

That golden age of saxophone solos

And so it went, when on the train this morning, tuning it all out so I could get my coding documentation done, that particular favourite from my youth came on. INXS, Listen Like Theives.

Everybody down on your knees / Listen like thieves for the eeeeeend sounds...

Man! It had been a while since I heard that one! I found myself really groovin' to it. Well, to the extent that "grooving" is possible for a middle aged white guy on a crowded commuter train. So, safe to say, the movement was mostly on the inside, I think. Were my neighbours shooting me looks?

Anyway, then came THAT part of the song. You know it. The part where the guitar is so unsubtly added on top of the baseline melody...

You've got it all / You've got it all / It's all in your... / haaaaaaaaands

Then, oh yes, then came that most magical of things. That mythical beast now lost to the ages. That creature that stirred the hearts of teenagers of the 90s everywhere. The saxophone solo.

Ahhhhhhhhhhh. Suddenly I was 14 again.

Smash cut to work

Well you've got to wake up sometime, right? No matter how good the dream is.

So I get to work and I start to think about some conversations that I've been having with colleagues lately. We've been reminiscing fondly about the old days, when we were coming up in banking tech around about Y2K, dot com, the bubble bursting, and all that.

Things were different then. And I'm not just talking about baggier pants and blackberries (my first work phones didn't even do email). All of us who were young and getting started in enterprise technology, every one of us, remembers being far more productive back then. Whether it was driving a new online banking site to market, coding up XSLT for newfangled SOAP messages, or trouble-shooting production problems, we just seemed to be able to get more done then. Like, A LOT more, right?

That was awesome. Like saxophone solos in rock songs.

30 Rock quotation courtesy of whatthewhat.tv

What was so different back in the day?

So what gives? What was so different 15 or 20 years ago? Or am I just turning in to that curmudgeonly old guy who misremembers the "good old days" - it's understandable, the mind does start to go in the autumn years - and talks about stuff like walking up hill to school, both ways, in blizzards, from September to June?

Could be. But I've been doing some digging. And reading some things. And I think there is more to it than that.

Teams back then: I remember two 

One thing that I remember from when I started is the team structure we had. I was on the development side of the house. We were responsible for coding apps (or taking COTS apps and customizing them a bit), and testing them. When we had finished with that, we'd hand the apps over to the operations side of the house. We'd give those guys documentation about the app, including deployment procedures, they'd make up a run book with our help, then we'd practice deployment together (roll forward and roll back), and finally, in the middle of the night, normally on a weekend, we'd sit down together in front of production consoles in the data centre to do the change.

The change involved the ops guy executing the deployment, and I'd sit there watching. If something went wrong, we'd trouble-shoot together - his hands on keyboard and mine safely in my pockets - and hopefully complete the change. If not, we'd roll back, like we practiced, and start planning for trying again later.

It was all pretty manual. Not totally, we automated things like builds. And we scripted the deployment process as much as we could. But still, there was little sophistication.

Yet our changes didn't cost nearly as much as changes made today - neither in terms of effort, nor dollars and cents. And we did more of them. 

Teams today: minimum four, maximum... n, I guess

Having formed a new team for API Architecture recently, I found myself working on an engagement model for the people who need our support. These people include Solutions Architects, who need to draw blueprints for systems that include APIs, Designers and Engineers in delivery organizations who need to build the APIs, and others who just need to understand what we're all up to. I considered developing an intake process. There'd be a form for people to fill out who want time with the team. Maybe it could be online on the company Intranet to make it super easy for people to use. Great idea. I congratulated myself on good corporate thinking.

Then I mentioned it to someone who builds software, and the look I got could have melted stone. My interlocutor sighed heavily, pleading "NOT another intake! Please!"

What this kind person patiently explained to me is that executing a project takes a minimum of 4 intakes, and depending on what is involved, that figure can go into the double digits. Wow.

Why so many intakes?!

This is what I've discovered. In the years of this new millennium, between the heady days of dot com booms and busts and the even more heady days of blockchain and FinTech unicorns we find ourselves in today, we've organized ourselves within the enterprise into independent consultancies of specialists, each of which is supposed to do a piece of the work that it takes to deliver code into production.

Here's what I mean. There is a team that does Development and Unit Testing, a team that does Quality Assurance, a team that does Production Deployment, a team that does Operations, a team that does Risk Assessment, a team that does Security, a team that does Business Analysis, a team that does Database Administration, a team that does... I'm tired now. Give me a second...

But you get my point. There are a lot of specialized teams around. And each one is expected to lend resources to projects to do what they each do very well, thus ensuring that the project is completed in the most expert, efficient manner possible. That sentence is not supposed to sound sarcastic. It's not meant that way. On the face of it, the model really does seem to make sense. If everyone is an expert at what they are doing, it only stands to reason that the project should execute at the highest efficiency and the lowest cost.

But that's not what's happening. Why?

The myth of efficiency through specialization

Turns out that there is a lot of overhead introduced by team specialization. We talked about one type already.

Intake, Estimation, and Change Requests

As independent consultancies, each team in the organization is called upon to allocate their resources to projects. But how much to each? We've got to estimate that. That's a process. To kick off that process, fill out this form. 

Once you've got your estimate, here's your resource. Have you discovered that the scope on that original form has changed, or maybe wasn't right in the first place? No problem, there's another form for that. A Change Request. Fill that out and we'll re-estimate effort.

Effort to align

Whenever someone is brought onto a project, there is effort that is expended bringing them up to speed. No matter how good your intake documentation, you are going to have to have a meeting between Subject Matter Experts (SMEs) to "align", the new way of saying "get on the same page", which I've got to admit sounds better. Such meetings are scheduled according to availability of the people involved. Between teams that specialize and support multiple projects concurrently, this is normally a formal arrangement, booked in people's calendars, and executed around a boardroom table.

Context-Switching

Each individual person involved in an organization where teams specialize and interact as in house consultancies is normally expected to support a number of projects concurrently. Time is split between the projects. Often, days are split between them. Especially when the switch occurs during a day, the resource affected needs time to reorient from the first project to the second. To borrow a metaphor from computing, time is required for that resource to bury the first project in memory and disinter the second project from there too. Depending on how old you get, like me, that process can be less like memory operations and more like disk I/O. And I'm not talking about solid state here people. There is going to be time needed to move the read-write head around the spinning media people. Count it.

Amplification

At this point you might stop me and say that it is not very much effort to fill out forms and have a meeting or two. Maybe not. But if you have between four and double-digit intakes to do, each potentially requiring more forms for changes, you are into the territory where you might have to make a new role for intake management. And that's exactly what happens.

Now think about what it takes to get everyone up to speed. You either have to repeat the same conversation numerous times, or you have to wait until everyone is available to have the conversation. This latter tactic is often employed, but it's made less effective by the fact that each person in the room has been context-switching all day, so often those conversations have to be repeated as well.

I firmly believe now that there is not a linear relationship between the number of teams involved in a project and the effort exerted to deliver the solution. That relationship is geometric, baby. I feel it in my bones.

How do we get our groove back?

You didn't think I could bring the conversation back to saxophones, did you? Or at least, that would be in bad taste. Well taste be damned!

So I remember the good old days when saxophones were allowed in rock songs. And I also remember the good old days when delivering software solutions was relatively faster and less expensive. Can either of those pieces of nostalgia be recaptured by this poor, graying Gen Xer?

YES!

And no.

Let's not fool ourselves. INXS isn't coming back, that post-Rockstar interregnum notwithstanding. So although I have no doubt that someone, somewhere will someday get the stones to bring back saxophones in popular music, and in a big way - how can that not happen?! - it will be with a new sound. One that the kids like, not me. And they can get off my lawn, thank you. Anyway, those new solos will never have quite the same vibe that I remember from the 90s. I know it.

Similarly, we're never going back to those days when I got a call from my end users and I could dial the pager of my ops guy to take a quick peek at things, maybe recycle a server or two to clear a memory leak. Those days are never coming back. Today, we don't even have servers, per se. We've got clouds. Ops guys don't sit around on raised floors anymore waiting for calls from me, powering hardware up and down with toggle switches. No one carries pagers anymore...

BUT my organization IS investing in Agile. And in a big way. We're retraining EVERYONE to do things in new ways. From the top on down, there is recognition that we need to get to some serious work to continue to deliver solutions that our customers need and want. Yesterday's ways won't cut it anymore.

And I'm all in on that. I've metaphorically pushed all my chips in the middle on that one. It's just my humble opinion that part of that change will, ultimately, involve changing how we team together too. I've already seen some big moves in the right direction and I've only been here three months.

So the new software development world might not look and feel much like I remember it from the good old days. Indeed, I can never go home to it. But will the new world be fast? You bet. Will we get things done? Definitely.

Can't wait. I feel like we're so close. As the bards sang:

You've got it all / You've got it all / It's all in your... / haaaaaaaaands

Cue the saxophone solo.

Epilogue

Who is this guy? Since when to LinkedIn posts have epilogues?

Cut me some slack. This is my longest post, like, ever I think. On LinkedIn or anywhere. Novels get epilogues sometimes.

Really, this is just for the boring stuff. Footnotes.

Remember I said that I read some stuff? Truth is, I didn't come up with all any of this on my own. Here are some of the things I looked at that got me thinking about team structure and how it affects how my colleagues and I do our jobs.

*The Mythical Man Month, Frederick Brooks, ISBN: 0201835959

*How Committees Invent, Melvin Conway, http://www.melconway.com/research/committees.html  

* A talk by Hank Kolk, who spoke at Dockercon in 2014 about ING's adoption of DevOps.

* Dunbar's Number and Monkeyspheres. 

Thanks for reading. If you got this far, include a comment with a comment mentioning one of the main characters of Neal Stephenson's Cryptonomicon to prove it, and you'll make my day. My personal favourite is Bobby Shaftoe.