The Future of Software: Hard Lessons Learned

I've had the distinct privilege in my career to work with some of the best, brightest, and most talented people in the world. I have had countless mentors guiding me down tremendously productive and successful paths. But I don't recall any of them summarizing and educating me on the lessons they learned. I had to learn most of it the hard way, by making mistakes and correcting them as quickly as possible, or by taking risks and sweating the consequences if I were wrong. So today I'll try to summarize the 3 biggest technology lessons I've learned in my career. I hope that is valuable to you in your future software development pursuits!

Hard Lessons Learned:

1. Not much of what you see labeled as new technology is truly new
2. Don't solve one problem at a time
3. Standardize before you automate

Lesson 1: Not much of what you see labeled as new technology is truly new

Taken from a pure technology perspective (not a sales or marketing perspective), when you analyze a new technology or product, I think you will find very few radical departures from what you have seen before. There absolutely are game changing technologies released from time to time, and it is important for you to understand the difference between a game changer and an evolutionary improvement. However, I'd strongly recommend your first approach to analyzing any new technology is to examine it as an evolution of something that is already in the marketplace. Almost every technological advance is some incremental step over what has been already developed and pretty well understood. The best sales and marketing people in the world will do everything in their power to make it look otherwise, and that's why they are so good at their jobs. But don't be fooled by the hype, analyze everything new and different against the base of what you already know.

This hard lesson has served me very well in my career for several reasons:

• Don't be afraid of new technology. Most of it isn't actually new, it is a tweak of something you already know.
• Don't jump rapidly into new technologies. If you have an existing technology that already does most of what the new technology does, you are probably better served using your existing technology which has proven itself in production. New technologies are not likely to be production proven, at least not at the scale and service level required in your business. And of course switching technologies involves effort and often quite a bit of pain. Also remember you probably aren't the only one that needs to understand and maintain this new technology, you have other team members to consider as well as your operational and technical support staff.
• Don't underestimate your ability to quickly learn this new technology when required. In order to more quickly learn the technology, look for the similarities and differences as compared with technologies you already know. If you take this approach, you will find your learning curve is greatly accelerated.

Lesson 2: Don't solve one problem at a time

This seems completely counter intuitive to what you have learned going back to your first computer programming class. But it is a powerful accelerant to your development process when applied correctly. The best products in the world, the most successful products in the world, name any you can think of, don't solve just one problem. They solve many problems. And here is the key to making them special: they solve those problems in concert. They solve the problems so well that the user often doesn't even know those problems existed. The key proof point here is how the Apple iPhone changed the market by solving a multitude of very difficult problems all at once.

When developing a new technology or product, look for opportunities to solve many problems with one comprehensive architecture and design. Look for opportunities to hide the existence of problems from your users. Their existence will be a mere bullet on a marketing slide where this and that problem is automatically solved (and imply that if you use someone else's product, you have to deal with those problems).

Now the difficult part of en mass problem solving (besides the massive headaches you may have thinking through everything) is translating your system architecture and design into an implementation. Implementation is typically done by less experienced people and therefore done in a manner where they solve one problem at a time. Implementation work is also typically executed according to a somewhat linear project implementation schedule. And I won't challenge that approach as it is so well known and easy to manage. However, you do need to watch out for the side effects of that approach.

You know your team is not applying this hard lesson learned when they get into the habit of exposing technology problems all the way back to the users or operational staff running your software. Challenge them every single time you see this proposed. Here are some of the more frequent issues I have come across:

• "I need a configuration file/database/... parameter to configure this or that." Challenge them, why do you need that? The answer is almost always because it will be easier or faster for them to code the system if they defer the problem to the user or operator. Resist the temptation to code the system faster by pushing problems to users/operators. Look for alternative designs which will automatically resolve the problem without user or operator effort. If that just can't happen, at least look for designs which will require the system to be configured one time, up front, and not continually adjusted going forward.
• "I need a new user interface screen for the user or operator to configure this or that." Challenge them with the issue that the system won't function properly if the user/operator doesn't perfectly set that configuration and maintain that configuration going forward. People make mistakes, they have limited time to be trained on your system, and they have limited time to log into the system and maintain the configuration. When they do not do so perfectly, which they will not, the system will fail. Examine alternative designs which will solve the problem without pushing it to the user/operator and introducing significant failure points.
• "I need a new database technology. Our existing database has this or that limitation." Challenge them with the issue that introducing additional database technologies will also introduce data consistency issues and operational issues. If your databases aren't in perfect sync, with perfect availability and performance, your system will fail. Examine alternative designs which will solve the problem within the limits of your existing technology. If you do not do so, you are pushing the problem to the user/operator to maintain not only new technologies but also debug and resolve data consistency and performance issues across multiple databases of data.

You as architect have the ultimate responsibility here. You must think about the big picture, the technologies you must deploy and integrate to create a useful product, the problems that get introduced along the way, what problems you will expose and those you will automatically resolve. Pay great attention to this up front and watch for the problem deferral warning signs as your team implements your system. Everyone has implementation constraints, you won't be able to solve every problem. Pick your problems well, prioritize them, iterate on them over time, and you will have a truly great product. One particularly effective way to prioritize problems is to calculate the multiplier. If I don't resolve this problem automatically, how many times will my user or operator have to resolve it manually? The higher the multiplier, the quicker you should tackle that problem.

Lesson 3: Standardize before you automate

I put this lesson last because if you apply lessons 1 and 2, lesson 3 will be much easier to implement. This is the single most pervasive mistake I have seen in every recent technology organization I have worked for. And it is actually made worse by the common practice to hire Site Reliability Engineers (SREs) in an IT shop instead of traditional operational roles. The focus of an SRE is automation, that's a majority of their job description. In order to automate, they naturally use the tools and skills they learned in previous jobs. And unfortunately most (but not all) do so without regard to what the tools the rest of the organization uses. The organization quickly becomes fragmented and every SRE uses different tools. I have seen the exact same operational automation problem solved 5 different ways with 5 different tools by 5 different people, in the same organization.

Now I want to be clear here, I have several friends that are SREs and all of them are incredible at what they do. I am a big advocate of SREs, I just want to harness their considerable skill in a way that optimizes their talents for the entire organization!

First of all standardize on a tool to solve a particular problem across your entire organization. My goal everywhere I go is to set one organizational standard for every technology in use by the company. Never have 2 or more ways of solving the exact same problem, designate a standard. Now that doesn't mean you turn off the non-standard technology immediately. That will take time, planning, and most important of all, buy-in from those making the switch. But you do need to work in that direction.

Next, make sure you apply lesson 1 to get people comfortable with that standard. Some people don't like change, they fear it. But there is no reason to fear changing technologies when you apply lesson 1 above.

Most important of all, to achieve buy-in to make that transition, make sure you devise with a super-set solution in the standardized technology. In the case above where we have 5 different solutions to the same problem, analyze all 5 solutions. Get all 5 people that created those different solutions involved in the analysis. You will find some solutions solve this particular area of the problem better than others and conversely don't address some problem areas as well as others. Take the best of every one of the 5 approaches and merge it into the one solution that runs on top of your standardized technology. In the end you have many benefits with one effort (that's lesson 2 applied!) as follows:

• Those unfamiliar with the standardized technology get exposure to it as they analyze all the solutions. As they become familiar, they lose that fear of change.
• They gain recognition from their peers as to the parts of their solution that are superior, which is always a good feeling. Conversely, they gain respect for their peers on the parts of their solutions that were superior.
• They get the opportunity to use the new standard technology to implement the portion of their solution that was superior. That experience will be critical when the organization has time to switch from the non-standard technology to the new standard.
• The new solution in the standard technology is the best solution in the entire company because it is a super set of the best parts of all the other solutions. The organization will benefit from the best solution in all parts of the business as it is rolled out.
• Going forward the super set solution will be 5 times cheaper to maintain as you eliminate the non-standard solutions.

A couple of final points on this lesson learned. Resist the temptation to let people use whatever tools they want to use, as long as they get the job done. I have heard this many times in my career and I just don't think it gets you where you need to be as an organization. All 5 solutions to the same problem above had different issues. If we lost one of the 5 people that created one of the solutions, we lost their expertise in the tool and their knowledge of what the issues and limitations are.

The worst part by far of taking this path is that you have many people working on simple problems. Because there is no leverage between the different tools and the different solutions, 5 people are at work creating and maintaining 5 solutions. They won't have time to address the more complex problems of the organization, they will always be working on the simple problems. If you standardize, then automate, your solutions will be much more robust. You will save a great deal of effort when the super set solution solves more problems and solves them with better quality. And then the magic happens; I love this part; I live to come to work each day to see this happen! The leverage you gain with standardization and automation will free up time for your people to tackle more problems. And those problems will always be the more difficult problems to solve (and therefore the most costly problems). Very few people will start off an automation effort solving the most difficult problem of all. They start with the simplest problems, get a win from solving those simple problems, and then move on to the next more difficult problem, and so on. When organizations do not standardize, they have no time to tackle difficult problems which tend to be the most beneficial problems to solve. They have just enough time on their hands to solve simple problems which usually does not result in much cost savings and efficiency.

Conclusion

The last thing I would want to do taking this approach to software development and operation is to use it as cause to reduce staff. Instead, use it as a way to free up your tremendously talented people to tackle more and more difficult problems. That's fundamentally what customers want, they want you to wipe away their problems. The more problems you address, especially the more difficult problems you address, the more likely they will buy your product! And applying lesson 2, you also solve another very significant problem. Every talented person I know wants to solve more and more difficult problems. Taking this approach you keep your talent challenged and you retain that talent for a longer period of time (the topic of my next post).

When all 3 hard lessons learned are applied as a whole, they unleash the power of your people to create technology, solutions, and services never seen before. I hope this works as well for you as it has for me, and I look forward to you posting your comments and results!