Part 4: Sustainability

Previous article: Part 3: The Product

So far, I've covered the first principles (Part 1 ), the people (Part 2 ), and the product (Part 3 ). Now we need to talk about building sustainability into your development organization (and, thus, your company). There's a lot of "it depends" when it comes to the right way to do all this. In order to navigate through this fog, you need people in your organization that have been there, done that. These seasoned professionals have seen it all go horribly wrong a few times and have also seen it go terribly right and can guide your organization in real-time to avoid the pitfalls and hit pay dirt.

I want to highlight a few things in this section. First, automation is key to building a sustainable product. Embrace build automation, test automation, deployment automation, and production recovery automation. Embrace it early. Invest in both the creation and maintenance of automation. It might seem expensive at times, but the alternative is far more costly. Trust me, I've been there and done that.

Second, continuous processes are critical to responding rapidly to changing priorities. Your priorities will change on you whether you know it or not, so you'd better be able to keep up. Large, monolithic stage-gate processes will not work well nearly any scenario. However, there are ways to keep these processes nimble and Lean, so you need to make this happen wherever possible.

Lastly, in a few months or year, do not be tempted to throw out a bunch of code and start over just because the product has become legacy. Rebuilding what you already spent a lot of time and money on is very, very rarely the best thing to do. Architect your product to have well-defined internal borders, strictly adhere to this approach, and incrementally build portions to keep your bits fresh (but only where it's the actually necessary).

Automation

Some products start with a single developer working in their spare time on something that will go on to become a massive hit (Minecraft). On the other end of the spectrum are products that start out with thousands of people and billions of dollars (Metaverse). I don't know where your company and product are on this spectrum, but you need to be using automation as much as possible.

To guide what should be automated, I say, "Let computers do what they're good at so the humans can focus on things they're good at." Don't make the humans do repetitive, rote, stepwise processes because they're just not built for this type of work. Eventually they'll forget a step, get burned out, or just get feisty. On the other hand, computers live for this monotony. If you can automate something, the computer will do it exactly the same time a thousand times a day or more and will never complain.

Automate your build pipeline. Do all your static code analysis (linting and cybersecurity) as part of the build. Use simulated sources and sinks for each architectural border (API, query, file transfer, enqueue/dequeue, etc.). Build containers as your artifacts and use deployment automation for VMs that are created and destroyed through infrastructure as code. Automate your production tests (synthetics, etc.) and use this as part of your observability platform for alerts.

Automation is your key to reducing entropy - all this automation also means that your development environment can very closely mirror your production environment (and vice versa). If everyone has the same environment (at different scales) everywhere, the consistent environment will prevent failures due to misconfiguration as it moves from one stage to another. It will also accelerate bug reproduction in development and save huge amounts of time when you need to fix something that no one found until it got deployed to production. It will also drastically decrease your variability from one environment to the next so you can spend time building products for customers instead of troubleshooting something that works on one system but not another.

Done well, automation is inherently Lean because the cost to do things with an automated process is (simplistically speaking) a one-time expense. Without automation, every time you wanted to do the thing, you'd have to pay a human to do it. This doesn't scale.

Continuous Processes

I've worked in many different development organizations and each had their own requirements for how to manage software development. While it's true that trends have moved us from the "old days" of waterfall (Gantt) development to new and shiny continuous methodologies, it's also true that a company typically needs to blend these approaches together. This is usually because the external requirements of the business (e.g., market expectations for how a company delivers a product) dictate a specific process that would be very inefficient if used for the product development operation as well. In these cases, the interface between product development and product deployment (e.g., to production) can be built to support the external requirements. Product Management and Release Management usually execute this interface.

Internally, there are typically three or four layers of decomposition as you go from the big roadmap initiatives to the smaller development tasks. Each of these can be managed using a continuous process. However, every part of the process will need to adapt to continuous flow as it moves from architecture to development to testing to deployment. At medium and large companies, these will be different people, but hopefully not different groups altogether. Managing them as a single virtual team responsible for shepherding requirements to specifications to code to production can be done with a single continuous pipeline.

Using Kanban to mange this process will make it easier to spot bottlenecks. In Lean, the Theory of Constraints dictates that bottlenecks must be removed and the overall resource utilization balanced to produce a continuous flow from start to finish. Anyone that's idle or dealing with a large backlog represents waste that must be eliminated. For this to work though, you must set and respect queue (or Work In Progress; WIP) limits. Setting a limit and then ignoring it is almost as bad as not setting one in the first place.

Some processes will benefit from linear, gated phases at a higher level than day to day software development. These processes ensure stakeholders and approvers have a chance to review the plan at each stage (RACI is a good model for identifying these resources and their roles). However, if any part of this process intersects with your continuous development flow, it will definitely disrupt it and lead to waste (not Lean). The easiest way to do this is wrap the gate around some increment of product development, perhaps synchronized with quarterly review cycles.

When just looking at a process diagram or wading through the daily barrage of emails, it can be difficult to see where your processes, or how they're being executed, is leading to waste. To get back to Lean, interview the people doing the work, talk to the people upstream and downstream, and workshop different scenarios with stakeholders and technical leads. It might just be a small tweak to how people interact, how workflows are implemented in a ticketing system, or a change to some templates. It might also be a bigger problem that will require an executive champion to push for changes in other parts of the company. Regardless, do this on a regular basis so there's no tendency to normalize misery, noncompliance, or just plain inefficiencies.

Optimize

Every line of source eventually becomes legacy code. Plan ahead for this day and architect around it. If your product is a giant bowl of spaghetti, you'll never succeed in removing sections without destabilizing or even destroying the entire thing. Use things like microservices, hexagonal architecture, or just plain old APIs (if that's all you need) to break your code into services that can be improved (or replaced) independently when the time comes.

Sometimes the architecture starts out clean, but people start breaking the rules because of ignorance, urgency, or both. Ensure that your code review and checkin procedures prevent this sort of hacking or it will eventually lead to mortal wounds to the code base. If you already have a bunch of this scar tissue, start removing it.

Place a high value on the code you've already written - don't be lured by the siren calls of the senior developers that replacing the whole mess will be faster and result in a better product than fixing the old stuff. Engineers love creating new things. We also typically view with great cynicism presumptions that someone else was as smart as we are. If you think it's better to start from a clean slate, you should probably just call it a new product because it will not be profitable for you to make a new thing with a new roadmap that's simultaneously the old thing with the old roadmap.

Also continue maintaining and improving your automation as you proceed. Deep bugs that get buried in legacy code will rear their heads at the worst possible time and are some of the most difficult to track down. This is made more complex because chances are good that those people aren't on the team anymore and you might have to reverse engineer the legacy stuff before you can begin fixing it.

Reevaluate old build vs. buy evaluations. Stay focused on your ability to create unique value in the market and don't try to reinvent things other people are already good at, even if you only need a small part of it. You will almost never be better off rolling your own user management or database system. And unless it's legitimately your expertise, just stay away from trying to make your own cryptographic algorithms and security systems.

Summary

Building a product and growing a company is hard work. Make sure you're creating a sustainable development process to go along with it so that you don't flameout just as things are getting interesting. Invest in automation for things a computer can do - don't make humans do menial, repetitive tasks. Maximize time and money efficiency through continuous processes and make sure the heavyweight "executive summit" meetings don't disrupt your flow of releasing value to the market. Finally, continuously optimize your architecture, source code, automation, and technology selections to accelerate the pace of delivering value back to the market.

Next article: Conclusion

Recommended Reading

The Goal by Eliyahu Goldratt (Amazon ) [The original storybook for Theory of Constraints.]

The Lean Startup by Eric Ries (Amazon ) [Measure, Improve, Profit (or Pivot).]

The Phoenix Project by Gene Kim et al. (Amazon ) [An adaptation of The Goal for software and IT people.]

The Service Startup by Tenny Pinheiro (Amazon ) [My introduction to Design Thinking.]

Turn the Ship Around by L. David Marquet (Amazon ) [An excellent example of inverted leadership easily adapted to software engineering organizations.]

Wiring the Winning Organization by Gene Kim and Steve J. Spear (Amazon ) [Probably the single best book I've ever read on technology management.]

Justin Wolf wrote his first line of code when he was about five years old. Since then, he's written hundreds of thousands of lines of code for consumer products in assembly language (Intel and Motorola), C, C++, C#, and other languages. In the mid 1990s, he began learning about computer networking while at Cisco Systems and spent a decade working on enterprise and residential networking products. He worked for about eight years at national laboratory on cybersecurity projects and then at a lab spin-out leading product development of a visual analytics platform fusing machine learning with human-in-the-loop analysis. Most recently, he led a large engineering team building massively distributed systems for battery energy storage systems (nearly 100,000 networked devices for the largest deployments). He lives in Southwest Washington state with his family (and pets) and enjoys sailing in the Pacific Northwest inland sea as often as possible.