The End of Planning

Over the last years, I have time and again run into the conflict that many companies experience between long term planning and continuous (agile) practices. In an embedded systems context, where the system contains mechanics and electronics in addition to software, the organization needs long term planning for “the atoms” because of manufacturing processes, ordering of parts, etc. As the company seeks to ship a system that also includes the software and consequently all R&D units, including software, are asked to commit to delivering a large amount of functionality at specified date at some point in the (far) future.

Software R&D, on the other hand, is concerned with agile practices and sprint-based prioritization of the most important things to take on next. Being asked to commit to a delivery of the software part of a product 12 to 18 months out is simply a bad idea from a business perspective. First, because all engineering disciplines, including software engineering, have great difficulty with long term plans, those responsible for creating these plans exercise extreme caution. Often the rule of pi is used: first the effort and time is estimated and subsequently all numbers are multiplied with pi (3.14) in order to maximize the likelihood of delivering on time. However, engineering involves humans and once they know they have 18 weeks for a task that was first estimated to take 6 weeks, engineers will find ways to fill the 18 weeks with work on the specific task. Second, in a fast moving world, from a business agility perspective the company wants the ability to decide as late as possible which functionality to put into the product. And it wants to change the prioritization continuously as new information becomes available, competitors make surprising moves or new technologies reach suitable price points.

When analyzing why this pattern keeps reoccurring across industries and companies, I realized that the notion of long term planning for software is based on several misconceptions. First, product managers, system architects as well as sales professionals who grew up in a mechanics and electronics driven world assume that the functionality delivered at the start of manufacturing is frozen and can’t be changed anymore. Because of this, software needs to commit early and deliver on the exact date. Of course this is no longer true. In an age where virtually all products are connected and most products move to continuous deployment, the software enabled features can be deployed throughout the life of the product.

A second misconception is the, often implicit, belief that post-deployment issues will be incredibly expensive to fix. While this is undoubtedly true for the mechanics and hardware of products, it is not the case for the software in connected products. When properly architected, the cost of deploying new versions of software rapidly approaches zero. And when achieving sufficient modularization, deploying can be component based, minimizing the impact of updates to the point of being unnoticeable.

The third misconception is that deep dependencies and connections between the different parts of the system is not a concern but actually a good thing because it allows engineers to squeeze the last bit of performance out of a system that is dimensioned as close to the minimum as possible. Instead, proper decoupling should be placed at those interfaces in the system architecture where the rate of evolution is fundamentally different. In practice, mechanics and electronics tend to evolve in yearly (or even slower) cycles whereas software may be released multiple times per week. This leads to two orders of magnitude difference in evolution speed and obviously the interface between both parts needs to be as clean, stable and decoupled as possible. Unfortunately, in many companies I have seen situations where the specifics of sensors or actuators used in the lowest levels of the system percolate throughout the entire system right up to the user interface so that even the user is exposed to the details deep down in the bowels of the system.

The fourth misconception is that bill-of-materials (BOM) is the only thing that matters. Especially companies that mass-produce products tend to hold an implicit belief that anything that lowers the BOM is worth any engineering effort. This belief is incorrect for two reasons. First, with the size of the software increasing with an order of magnitude (10x) every 5-10 years, the software related R&D expenses go up as well. For many companies, the per-unit software cost is now on par, if not exceeding, the cost of hardware. It’s just that many companies use separate budgets of BOM and R&D and consequently this is not obvious until one analyses the financials. Second, dimensioning the hardware capacity at the very minimum removes the ability to deploy new functionality to products in the field because there is no space, either from a storage or a computational perspective, for the additional functionality.

Although there are other incorrect beliefs around this topic, the aforementioned ones are, in my experience, the primary concerns. The solution is, of course, to embrace the fact that bits are different from atoms and need to be treated as such as well. So, for anything atoms, we keep the existing planning processes as these, for better or worse, are the best ones available. However, for anything bits, we adopt a continuous planning process where every sprint the most important work gets prioritized and put in the backlog. As there will undoubtedly be a few items that are non-negotiable for the product as a whole, the software R&D team may be asked or forced to make a few long-term commitments. However, as long as the volume of long-term commitments stays low, say below 30% or 50%, it allows the company to enjoy the business agility that it needs to stay competitive.

Concluding, forcing software R&D teams to make long term plans and commitments is not only bad for the software teams, it is a sub-optimal approach for the company as a whole. Embracing the unique and different nature of software and capitalizing on the business agility that continuous, sprint-based prioritization offers is the superior way of working. And, of course, this does not just concern products under development but also, in the age of continuous deployment, the products out in the field. Finally, as I have worked with several companies on these questions, do reach out in case you want to discuss or learn more.