Most Productivity & Process Improvement Programs are Bogus!

Today's leadership blogpost is dedicated to Eliyahu Goldratt, the originator of the?Theory of Constraints?and the author of several best selling books including The Goal, which is one of three books that Jeff Bezos made his top execs read.

The Goal is actually a novel (In his own words: Goldratt knew his customer base well - many managers/leaders don't read books, only novels. That's why he wrote business books as novels!) about Alex Rogo, a manufacturing plant manager, who is given the tough job of turning around a failing plant in three months. Things were going downhill at the plant - shipments were late, inventories kept on piling up and the production backlog was also growing. Alex luckily stumbles upon his old Physics professor, Jonah, who challenges him by asking some thought provoking questions. Jonah was an educator, not a consultant. ?Using his clues, Alex eventually turns around the company and along the way learns a few important management lessons. Alex realizes how the robots that were introduced in the plant negatively impacted the overall productivity of the plan, and stops chasing efficiencies. After all, "a plant in which everyone is working all the time is very inefficient."

In today's blogpost, I make the case that most claims of developer productivity gains & performance improvement programs are bogus. Productivity improvements have to be in service of something else - value creation or some innovative product/service for the customer. Innovation doesn't happen by copying your competitors. So, if many of your development teams are simply copying your competitors instead of leading them, one has to wonder if that is the right kind of productivity we need to pursue. Leaving aside this 'effectiveness' perspective, I'd like to question if productivity improvements that are driven at many companies (just for productivity's sake) are worth it in the first place.

Most modern corporations are divided into departments like marketing, finance, product, engineering, security, legal, etc. and those departments in turn have various sub-groups and sub-teams under them with each of their leaders driving their own local efficiencies without a clear understanding of its short-term and long-term implications to the global effectiveness.

As the Systems Thinker, Russel Ackoff elegantly put it,

“If each part of a system, considered separately, is made to operate as efficiently as possible, the system as a whole will not operate as effectively as possible.”

Take the example of Costco, which hasn't increased the price of its rotisserie chicken from $4.99 since the deal was introduced in the early 1980s. Why is rotisserie chicken so cheap at Costco? Costco could make a giant profit on chickens just by raising the price by only $1, as it sold a record-breaking 106 million rotisserie chickens globally in 2021 alone, up from 101 million in 2020. Instead, the store keeps the prices low on its famous rotisserie chickens and other staples as an incentive to get shoppers in the door. This is known as a "loss leader" in the retail world.

Costco deliberately makes some parts of its system inefficient so that the whole system can be more effective.

Corporations are complex socio-technical systems with a ton of interactions and inter-dependencies between its departments and their sub-groups. With a linear reductionist approach we'll not be able to figure out which parts of the system must be kept inefficient (less productive or not fully utilized) to improve the performance of the whole corporation. Instead, leaders need to have a systemic worldview of the corporation in order to effectively manage it - to figure out how to synergistically align the local efficiencies so that they actually end up improving the global effectiveness.

Constraints in Manufacturing

In manufacturing, the Work-in-Progress (WIP) inventory is visible (but not so much in software). You can see where the queues are piling up in the assembly line. Some processes/steps in the assembly might be a constraint - e.g. a furnace used to heat treat components before they can be assembled. But, the heat treatment duration is typically a set duration (say 2 hours or 3 hours depending on the components). There is no way to speed that up. The capacity of the furnace is also set once the plant is constructed. So, there are multiple design considerations that goes into the design and capacity of that furnace. It might be built for peak production capacity (think holiday shopping season) and kept under-utilized (inefficient) most of the days - the plant manager and the corporation is usually fine with it.

Even though the furnace run time is fixed, there are always some local performance improvements that can be done. For example: If the plant workers end up finding many poor quality components only during assembly (after the heat treatment), it'll result in them not shipping the orders that are due at the end of the day. There might not be enough time for another 3-hour run of the furnace. Instead a simple process improvement can be made by doing the components inspection before loading them in the furnace and/or adding a few additional ones as buffer. The key point here is not to waste the precious time of the constraint (the furnace).

While the above is definitely a local efficiency, plant managers can't stop there - they have to expand their "system" boundary to the entire plant and beyond (including say suppliers) if they have to truly improve the overall effectiveness.

InfoSec as a Constraint

Lets now translate & apply* the above manufacturing example to the various product & engineering teams engaging with the Information Security teams of their company. The engagement is usually orchestrated through a ticketing system and could range from a quick question on a security topic or for a security review or for a collaborative threat modeling exercise.

Security teams are a constraint on the whole system (the corporation) - they many times attenuate the variety of what the various product teams build and ship in order to keep the whole system secure. They are also limited in headcount in many companies compared to the size of the overall development organization and so end up becoming a bottleneck of sorts to the overall flow/throughput.

Local Efficiency: When a ticketing system is setup as free-form text input (say as a plain vanilla JIRA project), it may result in a lot of back and forth asking for additional documents, questions, etc. Instead, a ticketing portal can be setup in such a way so that a inspection of sorts can be done upfront via a structured intake form that makes sure all the necessary documents are made available in one shot - thereby not wasting the precious time of the already bandwidth constrained security teams. Further automations can be built and performance improvements can be driven based on this approach.

Headcount allocations to security teams can be made based on what the overall throughput the corporation desires. The relationship between capacity and cycle time is well understood - if all security engineers are loaded close to 100%, then cycle time will grow exponentially and the overall throughput of the corporation will suffer.

As Jonah, the character in 'The Goal' put it,

“What you have learned is that the capacity of the plant is equal to the capacity of its bottlenecks”

All other productivity gains & performance improvements made elsewhere may go to waste if the bandwidth constrained security teams hold up many development teams - the overall throughput and productivity suffers! If we want better cycle times, then we need to staff more security engineers and reduce their work load. Keep them deliberately inefficient (idle) so that the overall system performs better. Security is such a dynamic field and so the security engineers can potentially spend that idle time learning and improving their knowledge base across various security domains & also imparting it to engineering teams. More importantly, they can invest in cross-training and avoid too much specialization which in turn also impedes flow. Some large corporations may argue that cybersecurity is an arms race! The bad guys are constantly upping their game and so must the security teams. Any team that "doesn't have time" for learning will fail eventually anyways.

Global Effectiveness: Chasing better SLAs of a ticketing process is still chasing efficiency. Leaders have to make the paradigm shift away from thinking about 'work of security' to 'security of work'. Engaging with the security team for every project or feature to do manual security reviews is obstructive and impedes flow. Instead, leaders have to dissolve the problem by chasing effectiveness. Investing in 'secure building blocks' is a key aspect of that approach. With the right engineering investments made in the necessary secure building blocks (aka paved paths), security teams can be better assured about the products built using them and everyone can go fast with no manual processes! Security and velocity don’t have to be zero-sum games. As more and more secure building blocks are built and deployed, less & less time is needed to do manual security reviews and everyone can collaboratively improve the throughput.

Also, no problem lives in isolation in a complex system - security problems are no different. For example, let’s say vulnerability management is broken in your organization. If vulnerability management is broken, then so are one or more of asset management, configuration management, OS image management, automation, etc. So, the "wholistic fix" is really for the engineering stakeholders to focus on systemic and structural uplift to foundational capabilities like asset management, config management, automation, etc. The engineering organization has to collaborate deeply with security experts at the company and be focussed on investing, building and improving those foundational capabilities. This takes long-term thinking and calls for engineering and security leaders to think of the whole “system” and not just their own parts.

Wrapping Up

End of the day, it doesn't matter what the constraint is (InfoSec, Procurement, etc.) - leaders have to think of the "whole system" and understand how the overall flow & productivity is impacted when other parts of the system interacts with them. Reed Hastings, the co-CEO of Netflix understood this very well and dissolved these problems away creatively. If you are interested to learn more about that case study, please checkout my old blogpost here.

I'll end today's post with a couple of quotes from Goldratt:

"Striving to ensure that no resource be idle is the biggest generator of waste."

and

"Our fear of complex systems that drives us to dissect complex system into sub-systems, [which leads] to diverting management attention to chase local optima which are not inline with the global objective."

* (?????????? ?????????????????????????? ???????? ???? ???? ???????? ????????????/????????????????????, ???? ???? ?????? ?????? ?????????????? ?????????? ?????????????? ??????????-?????????????????? ?????????????? ???????? ?????????????????????????????????? ???????? ?????????????? ??????-???????????? ????????????????. ???????????? ???????? ???? ???????????????????????? ???????? ???????????? ?? ?????????????????????????? ?????????????? ?????????????????????? - ?????? ?????????????? ???? ?????????????? ?? ???????????????? ?????????????????? ???? ?????? ?????????????????????? ?????? ?????????? ???????????? ??????????????????????????.)