Organizational design models in the evolution of managerial thinking (part 1)

"You can't always get what you want. But if you try sometimes, well, you might find. You get what you need"

(C) The Rolling Stones

What kinds of different organizational design paradigms are known for software product development? How do organizations develop over time and are there any shortcuts on this journey? What should the company management and executives know about all this?

In the role of an organization agility consultant, I often help my clients (in particular leaders of mature organizations) in finding a better organizational design - i.e. finding a better fit between the structure of the organization and the needs of the business. This is a rather non-trivial task that has no obvious right answers. But what makes it even more difficult is the lack of a clear common basis (terminology, vocabulary) and shared understanding among practitioners. So if this article takes off, it can serve as a map of a management battlefield with its core strategies that practitioners can relate to and build on top of.

This article shares some most commonly observed models of how organizations that produce software get typically structured. Such organizations don't necessarily sell the software they produce (though, this is one option), but they ought to produce some software that is necessary to run the core business (whatever that is). And because nowadays arguable any business comes with some sort of custom software it relies upon, this article hopefully can be useful outside of the classical IT crowd.

I'll share six somewhat distinctive models of organizational design (or management paradigms). Some models described below are somewhat linked to each other with evolutionary ties - that is, they grow naturally on top of each other as the headcount and business desires grow. And some others require some sort of transformation of the management mindset to jump out of the status quo.

Since the organizational design (how things are structured) is inseparable from processes (how things are running), we will consider these six schemes that combine organization and operation of software production:

1. (Simplified) Startup.
2. (Centralized) IT development.
3. (Horizontal) Component development.
4. (Overcomplicated) Project development.
5. (Side) Satellite development.
6. (Vertical) Product development.

In this article, I will guide you through a typical evolutionary path from a Startup to a mature Product development organization with its unavoidable crises on the way. I will describe how one hypothetical organization goes through these models one by one as these were the stages of development - but they are not! It is important to keep in mind that such a chosen storyline is just one option from the wider set of possibilities how companies might decide to develop. I'm following this storyline for the sole purpose of more coherent storytelling. But for you, the reader and leader of an organization, it shall be clear that companies are not destined to following any given path, but a path being taken is a sum of all cornerstone decisions made.

So I do believe in the free choices of managers who consciously (or unconsciously) make them and by doing this keep forming their organizations - what we call an "org design". This article is about the causes and consequences of these uneasy management decisions and also about some irrational thinking that kicks in on the way.

This is also not an article about "waterfall VS. agile" (this would have been an overly simplistic view of things). In this article, you will learn about the broader field of decisions that the designers of organizations are facing.

Startup

You cannot start exploring how mature organizations function without having a quick look at how startups get formed, from the viewpoint of their structure and processes. Understanding startups is essential to understanding "mature" organizations as in the life of every organization there are times when an organization wants to add processes, structure, and rules and times when it wants to "hack around and just get things done". Such movements are like a "seesaw" are following one another - today a company wants to be more like an enterprise (meaning it is dreaming of more standardization and operational efficiency) followed by a wish to be more like a startup (dreaming of more results and better alignment) followed by a wish to be more like an enterprise again, and so on an so forth. Such back and forth movements are periodic and are provoked by temporary growth crises (more about them below) so understanding what makes this "startup dream" so powerful is important to understand the dynamics of any mature organization.

In a nutshell, an idealized startup-like style of software product development is an organization of cross-functional work of a small number of people around a clear business strategy. Usually with direct access to the clients and end-users. Note how close this is to a Scrum prescription of a cross-functional colocated dedicate full-time customer-facing team. This is not a coincidence. The goal of Scrum, if you will, is to help organizations return their lost youth...

Nowadays startups are also seen as lean operations (thanks to the Lean Startup movement that really became the norm and the status quo of how entrepreneurs operate their young ventures). Startups naturally don't have excess resources to waste so they have to stay laser-focused on a specific mission: for instance, getting traction from a wisely chosen customer segment. [Nothing stops more mature orgs to take on this lean approach as their resources are somewhat limited too, but for a startup, this is a clear survival mechanism creating a strong alignment and focus, not an artificially fixed "project budget" in a mature organization]. From a clear shared focus, other good things are got derived such as goals, expectations, metrics, and milestones. And they are shared by everyone involved, typically by the entire team. Until the moment this unity starts to compete with locally-set goals that are driven by some organizations changes.

"Sales" is most common the first department being formed that immediately starts to drift away from the cross-functional core (R&D in the future). This can be related to the "noisiness" of the salespeople's work, their extraversion that happens to be somewhat uncomfortable for the IT crowd, and other common beliefs - but these are just the stereotypes. The only reason this starts to happen because someone decides this is to happen. Remember the "free managerial choice" thing we talked about just a few paragraphs before?

Isolation of the sales dept can lead to the consequences which negative impact can only be clearly seen in the long run. Examples I've seen include a desire of sales folks to close contracts at any costs including negative profitability, agreeing on the unrealistic demands of customers for delivering impossible product functionality, setting wishful deadlines, etc. From this moment the R&D will be ever blamed for "low quality of estimates and missed deadlines". The consequences such decisions bring are usually hard enough to fight because they just turn everyone into being extremely busy, overstressed, and unhappy with whatever outcomes. And when everyone is like that - there is no time for a clear wide system thinking, and a shortcut-looking mindset kicks in. And then it quickly becomes a habit and a vicious cycle is only making things worse and worse with every loop...

The real task of a competent org designer in such a young company is to try to avoid this split at any short-term costs. This implies practicing long-term thinking at the cost of "quick cheap wins" that are so tempting at times when cash is burning low. Keeping everyone in the same boat of a common business cadence, making everyone believe in shared goals, practicing cross-department collaboration with the clients, holding a focus on the value delivered - these are some vital practices for keeping the startup spirit alive. If the management can withstand this first fight and keep the naturally-born product culture and client orientation of a startup, then with upcoming growth such an organization can jump over the next few models described below: Centralized IT Development and Project Work right into Product Development. And if not - the journey will be long and winding.

It is worth saying that the odds of sustaining the product culture in a growing startup are very low (like 1-5%, my observations). Why? Being a maverick sounds cool only if you read about one - no one really wants to be one and what I see all around are mere unthoughtful copy-and-paste solutions. Top management of a company copies ideas of others, though the others are not really doing any better. And then joined tuition in "Exec MBA" schools only strengthen their beliefs in collective righteousness. So seems all the business forces of the universe work together to keep management isolating software product development from the rest of the organization, from the business... Why? Simply because others are doing so.

And that's the beginning of a downward spiral that is pushing IT to the underground floors of organizations, literally.

(Centralized) IT Development

At this stage of development, the initial business model (or its adapted and improved increment) has been already proven validated by a positive market response. The company gets its first real investment round. The appetites of shareholders are starting to grow faster and with them the stress and pressure on IT. Growth in terms of getting more headcount is kicking in.

And this soon leads to the first management crisis - a crisis of direct management. It can be seen already with 30-50 engineering stuff. This is a growth-crisis that deals with the fact that the power of personal influencing connections, which before with fewer people used to glue it all together despite various issues, - now with more people these interpersonal relationships are not enough to serve as the center of gravity anymore. Smaller cracks in the organizations that had been created earlier are now opening wide more and more with every new person hired.

The situation worsens as the execs get themselves separated from the realities of work and get busy with "strategies" - sometimes just of pure incompetence of solving real organizational issues (I'm sorry to say that). To compensate for the lack of their involvement a special squad of specialists enters the company doors - process guys with "a good track record and prior enterprise processes experience". They start to form a new elite - vice presidents. And each department now has its own VP.

And of course, here we see the emergence of the VP of Engineering. His presence reinforces the IT silo with its opaque processes, lack of agility, and the known problems of "missing the deadlines and bad estimates". But now IT has more expensive excuses stated with the proud voice of an enterprise.

"We've overgrown the startup, we are a mature organization now, we need to focus on processes" - says the VPoE. "You want clear estimates from us? Fine, I'll handle it. But to control our estimates we need to limit the continuous flow of change requests coming down on us from the chaotic sales team. We need better engineering processes". (Or something like this).

From this moment on, the organization is in a trap. The appetite of the business elites will keep growing (resulting in more pressure). Marketing folks will be tapping into new prospects and new customer segments (resulting in less focus). But the R&D will be fencing itself off the uncomfortable requirement changes (by the way it is not the requirements that get changed, but our understanding that grows).

Such dynamics described above will be leading to a mutual lack of trust between the business and the IT side of the org. What would fix the issue is the presence of a business stakeholder holding these two worlds together, acting as a leader and owner (that would introduce a real Product Owner as we see this role Scrum). But ironically it is highly unlikely to expect a business stakeholder risking his skin and stepping into the IT side. It is too dangerous for the carrier as IT constantly fails. And because no business stakeholder will be willing to lead and guide the software product development, it will be staying in the hands of engineering heads, who will be focused on the constant operational issues - fixing the software holes and ruling permanent release dramas (due to constant pressure and rush software quality has been degrading now making it harder to introduce changes). So the IT heads will be busy dealing with all this chaos (for what they were hired in the first place), thus having a lack of strategic and business focus.

Can you spot the vicious cycle here?

An example of companies that might be facing these kinds of challenges: businesses (for instance e-commerce) that need some custom software to operate (maybe even a lot of it) but unable (yet) to see and embrace software production as their direct activity. Also, businesses that try to limit the amount of software they actually need - like a few dozens of IT staff plus several vendors.

By the way, using a vendors-only approach might be not such a bad idea here, for at least (provided the vendors are real software professionals) they can showcase to the business ways how to guide software development.

We've just covered two models:

• (Simplified) Startup.
• ?(Centralized) IT development.

A real company can stop at this level when there is no need or opportunity for growth. But in the next chapter of this article we will watch our sample organization passing through the next phases:

• (Horizontal) Component development.
• (Overcomplicated) Project work.
• (Side) Satellite development.
• (Vertical) Product development.

So read on!