Software architecture — The Hardest parts

I recently read the book "Software Architecture — the Hard Parts" by Ford et al. It’s good stuff. You should read it! Yet, in my experience, they are missing the hardest and the most challenging parts.

The hardest parts of software architecture are, ironically, the softest ones. They relate closely to the organization’s power structure, history, culture, and sources of uncertainty.

Here you have my top 4 list of the most complex parts and six patterns that I’ve found helpful when struggling with these vicious problems.

The article is orginally published in Medium with the same title and cross-posted as this linked in article.

?? 1. Overlapping, inconsistent domain models and multiple sources of truth

When you start a new business, everything is easy. You choose the tools, decide where to put data, and so on. Good options are abundant. The situation is entirely different when two mature companies merge and try to align their IT systems. You have a lot of partially overlapping data and IT systems. In addition, you often have a badly over-optimistic plan to merge data and IT systems.

It is common to pick the low-hanging fruits and accept that you have some overlapping systems. It makes sense, but it also costs a lot in terms of complexity. Consequently, you will have multiple sources of truth. Most likely, you cannot solve that problem with technology, and even if you can, the solution will be complex, error-prone, and hard to maintain.

On the other hand, you can try to harmonize all the IT systems and operations. This is also a suboptimal solution because now you need to spend a lot of time on work that customers do not care about, and all that work is away from items that add customer value.

Mergers are just a shock that messes up good plans and ideas for keeping things clean and tidy. Changes in policies, partners, and regulations are another common source of chaos you cannot adequately prepare for. The bigger and older the organization, the more likely you need to cope with multiple overlapping and inconsistent domain models and multiple sources of truth.

From the software development and architecture perspective, there are two hard things to do:

1. minimize the damage due to the lack of shared understanding and facts, and
2. justify temporarily plummeting capability to deliver more value to customers.

This will be especially challenging if the reasoning behind the change (e.g., a company merger) is to increase customer value and improve efficiency. Modern architecture patterns and practices give limited tools to manage expectations and risk after a radical change in the fundamental assumptions and when you don’t have a shared epistemic foundation to build on.

?? 2. Language & local regulation and policies

US-based IT companies seem to have big problems understanding Europe’s cultural and lingual fragmentation, let alone the rest of the world. If you work in a small country, such as Finland, the problem is even more significant. Even if an IT system does not need to support many languages officially, you still usually need to operate with two languages: English and Finnish or Swedish. I’m not talking here about localization but something more fundamental — the shared domain model.

The best practice is to be consistent with the language. Hence, the code is usually written in English. In many cases, the language of business is not English, and occasionally, they do not even know what concepts should be used in English. Sometimes, there are no good translations.

Let’s assume you’ve created a shared language for the domain model. Good start. Then, you expand your business from Finland to the rest of Europe and North America. Suddenly, your domain model must cope with the policies and regulations of Germany, France, Italy, Sweden, Spain, the USA, Canada, and dozens of other countries besides Finland. Sometimes, this is not an issue — you’re lucky. Yet, fragmentary and inconsistent policies and regulations significantly increase complexity in many other cases. In addition, you need to cope with localization and culture-related differences, and you’ll waste a lot of time with inconsistent representations of time, numbers, metrics, and measures. This complexity is social by nature, and it drastically impacts the architecture, the need for coordination, and the team topologies.

Even worse, the current tools and services with multiple overlapping domain models could be better. For instance, there is no good way to define family resemblance in code. Take company types as a simple example: company types are very similar in Western countries but still partially different. The language of the data yields another layer of complexity. Used units of measurement and definitions of the used and needed metrics yield another. And so on.

While architecture patterns and practices provide ideas on how to minimize risks and problems, the root cause of many issues remains unaddressed: that is, there are, in fact, multiple overlapping and inconsistent domain models, data models, and languages.

You may have similar problems if you’re building an IoT (Internet of Things) system and have a variety of devices from different vendors. Devices will provide different APIs and yield various kinds of data, and you’ll have problems consolidating the data, e.g., for ML applications or reporting. Consistent IoT data is an easier problem because you don’t have a political and social layer to cope with, and sometimes you can solve the problem with money. The problem in IoT is purely technical, and logic, math, and pragmatic engineering take you far.

?? 3. Business model clashes and corporate politics

In my experience, at least the following kinds of clashes are common:

1. Clashes between the old service and the new products and services. For instance, on the other hand, you should work on new products and services to ensure market position in the future, but on the other hand, currently, the money comes from the old ones, and you must recognize them too.
2. Bias due to the power centers. For instance, it’s common that in multinational corporations, the headquarters country’s business model dominates, and that will harm subsidiary countries’ business initiatives and architectures.
3. The clash between the needs and services of different business areas. For instance, self-service e-commerce vs. premium expert services. Often, IT development is centralized and services multiple business areas that fight limited development efforts.

Architectural patterns and practices are often practically ignorant about the impact of corporate politics, business models, and decision-making patterns. If and as one of the main distractors is corporate politics, architecture should address it. Single-eyed focus on solution-related aspects such as ease of development, best practices for the technical solution (e.g., SOLID principles and clean architecture), maintainability, performance, security, scalability, and so on, is na?ve and inadequate.

A good software architecture should extend beyond technical details and purely technical patterns and practices. It should outline strategies to (1) address the conflicting needs of stakeholders and (2) minimize the risk of wasted work due to the volatility the power struggles cause. For instance, if there is a business model clash between new and old or different business areas, the tech experts should always prepare for changes in priorities and goals. In such a case, it is a question of time when the priorities will change, and taking them for granted will lead to suboptimal results.

?? 4. Conflict and misunderstanding due to diverse expertise, experiences, and knowledge

Whereas the business model level clashes make priorities and goals volatile, there is another source of epistemic bias in decision-making: The conflicts and misunderstandings that arise from differences in expertise, experience, and knowledge. They will increase accidental complexity and decrease motivation and willingness to take responsibility.

The conflicts are not only a negative thing. It is good that the team has diverse backgrounds and expertise. Consequently, you will have misunderstandings sooner or later. The challenge lies in the patterns of resolving these conflicts and learning from them.

You naturally emphasize patterns, practices, and technologies you’re familiar with. At the same time, this bias may have a high price. If you have an architect or a tech lead on the team, they often over-emphasize their experience, expertise, and knowledge.

For instance, it’s common for an architect to be versed in application logic written with object-oriented programming (OOP) languages such as Java, C#, or Python and a lot less on SQL. They tend to prefer ORM mappers without considering other alternatives for large-scale and performance-critical data pipeline logic. On the other hand, experts with strong SQL backgrounds tend to ignore or undervalue the need for Dependency Injection and versioning. Standard SQL schemas, views, stored procedures/functions, and linked servers (in MS SQL and foreign data wrappers, e.g., PostgreSQL) allow you to implement dependency injection and versioned interfaces with pure SQL and offer alternatives for coordination on the application logic layer in some use cases and provided significantly better performance if you have a lot of data to work with.

Another typical example is from the front-end world. The new CTO, architect, or architect team wants to change or consolidate the frontend framework company-wide. In my experience, the benefits you can gain by changing the framework you’ve used for years are marginal compared to the costs of a complete rewrite of an app.

Solution principles

There are no silver bullets to address the problems I have presented above. I find ideas from domain-driven design especially useful here. Concrete methods such as Event storming, Domain storytelling, or User story mapping deserve a separate post.

Instead of diving into details, I rather share the general principle I try to follow, when creating plan and vision of architecture.

?1? Open-minded curiosity over logic and best practices

Best practices and cold, logical calculations take you far. Overall, it’s good to know the traditions and upcoming trends. For instance, trade-off analysis patterns and practices introduced by Ford, Richards, Sadalage and Dehghan in The Software Architecture — The hard part is great. Evans', Walsh's and Brandolini’s ideas about Domain Driven Design are awesome. SOLID principles & Clean Architecture by Robert C. Martin provided a foundation for software design. And so on.

Then again, software systems and software development are intrinsically (corporate) political and dominated by social rules of power, wealth and influence rather than the rules of logic. The corporate political and social aspect of software development is more extensively discussed in agile and lean literature than in architecture discussion. The whole agile movement is a political/social paradigm shift rather than just a technical or pragmatic one. The ideals of software architecture move slowly from the modernistic realm of logic toward a more pragmatic, pluralistic postmodern approach that addresses better power distribution and social influence.

In short, overemphasizing logic and the best practices is, in practice, annoyingly arrogant and contemptuous from the point of view of business stakeholders and other developers of the teams. Arrogance and contempt often lead to low trust and motivation and unnecessary complexity. Low trust and motivation alone harm the delivery more than sub-optimal solutions that are not aligned with all the best practices. You need to set the bar for technical excellence, but you should not do that at the expense of political trust and influence.

When planning a technical solution, consider the following:

1. How can you provide evidence of progress early and frequently thereafter to (1) increase trust and shared understanding among the stakeholders and (2) make possible lack of trust and shared understanding a painfully visible problem? I underline that not all evidence of progress is equally good. The wrong kind of evidence (partially done work, in particular) creates false optimism. You should seek political trust and strong social bonds and tackle foreseeable trust issues as early as possible by making them apparent. E.g., start by asking, “When things go like in a dream, what is that like? How do you know we succeeded in this project or initiative?”
2. Make tech language easy to relate to and approachable for everyone. How do you first convert technical jargon into language the business stakeholders can relate to and then help those who do not have expertise on the used technology expand their expertise and understanding with ease? You cannot discuss technology without using ‘technological language’. Yet you should consider how to make the entry point into tech wonderland more approachable and appealing. It’s na?ve to dream that anyone can translate the technical side of software into a ‘non-technical business language’.
3. Make business problems easy to immerse in, grasp, and experiment with. How do we express business problems so that it is easy for tech-oriented experts to relate the business side of the problem and immerse into the role of the end-user and the sponsor of the work? In short, how might you help people to be curious? Whereas you cannot and should not protect business people from technology, you should neither protect the tech-oriented people from business, no matter how business averse they might be.

?2? Subsidiarity principle and autonomy over coordination and coherence

If you don’t necessarily need to delegate decision-making authority to a centralized authority, you should not do so. A central authority should perform only those tasks that cannot be performed at a more local level.

Again, coordination and coherence are essential, but autonomy and power distribution are more important.

In practice:

1. Distribute power and trust by default. Those who do the work should choose the tools, libraries, and technologies unless there is a strong need for coordination with the organization and the teams. If coordination is necessary, it should be the minimal and grand maximum amount of decision-making power to those whom the decisions have the most immediate impact. For instance, I find attempts to consolidate the tech stack often dysfunctional. Business cases where you should, say, rewrite a large code base using React and C# with Angular and Java (or vice versa) are rare. Usually, rewriting solutions to consolidate the tech stack and maximize reusability is a waste of time and a guaranteed way to ruin motivation and proactive decision-making. If the benefits of consolidation and shared practices are apparent and significant, express them with precise and unambiguous language and make them open for falsification.
2. Facing reality is the foundation of wisdom. You should accept that no single source of priorities and business goals exists if that is the case. For instance, it does not help stubbornly insist that “the business should set the priorities” if they over and over again fail to do so. Instead, priorities should be seen as parts of pluralistic political agendas that are innately unsure and based on insufficient information. In practice, those who have the closest relation to the customer and those who feel the pain of poor architectural decisions are most likely to react in a timely order. Hence, they should also have the decision-making power to do so, even if this may reduce the coherence of the architecture and business models. Again, coordination is necessary, but it should focus on the constraints relevant to everyone and the transparency needed for accountability, not the control mechanisms and excessive upfront planning.
3. Beware “bullshit jargon” -trap. If finding shared priorities that hold seems to be hard, consider what kind of language is needed for an agreement and commitment. Maybe the prioritization language contains an excessive amount of bullshit and jargon. In my experience, the most common reason for the lack of commitment is that the goal, plan, and vision are seen as over-simplistic bullshit. People just don’t believe them, but they also feel that they cannot constructively disagree or ask questions. The used language is unreachable, and they fear that their questions are stupid. For instance, an architect might justify the deployment of yet another service by saying that it’s necessary because of the separation of concerns. It’s maybe true, but it’s also potentially bullshit from developers’ point of view. Overarching and abstract tech concepts (e.g., separation of concerns or inversion of control) can be handy. Still, the justification of tech decisions should be, most of the time, far more concrete, tangible, and testable (or falsifiable). Consider addressing concrete concerns instead, such as “What is the worst thing that can happen if we deploy solutions for these two needs as a part of the same service instead of separating them into two microservices and vice versa?”

Summary

The hardest things in software architecture are partially non-technical: the used language, the lack of epistemic foundation, the conflict between the models of reality, and the social and political reality of the organizations. Current architectural patterns and practices do not adequately address these problems, although the Domain Driven Design tradition especially offers multiple pragmatic and good tools as painkillers.

In my opinion, architecture should pay more attention on the (corporate) political realities and the language needed to agree on priorities and goals. The architectures tend to offer overly rigid “big plans” that build too much on unsound assumptions about unchangeable and relatively stable priorities and goals. It fails to reveal emerging trust issues and false optimism until it’s too late. The architects and tech leads commonly consider too little the psychological aspects of the teams and experts and harm motivation and willingness to take responsibility by delegating too little decision-making power.

Both tech and business leaders should beware “bullshit jargon” trap. People won’t truly commit to a plan that seems mostly bullshit from their point of view.

?? All images are generated with Generative AI (DALL-E 3) using Microsoft Bing image creator and with design tools integrated into LinkedIn.