Software architecture — The Hardest parts
I recently read the book "Software Architecture — the Hard Parts" by Ford et al. It’s good stuff. You should read it! Yet, in my experience, they are missing the hardest and the most challenging parts.
The hardest parts of software architecture are, ironically, the softest ones. They relate closely to the organization’s power structure, history, culture, and sources of uncertainty.
Here you have my top 4 list of the most complex parts and six patterns that I’ve found helpful when struggling with these vicious problems.
The article is orginally published in Medium with the same title and cross-posted as this linked in article.
?? 1. Overlapping, inconsistent domain models and multiple sources of truth
When you start a new business, everything is easy. You choose the tools, decide where to put data, and so on. Good options are abundant. The situation is entirely different when two mature companies merge and try to align their IT systems. You have a lot of partially overlapping data and IT systems. In addition, you often have a badly over-optimistic plan to merge data and IT systems.
It is common to pick the low-hanging fruits and accept that you have some overlapping systems. It makes sense, but it also costs a lot in terms of complexity. Consequently, you will have multiple sources of truth. Most likely, you cannot solve that problem with technology, and even if you can, the solution will be complex, error-prone, and hard to maintain.
On the other hand, you can try to harmonize all the IT systems and operations. This is also a suboptimal solution because now you need to spend a lot of time on work that customers do not care about, and all that work is away from items that add customer value.
Mergers are just a shock that messes up good plans and ideas for keeping things clean and tidy. Changes in policies, partners, and regulations are another common source of chaos you cannot adequately prepare for. The bigger and older the organization, the more likely you need to cope with multiple overlapping and inconsistent domain models and multiple sources of truth.
From the software development and architecture perspective, there are two hard things to do:
This will be especially challenging if the reasoning behind the change (e.g., a company merger) is to increase customer value and improve efficiency. Modern architecture patterns and practices give limited tools to manage expectations and risk after a radical change in the fundamental assumptions and when you don’t have a shared epistemic foundation to build on.
?? 2. Language & local regulation and policies
US-based IT companies seem to have big problems understanding Europe’s cultural and lingual fragmentation, let alone the rest of the world. If you work in a small country, such as Finland, the problem is even more significant. Even if an IT system does not need to support many languages officially, you still usually need to operate with two languages: English and Finnish or Swedish. I’m not talking here about localization but something more fundamental — the shared domain model.
The best practice is to be consistent with the language. Hence, the code is usually written in English. In many cases, the language of business is not English, and occasionally, they do not even know what concepts should be used in English. Sometimes, there are no good translations.
Let’s assume you’ve created a shared language for the domain model. Good start. Then, you expand your business from Finland to the rest of Europe and North America. Suddenly, your domain model must cope with the policies and regulations of Germany, France, Italy, Sweden, Spain, the USA, Canada, and dozens of other countries besides Finland. Sometimes, this is not an issue — you’re lucky. Yet, fragmentary and inconsistent policies and regulations significantly increase complexity in many other cases. In addition, you need to cope with localization and culture-related differences, and you’ll waste a lot of time with inconsistent representations of time, numbers, metrics, and measures. This complexity is social by nature, and it drastically impacts the architecture, the need for coordination, and the team topologies.
Even worse, the current tools and services with multiple overlapping domain models could be better. For instance, there is no good way to define family resemblance in code. Take company types as a simple example: company types are very similar in Western countries but still partially different. The language of the data yields another layer of complexity. Used units of measurement and definitions of the used and needed metrics yield another. And so on.
While architecture patterns and practices provide ideas on how to minimize risks and problems, the root cause of many issues remains unaddressed: that is, there are, in fact, multiple overlapping and inconsistent domain models, data models, and languages.
You may have similar problems if you’re building an IoT (Internet of Things) system and have a variety of devices from different vendors. Devices will provide different APIs and yield various kinds of data, and you’ll have problems consolidating the data, e.g., for ML applications or reporting. Consistent IoT data is an easier problem because you don’t have a political and social layer to cope with, and sometimes you can solve the problem with money. The problem in IoT is purely technical, and logic, math, and pragmatic engineering take you far.
?? 3. Business model clashes and corporate politics
In my experience, at least the following kinds of clashes are common:
Architectural patterns and practices are often practically ignorant about the impact of corporate politics, business models, and decision-making patterns. If and as one of the main distractors is corporate politics, architecture should address it. Single-eyed focus on solution-related aspects such as ease of development, best practices for the technical solution (e.g., SOLID principles and clean architecture), maintainability, performance, security, scalability, and so on, is na?ve and inadequate.
A good software architecture should extend beyond technical details and purely technical patterns and practices. It should outline strategies to (1) address the conflicting needs of stakeholders and (2) minimize the risk of wasted work due to the volatility the power struggles cause. For instance, if there is a business model clash between new and old or different business areas, the tech experts should always prepare for changes in priorities and goals. In such a case, it is a question of time when the priorities will change, and taking them for granted will lead to suboptimal results.
领英推荐
?? 4. Conflict and misunderstanding due to diverse expertise, experiences, and knowledge
Whereas the business model level clashes make priorities and goals volatile, there is another source of epistemic bias in decision-making: The conflicts and misunderstandings that arise from differences in expertise, experience, and knowledge. They will increase accidental complexity and decrease motivation and willingness to take responsibility.
The conflicts are not only a negative thing. It is good that the team has diverse backgrounds and expertise. Consequently, you will have misunderstandings sooner or later. The challenge lies in the patterns of resolving these conflicts and learning from them.
You naturally emphasize patterns, practices, and technologies you’re familiar with. At the same time, this bias may have a high price. If you have an architect or a tech lead on the team, they often over-emphasize their experience, expertise, and knowledge.
For instance, it’s common for an architect to be versed in application logic written with object-oriented programming (OOP) languages such as Java, C#, or Python and a lot less on SQL. They tend to prefer ORM mappers without considering other alternatives for large-scale and performance-critical data pipeline logic. On the other hand, experts with strong SQL backgrounds tend to ignore or undervalue the need for Dependency Injection and versioning. Standard SQL schemas, views, stored procedures/functions, and linked servers (in MS SQL and foreign data wrappers, e.g., PostgreSQL) allow you to implement dependency injection and versioned interfaces with pure SQL and offer alternatives for coordination on the application logic layer in some use cases and provided significantly better performance if you have a lot of data to work with.
Another typical example is from the front-end world. The new CTO, architect, or architect team wants to change or consolidate the frontend framework company-wide. In my experience, the benefits you can gain by changing the framework you’ve used for years are marginal compared to the costs of a complete rewrite of an app.
Solution principles
There are no silver bullets to address the problems I have presented above. I find ideas from domain-driven design especially useful here. Concrete methods such as Event storming, Domain storytelling, or User story mapping deserve a separate post.
Instead of diving into details, I rather share the general principle I try to follow, when creating plan and vision of architecture.
?1? Open-minded curiosity over logic and best practices
Best practices and cold, logical calculations take you far. Overall, it’s good to know the traditions and upcoming trends. For instance, trade-off analysis patterns and practices introduced by Ford, Richards, Sadalage and Dehghan in The Software Architecture — The hard part is great. Evans', Walsh's and Brandolini’s ideas about Domain Driven Design are awesome. SOLID principles & Clean Architecture by Robert C. Martin provided a foundation for software design. And so on.
Then again, software systems and software development are intrinsically (corporate) political and dominated by social rules of power, wealth and influence rather than the rules of logic. The corporate political and social aspect of software development is more extensively discussed in agile and lean literature than in architecture discussion. The whole agile movement is a political/social paradigm shift rather than just a technical or pragmatic one. The ideals of software architecture move slowly from the modernistic realm of logic toward a more pragmatic, pluralistic postmodern approach that addresses better power distribution and social influence.
In short, overemphasizing logic and the best practices is, in practice, annoyingly arrogant and contemptuous from the point of view of business stakeholders and other developers of the teams. Arrogance and contempt often lead to low trust and motivation and unnecessary complexity. Low trust and motivation alone harm the delivery more than sub-optimal solutions that are not aligned with all the best practices. You need to set the bar for technical excellence, but you should not do that at the expense of political trust and influence.
When planning a technical solution, consider the following:
?2? Subsidiarity principle and autonomy over coordination and coherence
If you don’t necessarily need to delegate decision-making authority to a centralized authority, you should not do so. A central authority should perform only those tasks that cannot be performed at a more local level.
Again, coordination and coherence are essential, but autonomy and power distribution are more important.
In practice:
Summary
The hardest things in software architecture are partially non-technical: the used language, the lack of epistemic foundation, the conflict between the models of reality, and the social and political reality of the organizations. Current architectural patterns and practices do not adequately address these problems, although the Domain Driven Design tradition especially offers multiple pragmatic and good tools as painkillers.
In my opinion, architecture should pay more attention on the (corporate) political realities and the language needed to agree on priorities and goals. The architectures tend to offer overly rigid “big plans” that build too much on unsound assumptions about unchangeable and relatively stable priorities and goals. It fails to reveal emerging trust issues and false optimism until it’s too late. The architects and tech leads commonly consider too little the psychological aspects of the teams and experts and harm motivation and willingness to take responsibility by delegating too little decision-making power.
Both tech and business leaders should beware “bullshit jargon” trap. People won’t truly commit to a plan that seems mostly bullshit from their point of view.
?? All images are generated with Generative AI (DALL-E 3) using Microsoft Bing image creator and with design tools integrated into LinkedIn.
Power BI | Tableau | Python | Data Science | AI | Machine Learner | Marketing
10 个月Great insights on the "soft" challenges of software architecture, highlighting the importance of addressing social, political, and organizational complexities alongside technical solutions for successful ML and software projects.