Beyond Layers: Choosing the Right Architecture for Complex Problems

You were not hired to solve trivial problems. You were hired to tackle the complex, the secret sauce that makes your company great\! That’s why they need you – to translate their core business logic into robust, efficient, scalable software.

Yet, many still default to them. This is a dangerous game. So trying out an idea with a simple MVP project should not be in your hand. “Premature Optimization Is the Root of All Evil”, indeed, but underestimating the problem is a guaranteed path to disaster.

If you are reading this, it is because you understand this fundamental truth: complex problems require well considered architectures. You are here to find the right one, and I commend you for it ??

Some Architectural Patterns

Choosing the right architecture needs a deep understanding of your project requirements. Once you have a clear understanding, you can explore suitable architectural patterns. Here are some common architectures.

Layered Architecture

Organize application into horizontal layers, each with specific technical responsibility (e.g. presentation, business, data access).

Suitable for simpler application with clear separation of concerns. Easy to understand and implement, but can become difficult to scale for complex systems.

Service Oriented Architecture

A collection of services that communicate with each other with a central service bus. Note that it is not a microservice.

Pipeline Architecture

Process data through stages or filters, each performs specific transportation on the data. It consists of filters and transformers.

Suitable for applications that process data in sequential manner.

Microkernel Architecture (Plug-in Architecture)

Consist of core system and optional microkernel (plug in module). This allows for extensibility and customization without modifying the core system. Think of web browser extensions.

Suitable for systems with high extensibility and customization.

Event-Driven Architecture

Focus on applying a particular process for a particular emitted event. When implementing, it can be challenging to debug and trace the flow.

Suitable for application with real time data processing, asynchronous communication, and complex event processing.

Microservices Architecture

Decomposes the application into small and independent services that communicate with each other over the network. Each service has its own clear business justification.

Suitable for complex applications with high scalability, independent deployment, and diverse tech stacks.

Orchestration-Driven Architecture

It involves a central orchestrator that coordinates the interaction between services. Not only do you need to maintain the process flow, but the running infrastructure as well.

Suitable for complex business processes that need to be coordinated all together through central command and monitoring.

Practical Tips

Effectively navigating complex architectural challenges requires a structured approach. Here are some actionable tips to guide you:

Deeply Understand the Problem

Don't rush into solutions before truly grasping the problem's nuances. Dedicate focused time to gather information. I find a focused session (like a Pomodoro) extremely valuable for this.

During this time, gather all available facts, evidence, and documentation. Critically verify incoming requirements and analyze the problem domain. Key aspects to consider include:

• Background: What is the context of this problem? What are the business drivers behind it?
• Scope: What are the boundaries of the problem? What is included and excluded?
• Explicit and Implicit Requirements: What are the stated requirements, and what are the unspoken assumptions or implied needs?
• User Flows: How will users interact with the system? What are the typical user journeys?
• Data Schema: What data will the system handle? What are the relationships between different data elements?

Strategically Review Architectural Patterns

Familiarize yourself with common architectural patterns and their trade-offs. As mentioned earlier, common patterns offer different strengths and weaknesses. The goal is not to force-fit a pattern but to select the one that best aligns with the specific requirements of your problem. Remember: there's no silver bullet. Consider these factors:

• Alignment with Requirements: Does the pattern address the key scalability, performance, security, and maintainability needs?
• Company Standards: Does the pattern align with existing company norms, standards, and infrastructure?
• Team Expertise: Does the team have the necessary skills and experience to implement and maintain the chosen pattern?

Validate with a Targeted Prototype

Before committing to a full-scale implementation, build a small, focused prototype to validate key assumptions and identify potential roadblocks early. This allows you to experiment with different approaches and refine your design before investing significant development effort.

While AI assistants can be helpful for quick explorations and code generation, remember that a well-designed prototype should still be carefully planned and executed to provide meaningful insights. Focus on these aspects:

• Key Functionality: Focus on implementing the core features that are most critical to validate the architectural choices.
• Performance Testing (if applicable): If performance is a critical concern, include basic performance testing in the prototype.
• Feedback and Iteration: Use the prototype to gather feedback from stakeholders and iterate on the design as needed.

Conduct Regular Architectural Reviews

Architectural reviews are essential throughout the project lifecycle, not just at the beginning. Regular reviews foster shared understanding within the team, identify potential issues early, and ensure that the architecture remains aligned with evolving requirements. These reviews should be:

• Regular and Scheduled: Establish a regular cadence for reviews (e.g., bi-weekly or monthly).
• Well-Prepared: Come prepared with specific topics to discuss and any relevant documentation.
• Documented: Use tools like Architecture Decision Records (ADRs) to document architectural decisions, their rationale, and any alternatives considered. This creates a valuable historical record and facilitates future discussions.

Conclusion

The practical tips outlined above—understanding the problem, reviewing architectural patterns, prototyping, and conducting reviews—are valuable in various scenarios: starting a new project, integrating with existing systems, adding complex features, or pursuing overall system improvements. However, simply knowing these steps isn't enough. The effectiveness of each step is crucial. This begs some important questions:

• How do you truly understand a complex problem? What techniques do you use to gather requirements and analyze the problem domain?
• How do you efficiently identify relevant architectural patterns? Do you have a systematic approach to evaluating trade-offs and selecting the best fit?
• How do you create a compelling and informative prototype that effectively communicates your architectural vision?
• How do you clearly and persuasively communicate your architectural decisions to stakeholders, ensuring buy-in and shared understanding?

If you're still with me, it's because you recognize that mastering these skills is an ongoing journey. Our discipline has developed countless frameworks, methodologies, and tools to enhance our effectiveness at each of these stages. From domain-driven design and use case analysis to architectural decision records (ADRs) and various prototyping techniques, there's a wealth of knowledge available. The danger lies in "wandering around," reinventing the wheel, and not leveraging these existing resources.

This moment is an opportunity for reflection: to acknowledge the vast landscape of knowledge that you may not yet have fully explored—the "unknown unknowns." Recognizing these gaps is the first step towards bridging them and becoming a more effective and impactful software architect.

Now it is your turn. Reflect on a complex problem you have faced and how you approved the architectural challenges. Above of all, what were the biggest lessons you learned? Share your experiences, insights, and even your mistakes in the comments below. Let’s learn from each other and collectively improve our architectural problem-solving skills.

Ford, N., & Richards, M. (2020). Fundamentals of software Architecture: An Engineering Approach. O’Reilly Media.

Khononov, V. (2021). Learning Domain-Driven design: Aligning Software Architecture and Business Strategy. O’Reilly Media.

The Guide to the Software Engineering Body of Knowledge (SWEBOK Guide)