How do you choose the right system architecture evaluation method?

Selecting the right system architecture evaluation method is key to a project's success. It's important to consider the evaluation goals, context, and methodology. For instance, if the goal is to assess scalability, simulation or load testing may be necessary. The context, including deadlines, budget, and stakeholders' needs, might require adapting the evaluation method. Methodology involves planning and reporting the evaluation activities, defining criteria, variables, data collection, and analysis methods. A variety of methods can be used, such as prototyping, expert review, simulation, formal analysis, and experimentation. Each has its benefits, from identifying issues to predicting performance and ensuring design consistency.

While I do agree and support many of the points of view shared, I want to point out a couple of areas I didn't see addressed: staff and team capability as well as training for ongoing support. In the attempts to modernize, there are commensurate concerns regarding the often overstated goals that encompass the FASTER, BETTER, MORE RELIABLE, LESS EXPENSIVE taglines.? Let's also hone-in on the customer experience; how do these goals all converge to improve a service?? Resolutions involve experimenting, taking risks, GOING BOLD, failing fast, and being prepared to adjust the architectural choices. Paper airplanes, as a metaphor for a design on media approach, will not be proven unless you build it, test it, and then adjust.

When discussing the review process for system architecture, it's important to consider the overarching goals and plans. In my view, we should establish a clear purpose, whether it's outlining a roadmap or detailing the design. These aspects should align seamlessly, with other factors like security, reliability, and scalability falling into place afterward. The roadmap lays out the direction we're heading, while the design offers specific guidance on how to get there. Essentially, the design represents the journey toward achieving our goals in a structured manner. Both aspects should be timeboxed to ensure progress within set deadlines.

I agree that clearly defining your evaluation goals is crucial. The metrics you measure should directly relate to the quality attributes that matter most for your system - scalability, reliability, security etc. Don't try to evaluate everything - focus on the 2-3 priority goals.

Involve relevant groups, including developers, users, and management. Their perspectives can provide valuable insights into the priorities and concerns related to the system architecture.

As part of the evaluation process, a second factor to be considered is the evaluation context, which refers to the specific environment or framework that will be used to conduct the evaluation. An understanding of the evaluation context is essential in order to identify the constraints and assumptions. The evaluation context contains assumptions referring to the underlying beliefs or expectations guiding the evaluation process. It is possible for these assumptions to influence the selection of evaluation methods, interpretation of findings, and conclusions drawn as a result of the evaluation.

Context refers to the guidelines provided by a framework to carry out evaluations. Constraints and assumptions are integral to the evaluation process as they shape its scope and influence its outcomes. Architecture evaluation involves a systematic approach to pinpointing problems and risks within system architecture. It serves to evaluate the quality and effectiveness of a system's design. Through this process, potential risks, trade-offs, and opportunities for enhancement in system architecture can be identified.

The second factor to consider is your evaluation context. What are the constraints and assumptions that influence your evaluation process and outcomes? For instance, what are the project's deadlines, budget limitations, available resources, key stakeholders, and associated risks? Depending on your context, you may need to tailor or customize your evaluation method to align with your specific situation and requirements. For example, if you're dealing with a tight deadline and a restricted budget, you might opt for a rapid and cost-effective evaluation method like prototyping or expert review, rather than a more rigorous and costly approach such as formal analysis or experimentation.

The evaluation context shapes what methods you can use. With tight deadlines, rapid prototyping or expert reviews may be best. With time and budget, more rigorous quantitative methods like load testing make sense. Adapt your approach to fit the constraints you face.

Time and budget are always not enough for any projects, thus consider those two factors during the evaluation process and then scale it for the stakeholder’s needs. Spiral methodology for safety critical systems, RAD or Agile for time critical systems, and rigorous and expensive formal process for mission critical systems. However, there is no magic bullet.

To understand large scale system, it's important to apply various complementing methods. The real win is uncovering new insights and sharing them with the right people, no matter the approach used.

I'd also add Maintainability, Cost, Integration , Future-proofing, User Experience snd organizational requirements. Cost: Assess the financial implications of implementing and maintaining the system architecture. Integration: Evaluate how well the architecture can integrate with existing systems and future technologies. Consider factors like compatibility, interoperability, and the availability of APIs or standard protocols. Future-proofing: Assess whether the system architecture can support future business requirements and technological advancements. Consider factors like flexibility, adaptability, and the ability to handle emerging trends.

The third factor to consider is your evaluation methodology. How do you plan, conduct, and report your evaluation activities and results? Depending on your chosen methodology, you may need to adhere to different steps and guidelines to ensure the validity, reliability, and utility of your evaluation. For example, if you opt for a quantitative evaluation method, such as simulation or experimentation, you may need to define key elements such as evaluation criteria, hypotheses, variables, scenarios, data collection methods, data analysis methods, and confidence intervals...

All the mentioned methods have pros and cons. Often a combination works best. Prototyping finds usability issues early. Expert reviews leverage others' knowledge. Simulation predicts scalability. Formal analysis ensures completeness. Experimentation evaluates real-world performance. Use the right mix for your goals and context.

Simulation is also useful to evaluate the expected architecture outcome - if, for example, the design should improve some metric of performance, a model that estimates why this is the expected outcome can be a rough KPI for the results. For example: 1. When we tried to optimized the memory use of multithreaded C++ code, we modeled the allocation scheme and evaluated the impact of several approached - this led to a 95% reduction in memory use. 2. When we evaluated a proxy server, we modeled the connection allocation scheme in order to figure out the expected maximum number of concurrent connections to find the theoretical ceiling, and then compared it against bare metal instance. This allowed us to optimize the configuration for the proxy.

The evaluation method imho is defined as part of the high level architecture. Often it will be combinatorial and for really innovative systems, there may not be a solution to measure with yet, that may have to be created as part of the process.

Everything already listed and more. Context also plays an important role here. For example, is it innovative, transformative or essential? The best methods are often a combination of those listed. The key differentiator then, lies in how we work those combinations for maximum benefit. The best ones I have seen, often use context and clarity to smooth out the differences, reducing risks along the way. The biggest mistakes stem from operating in pure isolation, with no overarching concept of why "this way/this method" is being tried & tested. To answer that, needs exploring how it all works fits together, an answer that is never that simple in practise.

Design requirements are often unclear, making it impossible to anticipate every evaluation criterion. Once the product is in users' hands and tested in real-world conditions, new factors for assessment come to light. Therefore, it's crucial to continue evaluating the product even after launch, and to incorporate user feedback into future designs.

I would like to pose a question to all contributors... as has been mentioned, there are often deadlines issues that might prevent all but expert references and requirements can be poorly presented leading to varied interpretations (especially when the customer is an external party). So my question... knowing that their will always be requirement changes based on user feedback in future releases, how much effort do you put into nailing the original architecture vs getting close based on expert input, then turning your focus to making sure your architecture is adaptable enough to easily accommodate those changes. From a DB perspective I ofton see applications designed such that it greatly limits the ability to seamlessly absorb these changes.

I agree cost, accuracy, scope and usability are key criteria. Additionally, consider how understandable and actionable the results are to stakeholders. The best evaluation provides clear, targeted insights to guide architecture improvements.

Pretty much explained above. Various factors like Scalability, Maintainability et al. I would emphasize on the cost-effectiveness of the system architecture should be carefully evaluated. This involves analyzing both the upfront costs and the long-term maintenance costs of the system. Organizations should consider factors such as hardware and software requirements, licensing fees, and ongoing support and maintenance costs when evaluating the architecture in terms of its cost-effectiveness.

Ensuring decent coverage across criteria, both essential and non-essential goes a long way into ensuring adequate stakeholder engagement. To ensure proper coverage, I use (a rather cheekily named acronym), so be careful while using it. It is in briefly outlined below, expand detail as needed. CUSS criteria: Cost, (Build, Operate, Maintain), Usage (Usability, Design, Accessibility), Scope (Interoperable, Extensible, Performance), Security considerations. I use it to being a sense of humour to an otherwise dry discussion and keep it as succinct as possible. The key is driving stakeholder engagement during the entire process.

?? I suggest implementing Architecture Decision Records (ADRs) using Martin Fowler's template. ?? These lightweight records documented the current state, constraints, decisions, and status of our architecture choices. ADRs proved immensely beneficial, providing a centralized reference for past decisions, facilitating onboarding for new team members, and promoting rigorous design thinking. ?? The use of ADRs fostered better design hygiene, resulting in increased alignment, improved reasoning, and faster onboarding. ADRs are recommended as a lightweight best practice for complex system designs.

Extensibility is equally critical. while evaluating system architecture, it is important to consider factors beyond technical specifications. Performance, security, scalability, and maintainability all contribute to the effectiveness and longevity of the system. By addressing these additional considerations, software architects can ensure that the system will not only meet present requirements but also adapt to future needs and challenges.

A lot can be said on all the above points to make sure one doesn’t miss anything… and we all know that there are many -ilities to consider including legal aspects. But one thing that always strikes me when I work with architects is that they don’t consider human aspects: you can produce the best architecture design, the implementation can be disastrous if you have not considered what the teams are capable of doing in term of time and resource capacity but also knowledge and skills. You can’t design an event-based architecture for example with developers that are used to monoliths and procedural approaches. You can, but you have to account for training and coaching.

In the mobile-driven landscape, a CTO's priority is a user-centric, reliable mobile experience. This involves developing with a mobile-first mindset, using progressive web app capabilities, and leveraging cloud services for scalability. For backend applications, which are the backbone of digital services, a robust, scalable, and maintainable system is essential. Microservices architecture, which breaks applications into smaller services communicating via APIs, promotes rapid deployment and scalability. DevOps integration further enhances efficiency. Lastly, for high-load web platforms, CTOs should implement CDNs and load balancers for swift content delivery and to manage peak traffic, ensuring an uninterrupted user experience.

I will keep this simple, since a lot of great ideas are already listed. Here is the main thing to keep in mind. "How does this contribute towards the finalised end goal?" There are a lot of nuances there, since things will change along the way. The earlier one can catch those changes the better chance you have to reduce risks, of all kinds.