What are the best practices for distributed system architecture maintenance?

It is important to note that designing for change does not mean designing for just in case scenarios. In many situations especially platforms and frameworks, all possible use cases will not be known in advance. This is the very nature of platforms. The focus must rather be on enabling “just in time” evolution of the system as new use cases come to light. While there is no standard measure for this, “refactorability” is a quality attribute that can be used to approximate how evolution-friendly a system is. The measure of refactorability would loosely be how much effort it takes to refactor a significant component of the system. Of course this is a very system and team specific measure.

Best practices for maintaining a distributed system architecture may include: 1. Monitoring and Logging 2. Regular Updates and Patching 3. Capacity Planning 4. Backup and Disaster Recovery 5. Load Balancing: Distributing the workload across multiple servers or nodes can improve performance and fault tolerance. 6. Security: Distributed systems must address security concerns such as authentication, authorization, and encryption. Implementing security measures such as firewalls, intrusion detection systems, and access control can help protect the system from unauthorized access and data breaches. 7. Performance Optimization 8. Scalability: Distributed systems

Software solutions are evolving creations, we start with simple components and modules that fulfill today's needs and requirements and expect it to continue growing during its lifetime that might expand between seven to twenty years. Yes, some solutions may expand over twenty five years keeping evolving through time. A proper design of a distributed system should consider and accept the fact that things will change over time, new features will be introduced, new technologies, new environments, and new behaviors. Considering change from the start is a key factor to achieve a successful distributed system design and architecture.

The changes can be predictable or unexpected. The correct design prepares the system for expected changes (tech evolution, some emerge tech that in the design was not adequate but later some time becomes more robust ...) and unexpected (bugs or new features), in this case modularity, policies and loose coupling becomes a key for solving a lot of these situations.

In this regard it is important that monitoring capabilities should not be an afterthought but rather a should be a non-functional requirement and deliberately designed for from day one. If implemented early on monitoring will help in iterative development of the system even during early stages of development well before the system is put to use in production.

As we mentioned before in other arctics on solutions architectures, distributed system are of no difference; you need to watch your system in live operations even more than you monitor it while in development phases. Experiment and elect a proper tool or stack of tools that suites your situation. It is important to provide a real-time monitoring that enables log aggregation, performance metrics measurements, and traceability. Such tools by themselves need to be well thought, architecture, and designed to serve the specific purpose of each distributed system.

Deployments should be fully repeatable via CI/CD. This means that the whole infrastructure should be deployable via such system. One key thing to keep in mind is that these run has to be idempotent (the same code should produce the exact same result even if it is run 100 times). In addition, infrastructure code needs to be tested in many ways as we do in software. Unit tests can be done with tools like Ansible-test. Secondly, the testing strategy could be augmented with integration tests in a staging environment. Last, it is even better to also add some user acceptance tests. These tests mimic user calls against a minimal copy of your real infrastructure.

Install automation is something that is worth investing in early on. “Build the installer first” is an approach that would pay rich dividends because the ability to setup and test down the system at will allows developers to experiment with and test the system in isolation thereby improving the quality of every increment of the system that is delivered. Note that “install automation” is used in a broad sense here irrespective of whether the “installer” is a deployment pipeline that is used to deploy cloud services; or an ISO image that is used to install directly on to hardware. The key is that it must be possible to bring up a fresh instance of use system at will.

While they usually start small enough to be well managed, distributed systems grow and evolve; and they grow wildly fast! You need to be able and be prepared for managing a reasonably large to huge set of components in a distributed system. Regardless of your expectations that the system will remain within controllable size limits; plan for automations at all levels: development through pipelines, deployments via CI/CD, runtime utilizing orchestration tools. Automate all aspects from the beginning: requirements elicitation, design, development, testing, deployment, monitoring, etc. It is a very good idea to use a runtime horizontal scalability using something like Kubernetes. Assess well the tools you select; as they might become a burden!

A robust versioning system is key during development and testing - you must be absolutely sure of what you are testing at all tiers of a distributed system. It should be easy to identify each piece of the system at run time and that capability requires design and implementation that impacts the entire build chain and runtime environment. I have wasted time testing new features against the wrong versions of cooperating components and gotten stuck thinking my component was in error when another component was an early version without full functionality. I assumed it was recent simply through a file date check, not truly knowing the version.

Environments differ and may offer different levels of reliability, scalability, availability, security, and networking capabilities. Do not commit to a new runtime environment without testing your solution behavior within that environments specific perimeter. While you may have reached maturity level running your distributed system in one environment; moving to a new runtime environment might shock you with unseen issues and hidden design flows. The bottom rule is never to trust neither the system nor the runtime environment; always test and verify.

Unit tests are always overlooked. Unit tests can cover -ve scenarios very well with mocking. It's easy to run and fast, 100% can be automated without any dependencies. Even with external service interactions, in a distributed system, we can capture request response (record) and put it mocks and use them unit tests. Always remember the testing triangle. TDD is also very useful in promoting writing tests before the actual code.

working with distributed systems for a good period of time is a great opportunity for a software engineer to learn and improve. You do not only learn about and improve your specific use case of a distributed system; you also learn how to make the analogy building other similar distributed systems. Improvement is not only around the system itself but your capacity as a software engineer.

In my day to day operations about improving systems, security should also be a key factor to consider. The more the gateways the system can accommodate as part of improving it, the larger the security threats and risks. Improving the system may incorporate various approaches from the feedback loops and thus developers have to also factor in availability, consistency and speed before doing any adjustments.

One way to constantly improve a distributed system in an evidence based fashion is to establish a feedback loop from operations back to architecture. Significant improvements can be conceptualised when the people involved in architecting the system are familiar with the actual issues being observed in production especially at scale. The feedback loop can be as simple as a periodic operational review in which architects and senior engineers participate. Root cause analysis of production incidents can also provide signals that trigger architectural improvements.

Especially for internal project and developer documentation, tools that support “documentation as code” must be favoured. This is because including documentation into the regular coding workflow greatly improves the chances of documentation being present, up to date and helpful. Documentation changes can be included in code reviews and can be committed alongside code changes.

Design Docs, decsion records, architecuture, APIs must be documented. However, I would encourage code should be self readable via proper variables, modularization etc. Code comments are preferred when there's a ambiguous choice taken etc. E.g. A sleep statement in the code always raises questions. You should justify that in comments.

Managing a big infrastructure is hard. For instance, I have been in an infrastructure including database systems, message queue cluster, physical and software defined networks, authentication services, and many other APIs. Sometimes, like in our case, all this is managed by a single team where everyone is expected to be part of the on-call. For your team to be successful in such environment, it is important to implement a culture of runbooks. Runbooks are procedure which describe a known issue in great details and provides steps to resolve it. When possible, the runbook can be automated (bundled as a single command) and be able to be performed via the CI/CD system. It is better to encourage everyone in the team to write/improve runbooks.