Adoption of Chaos Engineering

Fantastic article by Clifford Chetty about?Rabobank's?adoption of Chaos Engineering & their quest for Resilience (unleash-that-chaos-engineering).

This is a summary by Lucas Rincon & me from our weekly learning sessions.

Setting the stage

Resilience is crucial for a competitive edge, signifying the ability to harmonize numerous moving parts into a cohesive whole, not just advanced engineering practices or architectural considerations.

Architects traditionally focus on non-functional discovery processes to build resilience into their designs. However, with increasing complexity, new approaches are required. Chaos engineering is a practice that builds confidence in a system's ability to withstand turbulent conditions and scale effectively.

Benefits

Higher Availability/Reliability:?It is essential to detect failures before customers do. This can help you avoid any kind of embarrassment and ensure uninterrupted service.

Cost Savings:?Discovering defects in production can be pretty expensive. It is 3X cheaper to address issues during the build cycle than to fix them in production.

Risk Mitigation:?By transforming the unpredictable into the predictable, you can earn the appreciation of risk, compliance, security, and legal teams.

Innovation & Fascination:?This is an ideal solution for those intrigued by breaking things to understand how they work, particularly for SREs.

Tools

Since Netflix's Chaos Monkey in 2011, many tools have become available, each offering unique functionalities. If you want to learn more about the landscape, ping me.?

Process

Define a Steady State:?Establish non-functional values/requirements as the foundation.

Formulate Hypotheses:?Identify potential failure scenarios across various levels of the system.

Design Experiments:?Select hypotheses, define scope, & identify metrics for evaluation.

Execute/Verify/Learn:?Quantify experiment results to gain insights.

Implement Fixes:?Incorporate learnings into development cycles, prioritizing fixes based on severity.

BEST PRACTICES

It's essential to communicate clearly with operational stakeholders before conducting experiments. This can help mitigate potential backlash.

To effectively simulate real-world conditions, experiments should be conducted in production environments.?

Integrating chaos experiments into DevOps pipelines can streamline the process and ensure consistency.?

It's also crucial to minimize and control the impact of experiments to prevent widespread disruptions.?

By embracing chaos to achieve resilience, organizations can position themselves at the forefront of innovation and be prepared to navigate the complexities of modern systems with confidence and agility.