The Most Important Engineering Principle You've Never Heard Of

Having led major digital transformations across multiple organizations, I've witnessed firsthand the critical importance of understanding existing systems before changing them. Whether consolidating monitoring tools, modernizing deployment pipelines, or automating manual processes, I've learned that successful transformations require us to first understand why things are the way they are. This wisdom, as it turns out, was articulated over a century ago in a principle that every technology leader should know.

The Wisdom of Chesterton's Fence

G.K. Chesterton once wrote, "Don't ever take a fence down until you know why it was put up." The metaphor describes a scenario where one encounters a fence without an apparent purpose. While the impulse might be to remove it, Chesterton advises first understanding its original intent. This simple observation has profound implications for software engineering, where we're constantly tempted to tear down and rebuild the systems we inherit. Before making changes to a system, it's crucial to understand the original design intentions and the potential consequences of those changes. This thorough understanding helps prevent unintended side effects that could compromise system integrity or performance.

This doesn't mean we can't change things – it means we must understand before we act.

Consequences of Overlooking the Principle

A stark example of neglecting this principle is the 2012 incident involving Knight Capital Group. The firm deployed new trading software without fully understanding its interaction with existing code. This oversight led to a loss of $460 million in just 45 minutes, as the system executed numerous erroneous trades due to reactivated obsolete code.

Similarly, Microsoft's Windows Vista faced significant challenges because changes to security and driver models were implemented without a thorough grasp of how existing applications depended on previous behaviors. This resulted in compatibility issues, poor adoption rates, and damage to the company's reputation.

Successful Applications of the Principle

Conversely, organizations that have embraced the wisdom of Chesterton's Fence have reaped substantial benefits. Here are a few examples that are widely known and documented in the industry.

• Amazon's Microservices Transition: Under Jeff Bezos's directive, Amazon required teams to document all existing service interfaces and dependencies before transitioning to a service-oriented architecture. This meticulous approach facilitated one of the most successful architectural transformations in the tech industry.
• Google's Python Migration: The shift from Python 2 to 3 was a multi-year endeavor. By first understanding the rationale behind each piece of code, Google ensured critical behaviors were preserved while integrating modern capabilities, setting a precedent for other organizations facing similar transitions.
• NASA's Space Shuttle or ISRO Launches: Primary Avionics Software System (PASS) exemplifies exceptional reliability in mission-critical software. With approximately 500,000 lines of flight source code, PASS maintained an error rate significantly lower than the industry average, contributing to only one software-related issue in 135 missions. This conservative approach to changes ensured mission success and safety, preventing potential failures that could have cost billions of dollars and endangered human lives. ISRO (Indian Space Research Organization), where I worked, and in other high-risk, high-cost programs like NASA have clear detailed checklists based on RCA ( Root Cause Analysis) and lessons learned, to not repeat mistakes.
• Gmail's Infrastructure Evolution: Instead of opting for a complete overhaul, Google studied Gmail's existing architecture and implemented gradual changes. This strategy allowed them to scale from approximately 1 million users to over 176 million without major disruptions, maintaining 99.9% uptime.

I have several examples like this from my projects across organizations, maintaining high-quality products and exceptional customer experience—both internal and external—done with a thorough evaluation of existing systems during transitions, redesigns, and migrations, such as moving from on-premise to cloud solutions. By analyzing the rationale behind current practices, aligning requirements with new plans, and rigorously testing for compatibility, tools, processes, and automation evolved significantly. This helped document and maintain system reliability through these changes, providing valuable insights for engineers and leaders using these systems. Each new tool and process was built upon documented lessons from production experience, ensuring that changes enhanced reliability without disrupting established workflows.

Guidelines for Engineering Leaders

Here are my inputs on How To effectively apply Chesterton's Fence in engineering projects:

1. Document the Current State: Map out all dependencies, understand potential failure modes, identify stakeholders, and review historical incidents.
2. Research Requirements: Consider regulatory needs, business constraints, performance metrics, and reliability expectations.
3. Plan the Evolution: Develop comprehensive test plans, maintain rollback options, implement changes gradually, and monitor for unforeseen behaviors.
4. Conduct Post-Implementation Reviews: Evaluate whether changes achieved the desired outcomes, identify any unintended consequences, and document lessons learned for future reference.

Balancing Caution with Innovation

While it's crucial to understand existing systems, it's equally important to avoid excessive caution that may hinder progress. Engineering leaders can balance this by:

• Setting Clear Objectives: Define the goals of proposed changes and ensure they align with overarching business needs.
• Time-Boxing Investigations: Allocate specific timeframes for understanding systems to prevent analysis paralysis.
• Embracing Iterative Approaches: Implement small, incremental changes that allow for continuous learning and adaptation.
• Fostering a Culture of Experimentation: Encourage controlled experiments to test new ideas while minimizing risks.
• Regularly Reassessing Legacy Systems: Schedule periodic reviews to identify opportunities for modernization and improvement.

Conclusion

In my years of leading technology transformations, I've learned that the most successful transformations stem from a deep understanding of existing systems before implementing changes. Whether you're consolidating tools, automating processes, or modernizing infrastructure, don't just blow it up or run the chainsaw through it to show that you are bringing change; take time to understand why things are the way they are. By adhering to the principle of Chesterton's Fence, engineering leaders can ensure that innovations enhance rather than disrupt, leading to resilient, efficient, and future-proof systems.

Reflect on your own organization: Are there "fences" in your systems whose purposes aren't fully understood? Before initiating changes, consider investigating their origins to ensure that well-intentioned modifications don't lead to unintended consequences.