Agile is Becoming Futile, Impact Engineering is Here to Stay

I’ve been practicing Agile for the last 14+ years. Before that, I worked on waterfall model projects for a while. If you look at the broader landscape of the software development life cycle, Agile methodology is still dominant around the globe. But have you ever wondered what comes next when Agile crumbles? Nowadays, people have been voicing many valid concerns about Agile methodology and looking for a better alternative. I think it's better to stay ahead of the curve when you are a software engineer. That’s why it is essential to understand the rising concept named "Impact Engineering" methodology and how it differs from Agile. As Agile is becoming less effective day by day, Impact Engineering is going to thrive. You won't find many writings or diagrams on this topic except for the book "Introduction to Impact Engineering ". I purchased and read it, and it helped me develop a different but enriching perspective on software development. That motivated me to share my learning and articulate the entire concept in concise, lucid terms with the help of some self-drawn diagrams. I would surely recommend this book for your perusal. Please share this article with your network if you value its content.

If you find it insightful and appreciate my writing, consider following me for updates on future content. I'm committed to sharing my knowledge and contributing to the coding community. Join me in spreading the word and helping others to learn. Follow WebWiz: https://www.dhirubhai.net/newsletters/webwiz-7178209304917815296jur3il4jlk

Agile: A Flexible Approach with Pain Points

Agile is an SDLC methodology that emphasizes flexibility, collaboration, and rapid delivery. It is based on the principles of iterative and incremental development, where the project is broken down into smaller, manageable chunks, and each chunk is developed and tested in a continuous cycle. Agile highlights the importance of continuous improvement, adaptability, and customer satisfaction.

Agile's Pain Points

There are many pain points, as you might agree. Let me cite the top three prominent reasons why major players in the IT industry are losing faith in Agile.

1. Lack of Focus: Agile's focus on rapid delivery can lead to a lack of focus on the overall business goals and requirements, resulting in software that may not align with the organization's objectives.
2. Higher Maintenance Costs: Agile's iterative approach can lead to higher maintenance costs due to the continuous changes and updates.
3. Managing Stakeholder Expectations: Agile’s focus on rapid delivery can lead to high expectations from stakeholders, which can become difficult to manage at times.

Overview of the Horizon IT System Failure

The Horizon IT system, developed by Fujitsu and launched in 1999, was intended to modernize the operations of the UK Post Office. However, it quickly became notorious for its flaws, leading to wrongful accusations against over 700 sub-postmasters, with many facing criminal charges based on faulty data from the system. The scandal resulted in severe consequences, including wrongful imprisonments and significant personal and financial distress for those affected. The failure of the Horizon IT system at the UK Post Office has had a significant impact on the reputation of Agile methodologies. This case serves as a cautionary tale, illustrating the potential pitfalls of Agile when its principles are not effectively implemented.

The Horizon project exemplified a failure to adhere to key Agile principles:

1. Customer Collaboration: Agile emphasizes collaboration between developers and end-users. In the case of Horizon, the true users—subpostmasters—reported numerous issues with the system, but their feedback was largely ignored. This lack of engagement led to a system that did not meet user needs.
2. Working Software: Agile prioritizes the delivery of functional software. However, Horizon failed to deliver reliable working software, as it was riddled with bugs that were known to management but not adequately addressed.
3. Responding to Change: Agile encourages adaptation to changing requirements. The Horizon project did not effectively accommodate feedback or changes, resulting in a system that was not fit for purpose.

The Horizon IT scandal has contributed to a growing skepticism about Agile methodologies. A study indicated that software projects adopting Agile practices are 268% more likely to fail than those that do not, raising questions about the effectiveness of Agile principles in large-scale projects.

Impact Engineering: A Solution to Agile's Pain Points

Impact engineering is a software development approach that focuses on creating a lasting impact on the business or organization. It involves designing and building software that drives significant business value, often by solving complex problems or improving processes. Impact engineering emphasizes the importance of understanding the business goals and requirements, and then using that knowledge to create software that delivers tangible results.

Key Differences

There are some distinct differences between "Impact Engineering" and Agile. Let’s understand the top three differences between them.

1. Goals: Impact engineering focuses on creating a lasting impact on the business, while agile focuses on delivering working software in short cycles.
2. Methodology: Impact engineering involves a more structured approach, with a clear understanding of the business goals and requirements. Agile, on the other hand, is more flexible and adaptable, with a focus on continuous improvement.
3. Outcomes: Impact engineering aims to deliver software that drives significant business value, while agile aims to deliver working software in short cycles.

Fundamental Principles

The principles of impact engineering provide a structured framework for developing solutions that not only meet technical requirements but also deliver significant value to stakeholders. By focusing on alignment with business goals, user-centric design, and continuous improvement, impact engineering aims to create lasting positive effects in engineering projects. Now, let me state five fundamental principles that unperpin this whole concept.

1. Robust Requirements Engineering

A central tenet of Impact Engineering is the emphasis on a thorough requirements engineering process. This involves:

• Defining Clear Requirements: Projects that start with well-documented requirements are significantly more likely to succeed. Research indicates that having clear initial requirements can boost the likelihood of project success by up to 97%.
• Minimizing Late-Stage Changes: Reducing significant changes to requirements late in the development process can enhance success rates by 7%.

2. Psychological Safety

Creating an environment where team members feel safe to voice concerns and address issues is crucial. Studies show that when software engineers feel psychologically safe, their projects are 87% more likely to succeed. This principle encourages open communication and collaboration among team members, fostering a culture of problem-solving and innovation.

3. Focus on Real-World Problems

Impact Engineering prioritizes addressing real-world problems over abstract or theoretical challenges. This involves:

• Understanding User Needs: Engaging with end-users to understand their needs and challenges ensures that the solutions developed are relevant and effective.
• Iterative Problem-Solving: The methodology encourages iterative cycles of development where solutions are tested and refined based on user feedback, leading to more impactful outcomes.

4. Enhanced Agile Principles

It sounds a bit weird but it incorporates some enhanced Agile principles to address its limitations. Key Agile principles include:

• Working Software Over Comprehensive Documentation: This principle remains vital, but Impact Engineering stresses the importance of documentation that accurately reflects requirements.
• Customer Collaboration Over Contract Negotiation: Engaging customers throughout the development process is essential, ensuring that their input shapes the final product.
• Responding to Change Over Following a Plan: Flexibility is important, but it should be balanced with a structured approach to managing changes in requirements.

5. Emphasis on Team Dynamics

Impact Engineering recognizes the significance of team dynamics in project success. This includes:

• Collaboration and Trust: Building trust among team members enhances collaboration and leads to better problem-solving.
• Managing Workloads: Addressing issues of burnout and workload management is essential to maintain team productivity and morale.

I've drawn this diagram to help you grasp the summarized version of it.

Pros:

• Business Focus: Impact engineering ensures that the software development is aligned with the business goals and requirements.
• Tangible Results: Impact engineering delivers software that drives significant business value, making it a more effective approach for organizations seeking tangible results.

Cons:

• Structured Approach: Impact engineering's structured approach can be inflexible and may not adapt well to changing requirements.
• Higher Costs: Impact engineering often requires more resources and expertise, which can increase costs.

Real-World Challenges

To be honest, this is not battle-ready yet, but it will eventually become so because of its great potential. Still, it is important to understand the current challenges it poses at present.

1. Balancing Business Goals and Technical Requirements: Impact engineering requires balancing business goals with technical requirements, which can be challenging, especially in complex projects.
2. Managing Stakeholder Expectations: Agile's focus on rapid delivery can lead to high expectations from stakeholders, which can be difficult to manage.
3. Scaling Agile: Agile is often effective for small to medium-sized projects, but scaling it to larger projects can be challenging, especially in terms of communication and coordination.

How Impact Engineering Solves Agile's Pain Points

The research conducted for the book "Impact Engineering" revealed alarming statistics regarding agile project failures. It found that 65% of software projects using agile principles fail to meet deadlines, budgets, and quality standards. In contrast, projects employing the impact engineering approach exhibited a failure rate of just 10%. This stark difference underscores the potential of impact engineering to resolve the common pain points associated with agile methodologies that I stated above in this article.

1. Bring Focus on Business Goals: Impact engineering emphasizes a clear understanding of business goals and requirements at the outset, ensuring that software development is aligned with the organization's strategic vision.
2. Low Maintenance Costs: Impact engineering reduces maintenance costs by focusing on delivering software that meets well-documented requirements, thereby reducing the need for frequent revisions and minimizing long-term maintenance expenses.
3. Managing Stakeholder Pragmatic Expectations: Impact Engineering addresses this by establishing a robust requirements engineering process that clarifies project scope and objectives, helping to manage and align stakeholder expectations more effectively and pragmatically.

Key Findings from the Study

• Importance of Documented Requirements: The study indicated that projects with documented requirements before development are 50% more likely to succeed. This finding highlights the value of impact engineering's structured approach to requirements gathering.
• Psychological Safety: Projects where software engineers felt psychologically safe to address issues promptly were 87% more likely to succeed. This aspect is integral to impact engineering, which fosters an environment where team members can openly discuss challenges and collaborate on solutions.
• Real-World Problem-Based Requirements: Accurate, real-world problem-based requirements enhanced the probability of project success by 54%. Impact engineering prioritizes understanding and addressing real-world challenges, ensuring that the developed software is practical and effective.

The case study illustrates that impact engineering can effectively resolve many challenges associated with agile methodologies. By focusing on clear business goals, reducing maintenance costs, and managing stakeholder expectations, impact engineering offers a robust alternative to agile practices that often fall short. The findings from the research emphasize the need for a structured approach to requirements engineering and the importance of fostering a psychologically safe environment for engineers, ultimately leading to more successful software projects. This case study serves as a compelling example of how impact engineering can outperform agile in delivering high-quality software on time and within budget.