Is Agile Still Agile? Evolving Software Development Through Flexibility and Continuous Improvement

Introduction

Just over twenty years ago the Agile Manifesto was created, and Extreme Programming was a hot topic, on everyone's lips and bookshelves. Influenced by these ideas and practices, in 2001, I introduced three fundamental tools into Oracle Healthcare Applications Group. I called these The Three Pillars of Software Development. They were, simply, a good source code control system (originally CVS, then Subversion), a continuous integration tool (in this case Cruise Control), and a unit testing framework (JUnit). A fourth tool, the Clover code coverage tool, was introduced a little later.

Alright, so obviously I didn't introduce source code control, Oracle already had aRCS, an implementation of RCS that was hosted remotely in the United States. We hosted CVS (later Subversion) locally and pushed the changes to the main aRCS repository nightly, thanks to our local operations team. Maybe I could say we introduced a modern source code control system. Making this happen wasn't as easy as you might think. Change always meets some resistance. But that is an entire story unto itself, one for another day.

Extreme Programming was emerging rapidly at the time, and we adopted the "all tests must pass" rule. I was also influenced by an article by Ron Jeffries called "Extreme Testing" that was published in 1999. I created a unit testing "cheat sheet" to help developers get started. The unit testing regime and the continuous integration tool became fundamental components of development. After a few months no one questioned the value of the newly introduced unit testing regime, which was originally met with some scepticism. Our little Australian Development Centre brought these modern tools to the Oracle giant by way of our Healthcare Applications Development teams, and the tools and approach were ultimately adopted by the entire group. Oracle had become the Agile Giant (at least from the perspective of the Healthcare Applications Group). Often change starts with one, becomes a team, and ends with a community. It was a time when the future of Agile, and practices such as Extreme Programming, looked bright. Amazingly, twenty years later, I still see organisations struggle to implement and utilise these tools and incorporate them into what they would call a development process. But that's yet another story.

The remaining section of the article offers a perspective that is neither right nor wrong. Its intention is to take a provocative position to encourage critical thinking and debate. Ultimately, I hope you enjoy it.

Is Agile still Agile?

Over the last two decades, the Agile methodology seems to have been increasingly co-opted by organisational bureaucrats, draconian managers, and over-zealous project managers, leading to a transformation that diverges significantly from its original intent. Agile was conceived as a flexible, iterative approach aimed at fostering collaboration, responding to change, and delivering continuous value. However, as it has gained popularity, many organisations have implemented it with an emphasis on formal processes and rigid structures.

The original Agile Manifesto, created in 2001, emphasises “individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.” These principles were intended to create a dynamic and adaptive work environment where teams could rapidly respond to changing requirements and stakeholder feedback. Unfortunately, as Agile has been adopted more widely, there has been a tendency to impose traditional project management frameworks onto Agile practices.

This co-option often manifests as an overemphasis on formalised ceremonies, extensive documentation, and strict adherence to predefined processes. Daily stand-ups, sprint planning, retrospectives, and other Agile practices, which were originally meant to facilitate communication and improvement, have in many cases become checkboxes in a rigid procedural system. Some managers, keen on maintaining control and predictability, often impose detailed planning and reporting requirements that stifle the flexibility and responsiveness that Agile seeks to promote.

Moreover, the scaling frameworks like SAFe (Scaled Agile Framework) have sometimes contributed to this problem by introducing hierarchical structures and complex processes that resemble the very bureaucracies Agile was designed to negate. While these frameworks aim to enable Agile practices at scale, their implementation often leads to a dilution of the core Agile values. Teams find themselves bogged down by layers of management oversight and procedural adherence, which can hinder innovation and responsiveness.

The co-option of Agile has resulted in a process-heavy methodology that stands in stark contrast to the original Agile Manifesto. By prioritising control and predictability over flexibility and collaboration, many organisations have turned Agile into a rigid system that fails to deliver on its promise of adaptability and continuous improvement. To reclaim the true spirit of Agile, organisations need to return to its foundational principles and focus on creating environments where teams are empowered to innovate and respond to change effectively.

So, what is the solution? Strap yourselves in and get ready for the ride.

Shu Ha Ri

Shu Ha Ri is a Japanese martial art concept that describes the stages of learning through to mastery. People who are learning and mastering new skills pass through three stages of behaviour: following, detaching, and fluent. Alistair Cockburn (2000, 2002) introduced it as a framework for understanding, and learning techniques and methodologies in software development.

1.???People in the following stage look for one procedure that works. Even if ten procedures could work, they can’t learn ten at once. They need one to learn first, one that works. They copy it; they learn it. In this stage, practitioners measure success by (a) whether the procedure works and (b) how well they can carry out the procedure.

2.???In the detaching, or Level 2, stage, people locate the limitations of the single procedure and look for rules about when the procedure breaks down. They are actually in the first stage of a new learning; namely, learning the limits of the procedure. The person in the detaching stage learns to adapt the procedure to varying circumstances. He is now more interested in learning the ten alternative procedures, in learning when each is most applicable and when each breaks down.

3.???In the third, fluent stage, it becomes irrelevant to the practitioner whether he is following any particular technique or not. His knowledge has become integrated throughout a thousand thoughts and actions. Ask him if he is following a particular procedure, and he is likely to shrug his shoulders: It doesn’t matter to him whether he is following a procedure, improvising around one, or making up a new one. He understands the desired end effect and simply makes his way to that end.

-- Alistair Cockburn, Writing Effective Use Cases (2000) & Agile Software Development (2002)

What does martial arts have to do with it?

In this next section, consider Bruce Lee’s philosophy around Jeet Kune Do. We'll substitute the word Agile for Jeet Kune Do, and make some changes to terminology (italicised) in the following quote to reflect its applicability to software development.

"I have not invented a ‘new methodology’, composite, modified or otherwise that is set within distinct form as apart from ‘this’ method or ‘that’ method. On the contrary, I hope to free my developers from clinging to methodologies, patterns, or moulds. Remember that?Agile is merely a name used, a mirror in which to see ‘ourselves’. Agile is not an organised institution that one can be a member of. Either you understand or you don't, and that is that. There is no mystery about my methodology. My processes are simple, direct and non-classical. The extraordinary part of it lies in its simplicity. Every process in?Agile?is being so of itself. There is nothing artificial about it. I always believe that the easy way is the right way.?Agile is simply the direct expression of one's feelings with the minimum of processes and energy. The closer to the true way of Software Development, the less wastage of code there is. Finally, an?Agile?man who says?Agile is exclusively?Agile?is simply not with it. He is still hung up on his self-closing resistance, in this case anchored down to reactionary pattern, and naturally is still bound by another modified pattern and can [only] move within its limits. He has not digested the simple fact that truth exists outside all moulds; pattern and awareness is never exclusive. Again, let me remind you?Agile is just a name used, a boat to get one across, and once across it is to be discarded and not to be carried on one's back."

-- Bruce Lee, Tao of Jeet Kune Do, 1975 (modified terminology italicised)

The final statement aligns with the fluent stage of Shu Ha Ri. At times it can appear that there is no apparent process when mastery is reached. A high performing team may appear to be lacking process, but they know what to do, they are very efficient, and are highly productive (Cockburn, 2000 & 2002).

Lee also encourages us to “be like water”. Following from the above, using Lee’s words and substituting appropriate words to align with software development:

“Development methodologies should be as flexible as possible. Water can be used as an analogy for describing why flexibility is a desired trait in software development. Water is infinitely flexible. It can be seen through, and yet at other times it can obscure things from sight. It can split and go around things, rejoining on the other side, or it can crash through things. It can erode the hardest rocks by gently lapping away at them or it can flow past the tiniest pebble. A development methodology should have these attributes. Agile developers reject traditional systems because of the lack of flexibility. Agile is a dynamic concept that is forever changing, thus being extremely flexible. ‘Absorb what is useful; Disregard that which is useless’. Agile developers are encouraged to study every form of process possible. This is believed to expand one's knowledge of other development systems. ‘Empty your mind, be formless, shapeless, like water. If you put water into a cup, it becomes the cup. You put water into a bottle, and it becomes the bottle. You put it in a teapot it becomes the teapot. Now, water can flow, creep, drip or it can crash. Be water my friend.’”

-- Bruce Lee, Tao of Jeet Kune Do, 1975 (modified terminology italicised)

The above quote emphasises the power of flexibility and adaptability of a methodology, which can be applied to software development methodologies. In fact, it can be applied to any methodology, to "absorb what is useful and discard that which is useless", and in doing so create a superior and efficient way of doing things.

Absorb what is useful

Again, applying Lee’s philosophy to software development.

“This is the idea that a software developer can only learn techniques in their proper context, through a holistic approach. Methodologies provide more than just techniques: They offer training methods, theories, and mental attitudes. Learning these factors allows a student to experience a system called its ‘totality’. Only through learning a system completely will a developer be able to, ‘absorb what is useful’, and discard the remainder. Real work situations allow the developer to learn what works, and what doesn't. The critical point of this principle is that the choice of what to keep is based on experimentation and experience over time. It is not based on how a technique may look or feel, or how precisely it mimics tradition. In the final analysis, if the technique is not beneficial in practice, it is discarded. Only the individual can come to understand what works; based on critical self-analysis, and by, ‘honestly expressing oneself, without lying to oneself.’”

-- Bruce Lee, Tao of Jeet Kune Do, 1975 (modified terminology italicised)

In this last quote, we have completely dropped the term Agile. The practitioner is encouraged to not only learn techniques and methodologies, but to retain what is useful in practice, according to their own context and experience, and to discard what is not. Lee describes a pathway to learning the best methodology for one's own situation, strengths, and context. "It's just a name used, a boat to get one across, and once across it is to be discarded and not to be carried on one's back". Lee encourages us to use a process or methodology to get to a point of learning where we can understand how we can adapt and evolve to a better process for us (and our teams). Maybe it’s time for a change after 20+ years of Agile use, abuse, and misuse. Let me propose the term Jeet Daima Do, the way of the intercepting code.

Introducing Jeet Daima Do

The Way of the Intercepting Code?

Jeet Daima Do (JDD) is a software development methodology inspired by the principles of Jeet Kune Do (The Way of the Intercepting Fist), emphasizing adaptability, efficiency, simplicity, and the fusion of various techniques to achieve optimal results. This methodology is designed to help development teams create robust, maintainable, and scalable software by focusing on core principles that prioritise effective problem-solving and continuous improvement.?

The Eight Principles of JDD?

1.??Simplicity and Directness?

·?????Write clear and straightforward code that directly addresses the problem at hand.?

·?????Avoid unnecessary complexity and over-engineering.?

·?????Use the simplest solution that works effectively.?

2.??Efficiency and Minimalism?

·?????Optimise code for performance without sacrificing readability.?

·?????Minimise resource usage by eliminating redundant or inefficient code.?

·?????Focus on writing code that is both fast and resource efficient.?

3.??Adaptability and Flexibility?

·?????Be prepared to change or refactor code as requirements evolve.?

·?????Embrace new technologies, tools, and techniques that enhance the development process.?

·?????Design code to be modular and maintainable, allowing easy updates and improvements.?

4.??Continuous Learning and Improvement?

·?????Encourage constant learning and skill development within the team.?

·?????Regularly review and improve existing codebases and development practices.?

·?????Foster a culture of feedback and collaboration to drive continuous improvement.?

5.??Integration and Interoperability?

·?????Combine the best practices from various development methodologies and frameworks.?

·?????Ensure seamless integration of different components and systems.?

·?????Prioritise interoperability and compatibility with existing tools and technologies.?

6.??User-Centric Design?

·?????Focus on creating software that meets the needs and expectations of end-users.?

·?????Involve users in the development process to gather feedback and refine features.?

·?????Design intuitive and user-friendly interfaces and experiences.?

7.??Agility and Responsiveness?

·?????Maintain a flexible development process that can quickly adapt to changing requirements.?

·?????Use iterative and incremental development cycles to deliver value continuously.?

·?????Prioritise tasks based on user feedback and business needs, ensuring timely delivery of critical features.?

8.??Pragmatic Problem Solving?

·?????Tackle problems with a practical and results-oriented approach.?

·?????Use proven techniques and solutions while remaining open to innovative ideas.?

·?????Focus on delivering working software that solves real-world problems effectively.?

Guidelines to adopting JDD

·???Iterative Development: Break down the development process into small, manageable iterations, each delivering a functional piece of software. Review and adapt after each iteration.??

·???Code Reviews and Refactoring: Regularly review code to identify areas for improvement. Refactor as needed to maintain code quality and performance.??

·???Automated Testing: Implement comprehensive automated testing to ensure code reliability and prevent regressions??

·???Continuous Integration and Deployment (CI/CD): Use CI/CD pipelines to automate the integration and deployment process, enabling frequent and reliable releases.??

·???Cross-Disciplinary Collaboration: Foster collaboration between developers, designers, testers, and other stakeholders to ensure a cohesive and well-rounded development process.?

Conclusion

By adhering to these principles, The Way of the Intercepting Code aims to create a dynamic, efficient, and user-focused development environment that produces high-quality software tailored to the evolving needs of its users and stakeholders. This approach echoes the Agile methodology but serves as a reset, incorporating the philosophy of Shu Ha Ri and the flexibility of Jeet Kune Do. Rather than being a specific process, it represents a mindset and a commitment to continuous improvement, breaking free from the constraints of rigid processes. As innovation leaders have emphasised, significant gains require different thinking, and exponential progress demands a departure from the conventional. By embracing this philosophy, one may aim to drive substantial advancements and achieve remarkable success.