What Agile Failed to Deliver—10 Challenges to Tackle

Inflated expectations, misinterpretations, confusion, and simplistic solutions made Agile largely fall short of its intentions and potential. Here is what software organizations may still have to address.

by Ernani Ferrari

Agile approaches for software development have, in most cases, fallen short in delivering productivity, which in turn is an impediment to Agile’s key intrinsic promise: agility. An organization can be flexible to adapt but will be slow to deliver if it isn’t productive.

Promoting more flexibility to adapt, incremental development, team engagement, open communication, user involvement, and the aim at delivering value to the customer does increase the effectiveness of software development and may shorten cycle times for the release of the first working software components, but it is not enough for the long haul. Not being efficient and productive will inevitably delay what comes next. Software companies, IT departments, the gaming industry, and other software developers need consistent deliveries, month after month and year after year, so being flexible and agile only off the gate is not enough. Such organizations can’t just focus on delivering more in the short term—they need to also deliver more in the long term, addressing development and sustaining requirements. They must consider the needs of brown development while executing green development, and address requirements from cross-functional processes (such as customer/user support and users training) from the beginning to work efficiently.

Continuous agility and flexibility without productivity is an illusion. Yet, organizations widely adopted Agile practices as a business fad, not thinking about how such practices should be adapted or adopted to meet their own business needs and realities. Many assume Agile is a one-size-fits-it-all solution and neglect to pay attention to the nature of their projects, activities, and cross-functional processes.

Many organizations have benefited from Agile. Smaller teams that didn’t have an established operational framework quickly found one. However, inflated expectations, misinterpretations, confusion, and simplistic approaches made Agile fall short of its intentions and potential in numerous organizations. Too frequently, I meet managers and executives unhappy with the results they are getting.

I have worked in product management and software engineering for thirty years, with over one hundred small and large software companies and numerous IT departments—from “one-man bands” to hundreds-of-people teams. During this time, I have studied all kinds of methodologies, techniques, roles, documents, notations, concepts, and metrics I could put my hands on. I then analyzed and tested how they could be combined and applied to effective and efficient company-wide processes. I know there are thousands of books and articles, countless hours of discussions, and many different points of view on the topics of my list below. Thus, my intent with this article is only to call out what is important to most software organizations to promote timeliness, productivity, and quality along with flexibility, and Agile by itself has often not delivered for the short and long term (I repeat: because it was misinterpreted, not designed or adapted to the needs or business context at hand, or simply because expectations were inflated). I hope it will help readers reflect on issues their organizations are facing, identify their causes, and gain insights on how to become even more flexible and effective while reducing costs by reviewing their practices and becoming more efficient. ?

The 10 Challenges to Tackle

1 – Productivity – The following nine items impact software organizations’ productivity. The main direct offenders to squads, however, are avoidable rework, interruptions (a five-minute interruption of concentration tasks usually consumes 40 to 45 minutes in deceleration and reacceleration), unnecessary meetings, personal preferences of team members, and repeated questions from people outside the development teams.

2 – Quality – Frequently, companies kept their existing poor quality assurance resources (such as the lack of trained people and limited testing techniques and tools) to the newly adopted Agile processes. In contrast, systematic quality mechanisms such as quality planning and structured inspections have often been dropped and forgotten. Moreover, development organizations must also address aspects of product quality such as performance, stability, maintainability, supportability, and testability to meet users’ needs—and many of these aspects may require more planning and initial analysis to avoid extensive rework efforts.

3 – Knowledge Management – People need information about the software they will work with. They must also learn about the involved processes, policies, and tools. While trying to avoid unnecessary documentation, organizations too often cut down on documents that are key for effective and efficient knowledge transfer and communication inside and outside the development teams. Relying primarily on squad members to transfer knowledge to newcomers is often inefficient—even more so in growing organizations or dealing with software with long lifecycles for which sustaining costs are far greater than their initial development costs. ??

4 – High Performance from Specialists – While multi-functional roles simplify capacity planning and staffing, they usually don’t deliver the productivity or quality subject-matter experts do. Many developers don’t master the skills they need for their core tasks, yet organizations ask them to regularly perform tasks of other roles, such as testing, configuration management, or technical writing, which require skills they usually don’t have. Bridging other specialists into a project, such as a data or an infrastructure architect, has also become difficult or unnatural in many organizations. (Read more about using specialists in my article Specialization – Against Mediocrity (1))

5 – Key Benefits of Structured Project Management – It has been said that no two software projects are equal. Breaking projects into small pieces that look alike is not enough to have them executed the same way, because their parts will come together differently. If inherently different projects are handled as if they were the same, chances are that necessary tasks aren’t addressed, unnecessary tasks are performed, and yet others aren’t adjusted to the project’s reality as they should. Yet, even though Agile teams have great flexibility related to processes (in fact, most of them claim not to have a method to follow and have 100% freedom to do things any way they want), many of them tend to establish routines that disregard the big picture, particular opportunities, and challenges of the projects at hand. Unthinkingly avoiding any type of upfront design, overlooking quality planning, and skipping structured risk analyses are examples of practices that can represent small savings and great missed opportunities while frequently discarded for no solid reason. Moreover, repetitive tasks that can be more effectively managed as production lines are often confused with projects and vice versa. (Read more about Project Management in my article Software Project Management: Beyond Methodologies (2))

6 – Efficiency in Software Development versus Software Sustaining – Engineering teams often tackle new user stories and bugs using the same approaches and routines. While that may work well for homegrown systems or customizations of market solutions, providers of packaged software or SaaS solutions often miss the benefits of specializing processes for developing new features versus maintaining existing software, as well as having distinct deployment/release timing. For software products, such releases’ independence may also be necessary for their customers to simplify updates and upgrades.

7 – Clarity of Roles – Organizations often are led to understand some roles in Agile as one-size-fits-all. Companies frequently don’t adapt roles and responsibilities to their reality, leaving to-be-self-organizing teams with unclear roles for squads and supporting groups. For example, many organizations still debate the Product Owner role as if it were universally applicable to any organization, even though they can be as different as a software product company, an IS department, or a software development services provider.

8 – Good KPIs – Agile practitioners have adopted statistical methods and produced insightful metrics, such as cycle time, lead time, and throughput, which help teams understand their performance, plan, identify bottlenecks, and make data-driven decisions. However, managers still miss productivity and quality metrics that support benchmarks against other organizations and don’t give room for misleading manipulation by the involved people (an essential characteristic of any KPI).

9 - Good process management practices – Jim Highsmith3 wrote, “Large agile projects may ‘look’ like traditional projects, but they should ‘feel’ like agile projects. ‘Look like’ means there may be additional structure – organization, architecture, documentation, process.” More often than not, these elements (organization, documentation, and process) are not adequately addressed by Agile adopters. In fact, except for retrospective meetings, they seem entirely lost under confusion about self-organization. While experts in business process management declare there is no process improvement without process documentation, most Agile teams work with little to no process documentation (from written workflows to simple document templates), limited metrics, no process sponsors, no compliance checks, and little to no records of lessons learned—even dealing with complex and many times mission-critical software, projects and environment. ?

10 - Good planning framework – Today, the Agile toolbox has consistent resources for capacity and development planning and completion forecast. Yet, numerous organizations use little or none of them and, in many cases, estimate time and effort nearly blindly (in great part because of the misunderstood topics of requirements documentation and upfront design). Meanwhile, top-performing software organizations operate with systematic estimates, and their variations average less than 10% of actuals. Not rarely do we see executives begging for deadlines that are important for their business planning while hearing “it will be done when it is done” from development teams. (Yes, this is a controversial topic! I will soon write an article about it—follow this channel and stay tuned for thoughts and tips that work!).

?

So, what should you do if your implementation of Agile has failed to deliver on one or more of the items above? Don’t throw the baby out with the bathwater, and stop looking for cookie-cutter solutions. Understand that your organization may look like others, but it is unique regarding systems, products, software cycles, people, customers, structure, and goals. Do what is needed by your organization—what clearly will benefit your company. If you think something your organization has in place doesn’t seem to help improve flexibility, productivity, quality, and planning… it’s probably because it doesn’t! ?

?

Follow this channel to get alerts about a new series of articles with thoughts, techniques, and tips about the topics addressed in this article.

Read more articles by Ernani Ferrari at https://ernaniferrari.com/articles/

?

(1)??? Read more about using specialists in my article Specialization – Against Mediocrity

(2)??? Read more about Project Management in my article Software Project Management: Beyond Methodologies

(3)??? Jim Highsmith - Founding member of the Agile Alliance, co-author of the Agile Manifesto, and director of the Agile Project Management Advisory Service for the Cutter Consortium.

?

Ernani Ferrari (www.ernaniferrari.com ) is a former executive of large enterprise software companies and a consultant specializing in business performance in the software industry. He helps small and medium software companies accelerate their growth by orchestrating Product Management and cross-functional processes of the software cycle and improving strategic management. His clients include corporations such as Microsoft,?IBM,?and Wolters Kluwer. He is a former instructor at Stanford University Continuing Studies, a frequent international speaker (PMI, AMCHAM, SQE, IBM, ABES, Prosoft, BoS, government agencies, etc.), and the author of several articles and other publications for the software industry, in different languages. Ernani has helped over 100 companies around the globe improve their software processes and business performance.

?

Have a question? Please feel free to ask your questions and make comments, or contact me through my website – www.ernaniferrari.com . I will be glad to reply.