Why Do Organizations Struggle with Agile Software Development

Agile software development model is adapted in many organizations because it allows them to get started with software development without having complete set of requirements. Agile model also provides ability to change or improvise software requirements and even core product concepts as end user sees a product being built in each increment/iteration.

In other words, organizations wanted to escape fairly robust, but inflexible waterfall method, while getting on with building a software product in iterative manner. Waterfall method was risky because client did not see the product till end, at which time it was very costly to make changes.

Agile gives you flexibility to change requirements and incorporate new requirements but that does not mean that you can succeed without a product plan, product strategy, and of course it does not assume you have got infinite budget and time.

Product managers/owners are often unable to stick to product plan, deviate from product vision because they are excited by new ideas coming up as they build the product. This would mean abrupt and unexpected changes in software architecture because of incomplete requirements leading to technical debt and rework.

It is easy to dismiss or ignore technical debt. Like classical debt, technical debt must be repaid often at a great cost.

It should be kept in mind that every change request, new requirement has an impact on budget, timeline and quality of work which is accumulated in technical debt. In Agile projects, project managers will need to be very smart and proactive with managing overall budget and client communication.

Given infinite resources, anyone could achieve the desired goals; art of management is to achieve desired goals with limited resources

Rework and resultant delay in product development is something that product managers blame on development team without realizing or admitting that they never had complete requirements at any stage and in any form.

The flexibility to change requirements in every iteration is a double edge sword because it means more rework. Rework is the biggest enemy of productivity and compromises success of fixed budget product planning.

It is often rework combined with poor communication and resultant lack of expectation management that hurts the most in Agile project

It is no surprise that Agile teams often fail by working extremely hard and burning out. Constant firefighting is a telltale sign of poor project management, and resulting project failure. This would also mean that resources will start to quit, and retention of team will become a big organizational challenge.

This agility is also a recipe for failure in many cases. Here are some typical scenarios to illustrate how agility leads to failure of software projects :-

1. Fixed Budget Projects

In many outsourcing organizations, there are projects with small fixed budget in range from $5000 to $20000. In such cases, if requirements keep changing without realizing time and budget constraints; then this can easily lead to budget running out and project failure. Fixed budget projects can not succeed without precise requirements, and avoiding rework as much as possible.

Even if development team completes the so called requirements, client is almost never satisfied because they were never very clear about requirements and vision of the product in the first place. So, the first complete version of the product might be at least 10 steps or 10 complete iterations away from where they are now after completing iteration 1.

Here's a hypothetical project, where budget was 100 man-hours for ten requirements R1,R2, … R 10. Here each of requirements(R's), change requests(CR's) and new requirements(NR's) are all budgeted at 10 man-hours. Since team bandwidth was 30 man-hours per iteration, therefore project was expected to be completed in 4 iterations or at cost of 120 man-hours. We are assuming each iteration is 2 weeks.

What actually happened instead that in each iteration, there were change requests and new requirements which were prioritized over initial requirements. So essentially product owners are building something quite different to what they had originally planned. This product might still be considered a success, because a worthwhile product was delivered in the end. On the contrary, it might be deemed a failure because original requirements were never fulfilled, and something quite different was built.

It is very much clear that product owners have to take ownership at every step, till very end rather than shifting the blame on development team which is what happens most of the time.

If your software development methodology is Agile; this does not imply that your budget is also Agile

To make project successful, stakeholders need to sit together at the end of iteration 4 and either conclude that work done so far is enough to make project successful. Alternatively, they may extend the budget by another 100 man hours, while also deciding about freezing change requests, and new requirements.

No project can ever succeed if acceptance criteria or goal posts keep changing

Exhausting budget without fulfilling original/initial requirements often causes a lot of panic and stress leading to firefighting and sometimes project failures

As seen above, Agile project has already consumed 120 man-hours while completing only 70% of initial requirements, and this does not include all the other time that is usually spent on bug fixing, non functional requirements, and user experience enhancements. This is often due to wrong assumptions by sales team who are pricing as per functional requirements only.

Agile Projects can not be agile without Agile Contracts that have adequate provision for Change Requests, New Requirements, Bug Fixing, Non-Functional Requirements and User Experience Enhancements

2. Opportunity Cost

Agile allows you to change requirements and refine product in each iteration, but this should not be abused. In one extreme case, I have seen a product being iteratively refined for 4 years !

This ultimately leads to product concept becoming outdated, and no longer as much attractive as it was 4 years ago; not to mention massive cost overrun. Frequent requirement changes also act negatively for code quality, leading to unexpected bugs/errors. So requirements, change requests and refinements have to be frozen at some point before final release of the product. Best approach is to take up refinement, new requirements with client in mind; best requirements in cases come from real clients and not product owners.

3. Non-Functional Requirements

Product owners are almost always unaware of critical non-functional requirements including security and performance. It is also not possible to judge user experience upfront, if complete requirements and designs are not available. This can lead to serious disappointment for the client, if client is under impression that very first complete version of the product will have all the non-functional requirements as well.

Avoiding Failures for Agile Projects

To avoid failure in Agile projects, you must identify and mitigate major risks associated with Agile projects. Key risk associated with agile projects are :-

1). Unrealistic product timelines

2). Budget constraints

3). Significant New Requirements

4). Lack of prioritization from stakeholders

5). Stakeholders have not identified minimum viable product(MVP)

6). Frequent change requests

7). Significant non-functional requirements which are not being met due to client pressures

8). Changes in requirements leading to significant changes in design and requirements for product to rebuilt from scratch

9). Lack of feedback at the end of each iteration

10). Key team members leaving the team

11). Prolonged firefighting mode

Above are major risk factors associated with agile projects. It is a good idea to recognize and share risk factors with client and have an open discussion to mitigate these. These should also be highlighted with the client, whenever there is a need in order to mitigate these as quickly as possible.

To avoid failure in agile projects, you need to keep the following major points in mind :-

1). You should have a high level product plan, iterate and finalize the design before starting the development. In other words, agile software development cannot save incompetent or inefficient product manager from self-destruction arising from poor product planning. 

2). You should be prepared to iterate the whole product development at least couple of times to get the desired results. This means, you should have cushion in the budget to do that. In many cases, you cannot realistically determine hard release dates with agile development model. If software development is agile, then everything else tied with product development including release plans should be agile.

3). You must know when to stop and change course if necessary. You must not sink the ship, if you are not sure whether you will get desired results in reasonable time and budget. This means, you must know when to stop if project can not be turned around despite spending adequate time and resources.

4). Keep the non-functional requirements in mind while specifying functional requirements and planning timelines and budget.

5). Agile software development can not succeed for strictly fixed budget projects. Be flexible with budget, if you want to run the project on agile software development model.

Basic rule of thumb for planning budget in agile project is :-

Budget Spent (Upto nth iteration) + = Budget(Upto n-1 ith iteration) + Cost(Original Requirements for nth iteration) + Cost(Change Request for nth iteration) + Cost(New Requirements for nth iteration)

This can be easily put in excel, and illustrate impact of flexibility on budget, and overall progress in terms of completing the project.

4). Product owners often struggle in dark, where they are unsure of what will succeed in the market.

Product owners may be struggling with defining what exactly they want(e.g. product features etc), this may lead to a discovery phase leading to feature frenzy and rejection(or change requests) for most of these features at a later stage

Developing a large list of features(or their variations) to select most attractive features is an approach that is a luxury only large product organizations like Apple and Microsoft can afford. Most of the other organizations need to be very disciplined in product management. Selecting minimum viable product, prioritizing features with input from end users and client feedback is very important.

To overcome this challenge, they can release the product to limited audience and try to adapt a model where requirement ownership is transferred to end users and only market driven requirements are prioritized for development.

To minimize risks, it is important to train product owners(or client) in Agile Software development, so that they are able to plan budget, manage expectations and risks proactively

Conclusion

Agile software development methodology is a product development strategy, which helps many projects succeed because it allows critical interventions at right time allowing changes in requirements and design leading to success of product.

There are several major risks associated with such an approach as pointed out above including complications in product management, running out of budget, design issues and accumulating large technical debt.

Flexibility is great, but there's no free lunch. If you are willing to change requirements and design in the mid way than project timelines, budget and release dates also get changed accordingly.

Managing user expectations is very important, it requires proactive communication at all stages of the project and should not be limited to periodic progress reports. In most cases, it is obvious that first complete version of the product will almost never be ready for commercial release and would require several more iterations to before product is ready with all the non-functional requirements and appropriate user experience.