If Agile Is Not Working For You, You Might Be Doing It Wrong

Agile software development is becoming the mantra of organizations everywhere who are dissatisfied with the results of classical development approaches; namely poor quality software that is delivered late and over budget.

While attempting (or proclaiming) to make the transition to agile, a lot of these organizations are not really embracing the true spirit of the agile methodology. Instead, they are creating additional tension across the organization without realizing the benefits for which they were striving.

As a result, "agile" is becoming a dirty word in some circles; another methodology boondoggle being pushed by expensive consultants, overseen by doubtful executives, and implemented by skeptical development teams. This is truly unfortunate, because agile does have the potential to deliver great results for companies while improving the daily lives of their employees. 

The Ultimate Business Driver for Agile

Theres' a lot of technical stuff involved in adopting the agile methodology — scrum, sprints, DevOps, automation, kanban, and other cool things — but the primary aspect of agile that must be embraced is the notion of building things in an incremental manner where, for each increment, time and resources are fixed and content is variable. That last part is really important; the agile team prioritizes work but doesn't guarantee the final product on a certain date. As they go from sprint to sprint the team learns their velocity, and as they get closer to completion the release date estimate improves in accuracy. The visibility afforded by the agile process allows the development organization to see and deal with reality, versus believing the 'big lie' of the requirements specification and detailed project plan at the root of the waterfall process.

This notion must not be understood by the development team alone, it must be embraced throughout the organization and all the way up the leadership chain. From a business perspective, the agile approach is primarily about managing risks and expectations during the software development process. Each sprint provides a reasonable package of risk, because the code either works or it doesn't, and a chance for the stakeholders to provide feedback if they don't like what they see before months of effort are committed to a bad design.

This is a major deviation from the classical software development model, where the technical team spends massive amounts of energy documenting requirements to the Nth degree while at the same time leadership asks for an iron-clad commitment to deliver that functionality on a specific date — often more than a year away — with a specific set of resources. In the perfect embodiment of this model, time, resources, and content are all fixed at the start of a project. Yet we've learned why the classical model is a complete fallacy:

• Requirements are never completely understood at the outset of a project and in fact usually change continuously throughout the development cycle because of new things learned, changes in the business environment, and changes in customer expectations.
• Estimates are never accurate because of the evolving requirements and also because much of what is being done has never been done before (new functions, new technology, new interfaces, etc.), leading to poor assumptions, technical mistakes, and the re-work that results from these.
• Resource assignments are never valid because unplanned work pops up constantly during the project, and that work usually pulls key people out of the mix almost always at the worst possible times.

The result, as we all know so well, is a project that starts well but soon ends up in the ditch — teams working overtime and cutting corners with quality, managers upset because the project is late and over budget, and customers unhappy with what ultimately gets delivered.

The agile methodology, on the other hand, accepts that we can't know everything at the start of a project and minimizes risk by breaking down code delivery into concrete chunks of output delivered by small teams on a regular cycle with clear rules about completeness, quality, and the automation of testing, integration, and deployment processes. It helps to manage expectations by providing visibility into those incremental deliveries and the remaining backlog of desired features so that product owners and customers can develop a realistic view on when they can expect delivery of the final integrated release and make informed time/value decisions about leaving features in or out.

The Schism

When organizations start to implement agile, they usually focus on the technical teams doing development work. Defining scrum teams and implementing Jira, Confluence, and other tools becomes a focus. They do a couple of pilot projects usually with just one scrum team and a small, simple set of requirements that are easy to encode in user stories. They hire a consultant to train the technical team in kanban and scrum and appoint a technical leader to drive the pilots and report on the results, which are almost always good.

However, in many cases the company's leadership team is not adequately prepared for or supportive of this transition. The senior executives in sales, marketing, product management, finance, and other critical operational areas may not feel the need to participate in the transition process, and don't fully understand what it means to the overall mode of operation for the company. This can lead to a schism between the technical and non-technical teams.

That schism usually manifests when the product management team or the sales team insist on a classical delivery commitment instead of working within the parameters of the agile approach. Despite the stated desire to get more involved with the process of software development, these teams often cling to the model where they ask for something vague and expect the development team to commit to a final delivery date while they go off and work on other things.

When this happens, the technical team is left in a quandary. They want to adopt the agile approach but end up compromising on key aspects of that approach. They commit to sketchy requirements and a date, because they are forced to. They start to do sprints but eventually fall back into classical mode when the original arbitrary deadline approaches and they are far short of the expected delivery content. Sprints start to stretch in duration and overtime work begins to increase. Testing, packaging, and operational automation are de-prioritized.

The operations team sees this and starts to get nervous. To compensate they throw up walls, requiring operational architecture reviews and complex change controls to ensure they don't get caught holding the bag for software that is unstable or performs poorly after the deployment. They demand that developers sign up for 24x7 support after the release, which the developers balk at because they have been working around the clock for weeks to meet the deadline. All semblance of the agile approach is lost, and the result is no better than before.

As a direct result, the leadership team may harbor severe doubts about the time and money spent on "agile transformation", wondering if it was just another technology-driven boondoggle that delivered nothing of value to the business.

The Solution

To avoid this fate, the adoption of agile must not only include but must be driven by the leadership of the organization. They have to understand the business they are in and they have to accept the trade-offs they are making by adopting this approach. The sales, marketing, and product management teams especially need to understand and accept the trade-offs inherent in the incremental development model with sprints of fixed capacity and fixed duration, and the requirement placed upon them to participate in the process by helping to decide when an appropriate level of content has been delivered to declare a release and put it in production. They have to understand the concepts of backlogs and burn down charts, and why you can't skimp on testing or packaging to meet a date.

The most important role in the agile lifecycle is that of the Product Owner. They are solely responsible for making the final call on what will be released to the market/customer. They are the bridge between the development team(s) and the finance, sales, and marketing teams. Without a product owner who really grasps the core concepts of the Agile Lifecycle, there is no hope for success. 

More than that, the product owner must be truly empowered to resist the push of other organizations that are not fully bought in to the agile transformation or just fail to understand its nuances. If not, they will fall into the trap of accepting arbitrary deadlines and ill-defined content, and ultimately fail to produce the expected results for the company or the customers while at the same time crushing the spirit of the development team.

The visibility provided by the agile approach pushes responsibility for software delivery up the organizational hierarchy. Leaders are forced to make and live with decisions while prioritizing items in the backlog and dealing with reprioritizations due to unplanned work. As the project progresses, they need to continually assess each team's velocity against the backlog and project a probable delivery date that they can share with customers. As the project nears completion, they need to make the ultimate decision to release the product with a certain amount of content, leaving potentially important features in the backlog for a later release. This is an inconvenient truth for those leaders who were happy to delegate these decisions to the poor developers and then complain later when those developers failed to meet leadership's unrealistic expectations.

There's no magic in the agile approach. Fundamentally, it's just the application of industrial production concepts to the process of software development. When done well it provides better visibility to help manage expectations and mitigate risks, but when people try to wrap a waterfall process in a cloak of agile terminology it can lead to disaster. Not only does it fail to deliver the expected results for the project at hand, it soils the reputation of the agile methodology across the organization, potentially hindering future projects. So if you hear of an agile project that did not deliver the expected results, you might want to dig a little deeper to find out whether the agile approach was actually adhered to or if the team just used agile terminology on top of a waterfall process.