Has Agile Lost Its Way?

Fifteen years ago, a small group of programmers and project managers met at a conference in Utah to discuss a radical. Until that time, most software projects fell into two very distinct camps: those that involved hundreds of programmers working on big, multimillion dollar software projects trying to create broad enterprise level initiatives, or small projects written by one or a handful of people.

The first made use of a methodology called the Software Development Life Cycle (SDLC), also frequently referred to as the waterfall methodology, generally took several years to accomplish, and generally ended up failing about seven times out of ten.

The latter approach, on the other hand, relied upon a small group of people, with short sprints of activity and a constant reassessment process. What was most notable about the latter was that this smaller approach worked surprisingly well, resulting in software being written in months rather than years, usually producing code of higher quality, with frequent updates to the core code base that made the code much more responsive to the customer's needs.

This group produced a short statement of principle called the Agile Manifesto, and a new methodology - Agile Programming - was born. Over the last fifteen years, being an "Agile Shop" has become a badge of honor for software companies and IT departments alike.  However, increasingly, there are questions about whether Agile has become so watered down as to be rendered meaningless, most recently in a post from one of the original manifesto signers. 

Agile Processes vs Agile Philosophy

My own experiences with Agile - as an architect and in working with scrum-masters - have been mixed. I have seen agile projects that have worked successfully, where people used the tool to develop success evolutions of a project used the approach to get experienced developers working in concert to  achieve specific objectives in relatively short order. The problem is that I have also seen Agile projects end up become a morass of user stories without any clear objectives, tasks accomplished that in the end did not significantly matter towards an overall goal. 

Projects went far beyond their projected time and budget because agile was used as a time management tool, rather than a programming philosophy. Often times, such "agile"  approaches ended up looking a lot like Waterfall, even as people brandished the relevant hockey stick and pointed proudly to how many points they managed to complete on their burn down.

Put another way - Agile has become a brand, not a philosophy of development.

One of the facets of software development that is often so hard for non-developers to understand is that at the beginning of creating a piece of software, it is likely that even the developer does not actually know what he or she will end up with. In this regard writing software is much like writing a book - until you have written a chapter, you normally do not know, except very loosely, what the next chapter will look like.

Indeed, most software projects early on are processes of clarification - putting together enough of a foundation to be able to prototype a solution, less for the sake of the programmer than for the client to be able to see whether that solution is in fact the direction they want to go. This can be tough from a development standpoint, because most clients don't necessarily understand, or want to understand, what's happening in the background - in the database or the mid-tier. They want to see the interface. They want to see it completely working, and they want to see demonstrable progress on a module by module basis, even if there is common code that facilitates most of those modules.

The problem with this is that there are few if any forms of manufacturing that work this way. When you build a building, you dig a hole, add support pylons, lay down multiple floors of foundation, lay in the wall struts, wiring and plumbing, put in outer-wall supports, put in doors and windows, and about that time start putting up the facades, lay the roof and thousands of other myriad tasks, and the completed "application" doesn't appear complete until almost all of the other work is done. 

Software is often even worse, however, in that most software requires the construction of something that didn't already exist. If it did, then most people would have purchased that software already. This means that there has to be some room for discovery and uncertainty, not just in terms of how hard it is to implement, but in terms of whether what is created meets the needs of the customer, something that often won't be apparent until the customer actually has some form of that software to play with. Far more projects have been killed by a client saying "that's not what we wanted", than by programmers not meeting deadlines.

An Agile Story Set is Not An Architecture

In theory, with a competent programmer and an engaged client, that situation should never come up, but it is the rare client who can afford to be so engaged with software design that they can spot such things early enough to avoid significant rewrite. As applications become more data centric, this becomes even more paramount, because the data models themselves - the contract that the programmer has with the data - needs to be clearly articulated early.

Since this seldom involves writing code in the traditional sense (and since too many projects start with hiring coders who otherwise sit idle while data models are established), this usually means that data models and designs are not fully fleshed out until long after code has been written, code that will end up having to be rewritten because the data models no longer match the operating assumptions of the code. This is a methodological problem, because too many people assume Agile means that the developers don't HAVE to design.

This is exacerbated by the adoption of user stories and tasks. A user story is essentially a requirement that a particular role has for the application. Typically, most scrum based projects assume that there is, for each user story, one or more tasks that need to be accomplished. There are two fundamental problems with this, however.

First, this creates the implicit assumption that each task in turn is a separate solution to a problem. Saving a file for one type of user is different from saving a file for a second. In practice, actually saving a resource is usually consolidated into a single step (especially for document-centric processing), with minor variations that are handled by exceptions to the main routine. If the person doing the coding is experienced, they will look at this breakdown and attempt to identify patterns where the same or similar actions take place, and will code once. If they are not experienced, though, they will see such a task list as separate requirements to be fulfilled, and will have separate code for each similar instance without looking for those all important patterns.

Put another way - for an experienced developer, a scrum chart is a graph - and if they are intelligent such developers (or architects) will usually incorporate a System User to make such relationships clear.  They will also understand that such a chart describes user requirements, but not necessarily system architecture ... the two are often radically different.

Inexperienced programmers, on the other hand - and when you start dealing with large software projects with dozens of programmers you will be dealing with inexperienced programmers - will take the agile tasks and do exactly what those tasks do, even if there are dozens of other tasks that are outlined to do precisely the same thing. If you have a good architect working with the scrum master, then this can be avoided, but if you don't, then you get large numbers of redundant tasks that introduce both complications and lack of consistency for the final APIs, as well as introducing multiple points of delay or failure.

Paired Programming is not OTJ Training

A similar problem with Agile comes in the notion of paired programming. In theory, with paired programming, you have two programmers of roughly equal skill working together to cross check one another's work, to discuss alternative programming approach benefits, and to spell one another as necessary so that you have some redundancy in case one developer gets knocked out of play.

In practice, when pair programming is practiced at all, it's treated as OTJ training, with one senior programmer now mentoring a junior programmer. Such shadowing does have benefits, but if the disparity between skill sets is too great (or if both programmers are too junior) then you are also sacrificing the speed and efficiency of your senior programmer with comparatively little benefit.

However, even this is better than not doing it at all, because in that case you get no knowledge transfer whatsoever. This is usually the case in business circles, where the operating assumption seems to be that a new programmer should be able to drop right into the code another developer has written, seamlessly. In the real world, most code is filled with starts and stops as ideas are explored and discarded. Programmers tend not to document their code in the moment unless there are tools that make that easier, and when there are tight deadlines even cleaning up code is often done only very peripherally, because of the very real fear that something that appears insignificant may become very significant indeed.

In some respects, this is worse for poorly executed Agile than it is for Waterfall. SDLC is process heavy (too much so at times) but the advantage there is that a given module has been heavily tested by the time it ends up in the final build, and there are multiple eyeballs to question such things as documentation.

"Agile" Means the Agility of Design

Agile is the process of continuous prototyping, which means that unless there is a stage set aside specifically for code cleanup and documentation (there typically isn't), then it is likely that at best you'll end up with documented APIs, rather than those + documented procedural code. Again, establishing documentation practices at the start and then testing for conformance as part of the process for acceptance will ameliorate a lot of this, but this is usually observed more in the breach than in the practice.

What this points to is not necessarily a limitation just of Agile, but the perception of Agile that holds sway does make the problem worse. This is especially true of the notion that Agile programming means that there is at best a minimal design phase. For smaller products with few participants, that design phase may not be immediately evident - the design is held in the head of the primary programmer/architect, so there's a solid starting point even it is not explicitly committed to paper (giving rise to the myth that there is no design involved). 

However, the larger the project, the more the number of participants, the more critical that this design phase be made explicit, because it is this design phase that provides both the starting point and the rationale for why a certain task is needed. Especially in the earliest sprints, the design itself is also still somewhat mutable, and it is in fact this change in design, rather than changes in implementation, that typically is what needs to be evaluated at the start of a sprint throughout the process. This isn't just a matter of trimming features, but also is the process that the client should use to make the design more accurately represent their vision of what the product should do.

One thing that is often forgotten with Agile methodology is that an Agile commitment extends beyond the "completion" of the project. The accepted version of software represents a major release, but that release should still only be considered a snapshot of the project, albeit one that meets the current business needs and is as debugged as possible. Once you move past the 1.0 stage, then planning for adaptation to meet new business needs should begin (after the appropriate down-time to keep your programmers sane).

Summary

The takeaway from this is not that Agile itself is a bad methodology, but that as a tool it needs to be seen for what it is. Design is still essential, perhaps even more essential, with Agile methodology than with Waterfall, and that the incremental nature of Agile should be seen as the clarification of this design over time as functionality and workflow become more evident. User stories should inform the design, but do not mistake these user stories as the basis for an architecture.

Agile products are often used as project management tools (and there is nothing wrong with this), but it requires that there is an intelligent architect and scrum master working in concert to ensure that its primary purpose - meeting the specific business needs of the client - are clearly met. Moreover, it also requires that the clients themselves remain engaged, as it is their vision ultimately that needs to be satisfied.

What are your experiences with Agile methodologies and tools?

 Kurt Cagle is Principal Evangelist for Data Science with Avalon Consulting, LLC. Proper Agile development, with an emphasis on effective design, is a key part of our core production values.