"When" to focus on documentation in Agile Software Development?

The only time "Documentation" was ever mentioned in Agile (aka in the Agile Manifesto) is in this statement -

Working Software over Comprehensive Documentation

Followed by a very important warning which many manage to neglect - "That is, while there is value in the item(s) on the right, we value the items on the left more." From that, unfortunately, a ludicrous idea of not documenting at all has been generated over time.

Still a Radical concept!

Being part of many debates over years on this topic, it's obvious how this area of the manifesto is less explored. Which is why Business Analysts and Product Owners are still wondering when to document in the Agile way or should they do it alone. Collaboration is key.

Sure - BDD helps us document user stories with collaboration and help create the Acceptance Criterion for planning discussions. Some have successfully established ATDD which helps in writing failing Acceptance tests and creates an exceptional red-green-refactor cycle similar to TDD.

BUT

Among all these, a developer still wonders WHEN do they document the intricate details of the technical architecture, necessary information about a dependency and system designs while supporting "Technical excellence" without which no one can really become agile. We can tell each other that TDD helps us in focusing continuous attention to architectural detail, but how many developers actually use it and writes a test before each line of code.

There are many cases where the product documentations are nowhere to be found as the developers were told during planning - "We are now doing it the Agile way and all you will get are a bunch of statements written in Gherkin syntax."

Agile Principles clarifying the WHEN

#2 Welcome changing requirements, even late in development.

We are seeing changing specifications in a weekly/monthly basis. Based on the type of business it's highly likely to see changing expectations on a daily basis as well.

Can we really document anything before knowing what to build? Give it a deep thought.

When we talk about User Stories - most popular technique for specification gathering - we are not documenting a requirement. We are starting of with WHY, the least amount of known high level outcome, with or without knowing how the output might look like.

If a team never documents anything else other than the stories, that's when it starts getting impractical with a huge risk to quality, understanding and sustainability. Documentation is required.

#7 Working software is the primary measure of progress.

Traditionally we have learned to document first as we used to plan heavily to begin with. We know how that's against agility according to Cone of Uncertainty. We can never know about a feature/specification well enough when we start of. We learn through early feedback on iterations that can generate enough discussion.

Reason why we should build a piece of software first based on high level WHYs and review how it may look like. Once confident and final on a characteristic or functionality, then we should document.

Only document to record evidence of a built feature which is accepted.

#9 Continuous attention to technical excellence and good design enhances agility.

With even changing specifications on continuous feedback, we need even changing updates on the documentation in real time. Documentations essentially should reflect current state of affairs rather than what we hope it could be.

When working in huge organisations where a single monolithic code base is being used and updated daily/weekly by many many developers, that's how chaos starts. Documentation should be in codes - true. It is also true that not every developer are capable of doing it as we hope they do. If they can't in the code, make sure they do anywhere else that can be shared and available for everyone.

Duplicate documentation is better than no documentation, as long as they provide the same recorded evidence.

Conclusion

Therefore, the original manifesto statements about ".. over comprehensive documentation" actually translates into the following:

First build a working piece of software and then focus on documenting how it was built and what it does, for shared understanding.