The Agility Angst

First of all: let me start by showing my age. I was born in 1973, and yes, that does mean that there's a big, and I mean BIG, party coming later this year. Big Five Oh looming around the corner - and to be honest I am very, very fine with that. I am happy with my age - except when I am being dropped like a baby at the back of the cycling bunch. Then not so much - dammit.

What does my age have to do with anything? Well - it basically means that I grew up with a very, very different type of software development practice. OO was still young. The mythical man-month was still a thing. And methodologies all depended on strict, rigid waterfall development models.

Nothing wrong with the odd waterfall, of course, but with the emergence of more and more, and better and better, software tools, libraries and frameworks - this whole idea of the time consuming waterfall process has kind of become irrelevant. Who would still go through this entire process and ask their users to wait for months before they would be able to give any feedback? What users would accept that type of arrogance on behalf of their IT department? Seriously?

Enter Agile!

I am not going to try and explain this in too much detail, but suffice to say that the waterfall is now replaced by a circular, iterative process that looks like this:

And that circular process is then repeated, time and again, during the entire software development project, so that users can constantly course-correct the development team, and everyone is bought into the entire process of developing a new software product that actually delivers business results. Every iteration, is called a sprint - and every iteration is a little bit like a mini-waterfall that delivers some visible and verifiable requirements, every time.

Good. Now that we have established that - let's talk about the topic of this article: The Angst for Agile.

What Angst are we talking about?

Alliteration aside, I am of course talking about FEAR. In our industry, I have still seen a surprisingly large number of cases where people are AFFRAID of a newish, largely unknown and therefore unloved, development methodology called agile. And I think that you can look at it this Angst for Agile from different Angles. No kidding.

From a business user's perspective, who theoretically at least is the customer that we are doing all this for, there should really not have a lot to be afraid about. Except for the fact that, and this is not a minor point, Agile requires the business user, or its representatives, to be VERY INVOLVED in the software development project. You cannot do without the frequent, and repetitive, business user review conversations that happen before, and even during, every iteration. That's not trivial. It requires the user to spend the time and effort that it takes to exactly think through and communicate what it is that they want. I can think of a zillion reasons why those end users would not enjoy that, but hey, this is the best answer that the software industry has come up with while we wait for telepathic interfaces to suck the info out of your brain with no conscious effort.

From a developer's perspective, there are also some interesting reasons while Agile could be a teeny weeny bit scary. All of a sudden, you are on the hook to DELIVER your work, every few weeks. Every sprint, ALWAYS, leads to a demonstratable deliverable, which can be shown to and reviewed by the end user, to sollicit their unsalted feedback. You better be there! No rest for the agile developer, that's for sure. So if you wanted to have a comfortable job doing some easy relaxed coding once in a while - agile is not your friend! On the contrary.

From an operational perspective, Agile also turns the world upside down. All of a sudden we are releasing software MUCH more frequently than before. We are not talking 2 releases a year - no no: we are talking a release after EVERY sprint. Between iterations, all kinds of parameters of the deployment could change, dramatically, and it can do so almost automatically as the deployment will be highly automated. It has to be automated in an agile world - we can't afford the overhead of manual deployments after every sprint! This is exactly why flexible deployment platforms in the cloud, and CI/CD (Continuous Integration / Continuous Deployment) are 100% part and parcel of the move to agile, and where some of the real benefits of the Agile approach are to be found: in that automation of the integration and deployment procedures.

And last but not least, I think I should mention that there is a real, fundamentally human fear involved here as well, which is related to the governance of this entire process. It deserves a separate heading.

Governance in Agile

During the entire, highly dynamic, fast-paced, goal- and progress oriented process of an Agile development process, things CAN become a bit chaotic, for sure. I am not going to talk about the role of solid leadership in these setups, but I would like to make sure that everyone understands that this is a real and important hurdle to overcome. If you are a large organisation, subject to strict government rules and regulations, subject to GDPR, various privacy rules, financial rules, manufacturing and quality assurance stipulations - then you need to make sure that what goes on in the Agile process adheres to all of these playbooks, and that you properly govern the process.

In my experience, it is actually this governance perspective, and the associated fear of an ungoverned development methodology, that is preventing many large, regulated institutions from really adopting Agile. They just can't afford for things to go wrong, and they have all the checks and balances setup in the Waterfall model - but not in the iterative agile model. So they stick with the trusty waterfall - even though they know it has massive issues.

Metadata as code as the antidote to Agile Angst

I took a long and winding road to get here, but here's what I have come to believe: I believe that there are very good ways to deal with this angst. There are tools out there, like, unsurprisingly, Hackolade 's tools, that allow you to make massive strides on the journey to removing the fear for the chaos that Agile might bring.

Essentially, there are two components to the solution:

1. We need to have explicit, formalised, human- and machine-readable models, schema's and detailed documentation for the data that we are going to be processing in our Agile systems. It cannot just cover a part of the system - it needs to cover all parts. The relational database systems. The document databases. The column-family stores. The key-value stores. The graph databases. All of them. All parts of the polyglot persistence architecture need to be covered by this documentation.
2. We need to be able to update, analyse, version control, deploy that documentation as part of our iterative agile integration and deployment processes. This, and only this, strategy will ensure that the models, the schemas and the documentation remains up to date, at all times. There is no other way to do this that actually works: we need to treat our metadata (models, schemas and documentation) as part and parcel of the code that we deploy. Always.

If you want to learn more about this, I would really recommend that you take a look at Hackolade's Metadata-as-Code webpage . It's really full of great information, and has some very cool pointers illustrating how to achieve all of the above using some really clever integrations with source control systems like git/gitlan/bitbucket, and lots of fancy fun with CLIs and the likes.

I hope all of this makes sense to you, and that it also explains why I am so excited to be working with the Hackolade team to develop their presence in the market. It's a great story - and it's here to be told.

Hope this was useful. Please let me know if you have any comments or questions.

All the best,

Rik