Will Agile Undermine Data Governance?

“How can we deliver systems with more agility and clearer demonstrable benefit than ever before, and yet retain effective Data Governance?”

As we are all aware, many organisations have questioned why they are unable to deliver systems with the agility and rapidity that they desire.

Even when systems have been delivered, they sometimes didn’t fit what the organisation originally wanted, and often not what they ideally need at the point of delivery.

There have also been the bigger failures; these were the massive programmes or projects that crashed, or simply fizzled out. In many cases, these delivered nothing, apart from attention grabbing headlines, despite millions being spent on them. The majority of these epic failures, used SDLC methodologies and because the case for the prosecution repeatedly cited SDLC as the cause, collectively there has been a growing distrust of the methodology itself.

But in reality, the real causes are complex. They often reflect organisation cultures, or are driven by internal divisions, or sometimes, even by strong willed individuals. If we look more closely inside the organisations, we can clearly see the dysfunctional fractures that have paralysed their ability to deliver systems effectively. For many, these fault lines run along the divide between the Business and IT. With this backdrop, another significant destablising factor for some organisations, has been caused by its Enterprise Architecture function retreating into increasingly rarefied orbits.

This internal dysfunction is taking place within the context of an accelerating pace of technological diversity and Business environment fragmentation. And as if all these factors weren’t enough, organisations are also experiencing mounting peer group pressure to ‘keep up’ and ‘get ahead’.

Quite clearly, organisations have needed to find a way to overcome all of these impediments, and recently a fresh approach has rapidly gained ground to do just that. It promises a responsiveness of system delivery that many had assumed was just not possible. In reality, this is not a single approach, but many broadly similar approaches that use ‘Agile’ as a collective banner.

The clarion call of the Agile evangelists to solve all of these complex and often opposing forces, has reached the ears of senior management. These decision-makers, are being wooed and seduced, by the promise that Agile will extract the maximum Business benefit from system delivery, and streamline development.

It will finally give them what they need – rapid and focussed delivery.

Although a recent breath of fresh air for a lot of organisations, Agile has actually been sneaking up on us for quite some time. It has emerged from the early incarnations of Rapid Application Development, maturing step by step, and more recently, assimilating thought frameworks like Lean. In the last few years it has crossed the final frontier, smashing down the barrier between development and deployment that for far too long had seemed un-breachable.

This addition is known as DevOps.

The simple powerful message of this collaboration is:

Agile dovetailed with DevOps, will deliver discrete system increments, more frequently and with demonstrably positive Business outcomes.

As a bonus, both of the Agile and DevOps methodologies, have an emphasis on communication and learning that should help to span any organisational divides, and reverse the forces of fragmentation. These and other factors, would seem to provide a road to salvation; healing the rifts caused by years of disappointment and unresponsiveness.

However, not all stakeholders are quite as enthusiastic about the promises.

The idea of insulating the development teams so that they can just get on with it, and deliver features to ‘delight their customers’, runs counter to the instincts of many Architects.

This is not because they oppose the ambitions of Agile+DevOps. It is simply a fear in their minds that this mantra can become a ‘developer charter’, leading to each delivered component being a slightly different re-invention of the wheel.

They warn that a lack of governed and co-ordinated effort will cause fragmentation of the Enterprise vision, and quite possibly, the breakdown of system interoperability. This view would seem to suggest that Agile+DevOps and Architecture are intrinsically at loggerheads.

In other words, there will always be a trade-off; more control reducing agility or, more agility lessening control.

Figure 1 – Agile+DevOps versus Data Governance

But we cannot afford to lose control. Most organisations recognise that data is their operational lifeblood. This must be fit for purpose and therefore must be controlled at all costs. There is also the inescapable requirement for many organisations, to be compliant with legal and regulatory frameworks.

For so many of our organisations, governing their data is not optional - it is absolutely critical to their survival. As a result, if they adopt an Agile delivery approach, they have to ensure that:

Data Governance does not undermine the benefits from Agile+DevOps

and

Agile+DevOps does not undermine Data Governance

Therefore, it is paramount for us to retain and even enhance the benefits that can be delivered by Agile+DevOps and Enterprise Architecture, by making them work in concert.

But surely this is an impossible dream?

Well, I believe that it is possible, and that actually, it is very easy to make this dream become a reality. I have always held the view that if Enterprise Architecture is to be a worthwhile endeavour, it must have a direct impact on what happens at the code-face. It is only through this approach that we can deliver true agility for an organisation and, at the same time, preserve the architectural wholesomeness of its system landscape.

In the same way that many organisations will rethink their delivery methodology into Agile+DevOps, they must rethink in parallel, how they can provide a real boost to agility. This approach must be to inject Enterprise Architecture directly into the delivered systems.

Within the scope of Data Governance, we should re-use existing Data Governance artefacts and processes, to embrace this approach with negligible cost and disruption. Not only will this speed adoption and reduce costs, but importantly, it will also allow these processes to remain effective for any other methodologies, including the much maligned and out of vogue, SDLC.

Figure 2 – Symbiotic Agile+DevOps and Data Governance

As a result, the challenge can be reduced to integrating existing artefacts and processes. If this can be achieved, it will mesh Agile+DevOps with Data Architecture to boost delivery and deliver sustainability directly into the system landscape.

If these challenges resonate with you, then my latest book might just be of interest.

It has taken the principles from my Enterprise Data Architecture book and describes how to use these to successfully integrate Agile+DevOps methodologies with Data Architecture and Governance. It provides a simple step by step guide, using practical advice drawn from working with many organisations that adopted Agile+DevOps and yet needed to safeguard their enterprise Governance and Compliance.

The book is available from a range of book sellers, including Amazon here in the U.K. and here in the U.S.