A Data Deadlock & Breaking The Ice

It is 17 degrees below freezing on a cold winter's morning. A look out through the window reveals a park covered in a thick blanket of snow and pedestrians braving the frost-biting cold - but unable to conceal the white frosty breaths left in their trails.

Back in the room and the volcanic heating system is struggling to diffuse the frosty atmosphere resulting from a critical political deadline having recently been missed. The international press has made a meal of it and the tension in the air does not only feel tangible but intense and uncomfortable. The inquisition is in full swing and the finger-pointing feels physical.

Where did it all go wrong? They had all the time in the world and the risks were highlighted over 6 months before the novel publication was due to go online - but it now seemed clear that complacency was their greatest enemy.

And, where do they go from here? A radical rethink is suggested, but the new project manager insists "Muturu said nothing is to change!". Muturu was the old project manager, whose ill-planning and poor risk mitigation was arguably largely to blame for the avoidable catastrophe that has now befallen the entire organisation.

He could have accepted that the purpose-built traditional data warehouse was - indeed - not fit for purpose. He could, perhaps, also have supported the proposal to adopt a more iterative approach - for both integration and testing - but alas, the contracts for the required resources were intentionally put off until 2 months to go-live.

But, how has he - most incredibly - managed to portray himself as a hero of sorts, in spite of all his obvious misgivings, and somewhat impressively left a legacy of fear of overturning even his most outrageous decisions?

And how were the desired outputs expected to change if the inputs are necessarily compelled to remain unchanged?

The weeks that followed saw an intense debate ensue - in a bid to resolve the deadlock. However, with status quo lacking in integrity, predictability, and pace, the executives ultimately got involved and backed a radically innovative approach. This act of bravery was to deliver huge successes on multiple fronts - both in terms of operational efficiency and in terms of process adaptability.

The new and atypical approach focused on two key principles: visibility and flow.

Visibility

"Are you suggesting that we ditched the data warehouse - and forego the sizeable investment committed to it over the years - for a yet untried and yet untested, basic and simplistic, extremely novel approach?", asked the new project manager, and in the calmest - yet most confident - of tones, the consultant responded "Yes - I am.".

And so, in a significant shift from status quo - where logic was cryptic and processes near impossible to audit - this unconventional approach ambitiously set out to deliver an unprecedented level of transparency. It achieved this by exposing all inputs and outputs (including all intermediate processing steps) and making it all available - via simple, accessible and traceable data sets. The end result did not only exponentially increase stakeholder confidence (leading to an unimaginable 85% reduction in test effort) but also drove operational efficiency to levels previously inconceivable.

Flow

A key deciding factor for executives was the compelling revelation that - unlike traditional projects that struggled to deliver tangible outcomes for extended periods of time - this creative approach would lead to iterative deliverables tangible enough to be published online and across the world wide web.

"But how are we ever going to be able to deliver something so big and so complex in small and tangible iterative phases?", asked a data engineer, in an undoubtedly concerned voice. "Well, the first this we must not do is think of the complex project. We can - for instance - start by thinking about the simple things we need to achieve? For example - since we ultimately need to determine what data gets published and which does not - we could simply start by creating a flag for that?". The reassuring composure the consultant displayed - and the simplicity of the logical reasoning approach they proposed - quickly moved the team's focus, towards the small, the immediately possible and the pertinently viable.

And so - after months of uncertainty and weeks of deadlock - progress was finally being made.

Within weeks, the first data sets were published online - setting the foundations for regaining stakeholder confidence, radically reducing the tensions in the air, ultimately resolving the deadlock and breaking the ice, for good.

Thank you

PS

Names obscured or anonymised to preserve confidentiality