Available but Unusable Data - Emerging Organizational Challenge

(This was motivated a recent article on inability of organizations to apply data. This is something I am deeply thinking about these days. In this and future articles, I wish to explain in simple English what is happening, why, and what to expect over time)

Unusable Data : A Real Problem

Scribble Data launched its analytics automation service recently, and I have been on the road talking to various small and big customers about their data challenges.

I had a situation recently where a CEO asked me to help him with analysis of some operational data. I was very puzzled because I know for a fact that his 500 Cr+ organization has a 40-50 people IT team. On top of that they have BI system for a well known vendor, and are receiving services from another analytics company. It struck me as odd that they would bother to talk to a startup like ours that has existed for less than four months.

What it points to more than anything else is that decision makers are faced with more and more difficult decisions every day as competition intensifies, and they would like to draw upon data that they have within their organization, but are unable to for some reason.

My main suggestion here is that data and system complexity is the reason for the friction, and that the friction can be reduced by making usable data a first class requirement for systems - which shapes how information is represented, stored, and accessed.

Complexity : Reason Why Data is Not Usable

The problem is not access to the raw data bits but complexity of the data and the systems in which they live. High-level information such as a sale of a coffee is broken up and distributed across various tables and systems in order to make the end-to-end application work. The understanding of how the bits are laid out and what each bit means is hidden deep inside databases and application code, and takes a specialist to understand and extract.

If decision makers know that they have to consume the data, why aren’t technical systems aligned to the goals?

Systems have been historically designed with the implicit assumption that nobody outside the small team of architects, DBAs, and application development experts will look at them. The idea that a business team might want to look deep inside the application to get insights about their business is a very recent phenomenon.

Further most architects have not fully internalized the emerging need to account for frictionless access to data. Decision-makers ask for data is often narrowly interpreted as a request to add some bits to dashboard.

And we are building ever more complex systems. It takes significant skill and experience to reason about technical systems at the business level.

Usable Data : A First Class Requirement

Complexity is not an accident.

I spoke to a large governance platform company recently whose staff mentioned that dont know how to interconnect modules within their platform that they themselves developed. Therefore they cannot correlate data from the two modules. If they can’t do it, how do they expect technology experts, forget business staff, at their customer end to do it?

The technology staff that designed and developed the system is world class. It is not a competence issue. The system was built the way they did because decision makers prioritized features over depth. Data, analytics, and coherence were not even considered. Now all of us who are users of the system are stuck with a complex unwieldy platform that provides little or no insights about organizational governance processes or people.

Frictionless access to data should be first class requirement while building application systems.

The critical infrastructure that will make data accessible and usable such as interfaces, descriptions, and access methods should be defined – even if not implemented due to pressures of the situation. They can’t be easily retrofitted into complex systems.

Friction grows with time as the complexity of the system grows. It needs to be continuously monitored and addressed. Ideally a cross functional team that has substantial experience with data architecture and flows, business analytics, business operations, and technology should help drive the process.

I will expand on this subject in future. Feel free to let me know what you think at [email protected]