DWH to BI to AI: "Data Driven Architecture" creates sustainable business advantage

 It was late 90s. Spring Time. Or so it seemed to me as I had just landed from the agriculture hinterlands (miss the real America) of Cedar Rapids to the Warmth of California Coast in Bay Area. Predictable sunshine made it feel like a bit of being back home in Bangalore. The good number of Indian Restaurants were a bonus too. And the thrill of being in the middle of Silicon Valley and trying to understand what it means was little overwhelming.

It was a great amount of excitement and fun, to start working for the Datawarehouse Team at One of the tech giants. Opportunity to work with some of the best industry veterans. Delight to be working on what was then the largest deployment of the Oracle Database in the world. It didn’t take long to realise that I had been blessed with the chance to be working with some of the best minds in the DWH and BI space.

To date, it has been one of the most comprehensive data model that I have come across. Spanning across every business function of the enterprise. Business Reporting requirements were met with a combination of sequential dependencies: Business analysis, ETL, Reporting, Production deployment. With data from three time-zones across the globe integrated and reconciled at the end of every business day. Addressing the typical issues such as late arriving dimensions and ledger entry udpates. Ensuring the jobs ran within those few hours of limited time window.

And then there came Brio (It was there in the enterprise for some time, my exposure to it was little later). A window to everybody into the hidden area of underlying tables and columns. A magic wand of drag and drop and integrate. A simple way to configure and publish reports. A freedom from the “Surgeons of IT World” who always needed more compared to when the business actually needed it.

I loved the tool. It could do most everything in terms of everyday requirements, it appeared. In fact, at that time, I could think of very little beyond what it could already do. It certainly made things easier and simpler. Console power was transformed to screen power. Programming was translated to configurations and clicks. It certainly changed the paradigm.

An Early Nugget of Wisdom:

I remember that coffee conversation in the cafeteria. With quite a few senior members of the team around the table. All of them reputed experts of the state of the art technology. With some young and eager minds including myself attentively listening in with curiosity.

The future of the Data-warehouse seemed to have found its elixir in Brio. Or the new age tools of that time in that league. Data could be dragged and dropped at the click of a button. Reports could be configured by the business analysts. Data Marts could be directly managed by the business owners. There need not be translation gaps from business to IT.

Of course, the IT team would be needed, but dependency on that team will reduce. Data administration will continue to be addressed by IT, but Data Integration and Reporting will progressively get addressed by business analysts.

Tom, who had seen the generations of technology, commented wisely, “Just as data volumes continue to stay ahead of our ever expanding disk capacity and speed; it is to be expected that the complexity of the data integration and reporting will continue to stay several steps ahead of the available technology.  New age tools will help, but it will not suffice. They will complement, but will not replace the need for the “business specific data engineering”.

“There is another point to note and remember”, continued Tom. All these tools are like the cooking oven or microwave, which will bake what you put in. But what matters is, what you put in to bake. And how much you know about baking. And the conversation drifted off to the home-made cookies one of the members had got, for the post lunch coffee dessert.

Fast Forward a few years:

Fast Forward a few years to the 2000s. We saw the era of specialist tools taking the centre stage. ETL companies such as Informatica and Data Stage, specialist BI companies such as Business Objects and MicroStrategy. And many more at varying degrees of adoption, maturity and excellence. Each with its own distinct features and capabilities and unique offerings that would make it easier for the enterprises to realize the DWH and BI roadmap for the enterprises. They rose in their stability, popularity, adoption. The debates on Kimball or Inmon and ROLAP / OLAP / HOLAP filled in days and days of corporate war rooms pending the final decision on the Million Dollar software acquisition.

Fast Forward to the 2010s, to the world of Self-Service BI. Many popular self-service BI tools empowered the businesses and business analysts. It became as easy to whip up a new dashboard, as popping the popcorn in a microwave oven. Business analysts felt empowered. Stakeholders excited as all reports were available when asked. Any internal and external data can be combined with any data from any department. It was a bingo time.

Until the watchful stakeholders started seeing that something seemed to be amiss in this “You have it when you ask for it” reports and dashboards.

The report that was shared online last month did not seem to correlate with the report that is shared this month. And when asked, “Let us go back to the last month’s report and check that with this new report”, were told tool does not allow that. And it became little more embarrassing when a watchful CXO or a board member noticed conflicting numbers on the presentation, and probed to understand what is the right number.

And it became untenable when the numbers from manufacturing and sales; and sales and finance, procurement and manufacturing, and one department to another, started playing “my number is the right number because your number is not my number” game.

Not that these numbers from across the departments were in tandem prior to the self-service BI tools. But there was an understanding and appreciation of the lack of consistency and the reason for it, which were factored in by the IT surgeons. There was a gate-keeping release mechanism. There was a point of accountability. With the “it is all on the web” self-service BI reports, there developed a false sense of being in synch. Everybody assumed that everybody else was also looking at the report being looked at by her/him.

Coming to the “Today” of AI:

2020s are the days when anything less than Analytics is passé, and a mere BI is old and outdated. Ambitions of the organizations make it feel like predictive analytics is well below their dignified needs and anything beyond the Deep Learning AI would underserve their decision intelligence capabilities. Leave alone the BI. Don’t even remind about the DWH. If BI is old, DWH is ancient.

However, both the business and the IT invariably start discovering the issues and challenges around data quality, data consistency, dimensional divergence, differences in granularity, hierarchy and vocabulary of the metrics and business rules. And the lack of the data to support robust predictive analytics. The routine task of calculating a derived variable such as “sales to shipment ratio” would suddenly become a mammoth task. And the force of the powerful tools will make it easier to continue with the siloe approach. Building on the “data deltas” and contributing to data explosion, duplication, multiple versions of truth.

These are the kind of small issues that complicate the analytical journey across the organization. They make the Deep Learning AI ambition seem like driving a Fast Ferrari in the crowded Silicon Valley traffic (be it the Road from Silk Board to Whitefield in Bangalore; or the 101 stretch leading into San Francisco).

The differential:

Tom had a point, even though it was way back in 1998. “Architecture is not the same as the tool, as much as the quality of cooking is not the same as the class of the cooking range”. One could have the latest and the best possible cooking range in the kitchen, along with the best of the recipe, but the quality of the dish is going to be dependent on the two factors. Quality of the Ingredients. And the Ingenuity of the Chef. Both are essential.

Tool Driven Architecture:

We see recurring waves in the industry where there is a generational shift in thinking “The tool is the Architecture”. It is like mistaking the presence of Treadmill for the actual workout. It takes a few years of experience and few cycles of implementation to realize that the “Good tool can be great with the right Architecture”. And “a great tool can become part of a very bad solution”, without the requisite data and DQ (The Ingredients) and the thoughtful design and architecture (The Chef).

It is an early and important choice every company needs to make; in terms of the solution path they undertake. Whether to follow a tool driven architecture. Or to follow a Data driven Architecture. Few suggested considerations to take into account and keep in mind, while arriving at the right decision:

1. Half life time of the tools are shrinking rapidly. The time a tool can remain as a leader is much shorter, the probability that the tool will be outdated sooner than later is much higher.
2. The rate of evolution of the technology is increasing, bringing in more competitive tools into the market.
3. The concept of “my source system” is vanishing, or the line is being blurred. As a result, “Architecture dependent on my system” will continue to taper away. Closed architectures will be forced to open up.
4. The nature of data integration itself is changing, and will continue to change, due to increasing nature of collaboration and co-opetition in the industry.
5. The rate of acquisition and consolidation in the industry is on the rise. Of the 32 DWH and BI related ticker symbols saved in my Yahoo finance portfolio in 2000, there are very few that remain, and even among them, there are only a couple with their original ticker symbol being intact.

That is, the tools, technologies, applications, interfaces will continue their journey towards continuous change, with increased speed of technology transformation. Rate of acquisitions and ownership change is almost assured, every few years. Leading to the change in the direction and focus of these tools, as a result.

Nature of the data and decision making:

While the tools, technologies, applications will continue their journey towards continuous change, the content and nature of “Data” remains fairly constant. Consider these,

1. The “informational needs of the business” are not influenced by the underlying database technology either. They remain the same independent of whether the data is stored in a flat file or rdbms or document-db or big-data or fluid-data.
2. The nature of the business questions remains the same. “What is happening where, why?”. “What can influence it, and how do we make that happen”. “What is changing or has changed”. “What is influencing the change”. “What can we do about it”. And the like.
3. Customer attributes remain customer attributes, save for the fact that more attributes could get added (with changing lifestyle and increasing channels of interaction)
4. Data associated with the process lifecycle, say customer lifecycle, remains the same, save for the fact that more granular details or events and interactions could get added or removed.
5. The definition of the Sale or a shipment or a return or a payment remains the same, even if the channels and modes are to change.
6. The demands of the reporting, auditing and compliance worlds remain focused on these foundational metrics, and have no bearing on the underlying tools or technologies. But for the difference they can make in getting them in timely and accurate fashion.
7. The “informational needs of the business” are not influenced by the underlying database technology either. They remain the same independent of whether the data is stored in a flat file or rdbms or document-db or big-data or fluid-data.
8. Data is flexible to be morphed into different formats and structures (read db technologies), but its form and substance remains the same. It (data) still has to point to what it is about (dimensionality), and what it actually is (metrics), when was it about (time) and what action by whom(persona).

The metrics and KPIs the business stakeholders need, at their fingertips, does not change. It remains the same, independent of the tool and technology being used. The speed and timeliness and accuracy with which they need to be delivered may change, the context of when and where and how it is to be delivered may change, but what is needed remains a foundational requirement.

Data Driven Architecture (DDA):

It is high time enterprises and businesses, whether in the start-up mode or in the fortune five hundred league, need to move towards the “Data Driven Architecture” for their Operational and Strategic Decision Intelligence Journey. Many have successfully traversed this journey. Many are on the way. Some are doing it selectively. Safe to say it is a continued journey for all growth companies.

DDA calls for a Trusted Data Foundation to be orchestrated; either physically, logically or a combination thereof. With the data foundation that is not shackled by the constraints of any tool. Instead, the chosen tool empowering the data foundation of the enterprise, for scale and speed and consistency.

So that the ‘decision intelligence’ that is being dished out is powered by the “trusted data ingredient”. The “cooking range” of good tools can then make the underlying data more appetizing. To serve the “Consumable BI” that is healthy and nutritious. The good tool now becomes a great tool. And it empowers the data and business together.

As my colleague Gowthaman says, Data is Chiranjeevi (immortal or everlasting), and a data point once produced does not change its form (may change the format) over the life of the enterprise. And the power of this granular data with point in time snapshots is like the Hanuman (powerful and adoptable), when put to the right use.

Make your data assets pay off. Make this immortal asset of the enterprise create that competitive moat for your business. The right time to start this journey is today.

Creating the trusted data foundation to enable the “Data Driven Architecture” calls for the specialised knowledge of “Business Acumen + Process + Data + Domain + Technology”driven approach. COMPEGENCE brings in this unique methodology with deployment ready solutions towards enabling the “Data Driven Architecture” moat for your enterprise. Reach out to us to evaluate the health of your DDA and how you can increase the resilience and the speed of response to your business.You can reach out to us at info at COMPEGENCE dot com and follow us on twitter handle @COMPEGENCE

This originally appeared on COMPEGENCE Knowbase