AVOIDING THE HORRIBLE TASK OF INTEGRATING DATA

AVOIDING THE HORRIBLE TASK OF INTEGRATING DATA

By W H Inmon

For many years now, vendors and consultants have avoided the practice of integrating data. Integrating data involves – complexity, making mistakes, searching through undocumented systems, making assumptions, understanding decisions that were made long ago, etc. Integrating data requires a four letter word no one wants to hear – work. And lots of it.

So how have vendors and consultants avoided integrating data? Vendors and consultants have a long list of excuses for not integrating data –

??We didn’t invent the technology for integrating data so it isn’t worthwhile doing

??We do ELT, not ETL (And we conveniently forget to do T)

??There’s just too much data. We can’t handle that much data

??We are already busy. We don’t have time to do all of that work

??We can’t get our users to agree on anything.

??We can just federate data and use data in place. There is no need to unify or integrate the data

??Integrating data is complex and hard.

??We threw the documentation away (or the documentation never existed)

And the list goes on. Ho hum….

But there is a new excuse that we need to add to this list of excuses for not integrating data. The new excuse is – let’s use data in place. Copying data is expensive and time consuming. Let’s just leave our data where it sits.

This sounds like a reasonable argument, but this line of thinking is anything but reasonable.

First off, when people go to unify and integrate data, the fundamental act they are performing is TRANSFORMING data, not copying data. ?It is true that some amount of copying of data is necessary in the act of transforming data. That is incidental, collateral damage. But when people go to integrate data, a certain amount of copying of data is absolutely necessary and absolutely unavoidable.

The net result of transforming data is the ability to unify and integrate data. Once data is transformed the data can be viewed as a single entity. But without transforming data you cannot have a unified view of data across the organization.

The choice is simple and basic – you either transform data and have a unified view of the data or you don’t transform the data and leave it in place and have a fractured view of your data. This choice is binary. It is one way or the other. It is as simple as being alive or being dead. You can’t have it both ways.

I am aware that the transformation data – integrating data – is difficult. Messy. Time consuming. Complex. Full of guess work. I am aware that the copying of some data has its pitfalls. But that is the price you pay for wanting to look at data in a unified, integrated manner. The choice is binary and simple – either you transform data and look at it in a unified integrated manner or you don’t transform data and you look at your data in a fractured manner. You can have either A or B. But you can’t have both

The good news is that you don’t have to boil the ocean. Not all data needs to be transformed. And certainly, the data that needs transformation can be divided up into phases, based on criticality of need. In fact, most data does not need to be transformed. Only the data that needs to be shared and examined in a unified, integrated manner needs to be transformed. But trying to leave data in place that needs to be transformed for the purpose of having a unified, integrated view of data simply is not a long term winning proposition.

There unfortunately is no compromise in this proposition. You can’t have one foot in the water and another foot on land. NO COMPROMISE. You choose A or B. But you can’t have both.

NO SHORT CUTS.

Bill Inmon lives in Denver, Colorado. Bill’s company – Forest Rim technology – does textual disambiguation. Textual ETL reads text and turns text into a standard data base.