UNSTRUCTURED DATA - STEPPING UP THE LADDER

STEPPING UP THE LADDER – UNSTRUCTURED DATA

By W H Inmon

Recent announcements by large vendors indicate that there is increasing interest and support for including unstructured data in the mainstream of decision making. This is a very positive trend and the vendors are to be congratulated for moving this direction.

However, the vendors are about to discover some truths hiding behind unstructured data. One of those truths is that merely storing unstructured data does not yield any great business value. Certainly, unstructured data must be stored, but merely storing unstructured data does not – in and of itself – yield any great advantage.

The first truth that the vendors will discover is that there is a big difference between text and machine generated data. These two types of data are as different as East and West.

The second truth the vendors will discover is that there is an untapped marketplace bound up in text that has not been touched. There is call center data, the voice of the customer, medical records, contracts, warranties, email analysis, … all just waiting. And to date no one has really cracked the code for these marketplaces. So the textual marketplace is like California in 1848. There is gold there just waiting to be picked up if you can get there.

But for all of the promise of the textual marketplace, the vendors that announced support for the storage of unstructured data are about to discover that they are merely on the first step up a long ladder.

That ladder looks like –

In order to yield any great business value the unstructured data must be disambiguated. There is simply too much text to try to process it manually. The human brain just can’t handle it. What is needed is the computer. And there have been attempts to disambiguate text using the computer.

There are some simple ways that unstructured data and text can be disambiguated. There is stemming. There is soundex. There is stop word processing. There is proximity analysis. These are all interesting activities but these disciplines don’t really produce any great business value, to speak of.

The next step up the ladder is key word processing and NLP. Indeed, there is a trickle of business value that can be obtained using these approaches. But these approaches are clumsy, complex and expensive. They are very academic and require an army of high priced consultants. These approaches take a long time to manage and trying to wrangle business value from them is a complex hassle.

At the top of the heap for disambiguating text is textual ETL. Textual ETL is inexpensive, easy to use, and fast. And textual ETL is designed to produce business value quickly and simply.

So the movement to unstructured data is a very good thing. But if the vendors that support storing unstructured data are looking for quick wins, they are going to be disappointed. These vendors have got to move up the ladder before they can start to gain the confidence of the customer.

At the end of the day, producing business value is where the vendors have to be if they want to stick around.

Bill Inmon is the founder of Forest Rim Technology, a company in Denver, Colorado that helps customer hear and interpret the voice of the customer.