RIGID METADATA/FLUID METADATA

RIGID METEDATA/FLUID METADATA

By W H Inmon

Programmers and designers have learned metadata from the day the first program was written and the first data base design was created. Designers learned about rows and columns and column headings and the like. And these notions of metadata have served well and long.

But the world is changing. Once there was only structured data. And the day of structured data was the day when metadata was first discovered. But in todays world people are discovering that there is great value in looking at textual data as well as structured data. Structured applications have long been exploited. But the world of text is a virgin world and it is full of opportunity. Textual data sits there as a ripe plum just waiting to be picked.

The textual world is very different – in many ways - from the structured world. The rules and practices about metadata learned in the structured world all have to be reimagined and relearned in the textual world.

One of the most fundamental set of rules that needs to be revisited is that of the discipline surrounding metadata. In the structured world metadata can be classified as “rigid” metadata. But in the world of text there is a new form of metadata – “fluid” metadata. The stark differences between rigid metadata and fluid metadata are but one of the many fundamental differences between the two environments.

To explain this difference between rigid metadata and fluid metadata, take a look at the two data base structures shown in the example. One data base structure represents the structured world and the other data base represents the textual world.

On the left hand side is a classical structured data base. The data base on the left has three types of data – name, city, school. Every row value conforms to the definition of the column. Every cell of the name column is a name, and so forth. This conformance to the definition of the column definition is characteristic of the structured world. The data in the column is rigidly placed there.

Now look at the data base structure on the right hand side. The data base on the right hand side is typical of a data base rendition of textual data. There is a record id. Then there is a word column. The word column reflects the words that have been found in the text document that has been read. Then the context column describes the context of the word. The word column is not rigidly defined at all. The word column is defined in a fluid fashion. Any word of any type could find its way into the word column. There is no conformity to the column heading whatsoever in the word column.

The metadata then that is used to describe the structured data is very rigid. The metadata used to describe the textual data is starkly fluid. The data in the word column is not rigid at all. It is fluid and malleable.

The following figure shows the difference in the types of metadata.

When a metadata definition is given in the structured world, the metadata rigidly defines the data in the column. But in the textual world, the column heading makes no mention of the type of data that can be contained therein.

The ability to define metadata fluidly is a necessity for the textual world. In the structured world data typically comes from transactions which are run in a repetitive manner. In terms of structure, one transaction tracks the same data as any other transaction. Only the contents of the transaction vary. The structure of the transactions remains constant.

But in the textual world there is no repetition that occurs in a regular fashion. The author of the text can say anything he/she wants. Therefore, a different mode of capturing metadata and defining the context of data is required for textual data.

This discussion has been on only one of the many differences between the structured world and the world of text. It is the proverbial tip of the iceberg. More to follow.

?

Bill Inmon lives in Denver with his wife and his two Scotty dogs – Jeb and Lena. It snowed earlier this week. But the snow has pretty much melted off. Jeb and Lena are enjoying the backyard. Except where there is a snowdrift that has not gone away.

?

?