Understanding Data Quality

The fundamental paradigms underpinning data quality and quality data have reversed from where they were even 20 years ago.? The costs of storage and industrial strength relational databases have fallen so dramatically that data collection has completely replaced data quality.? Once much of the world’s data was trapped in flat files and each kernel had to be of critical value. At the time, the expectation was that the integrity of data was verified before it was entered. There wasn’t the money or space for storing poor quality data or for fixing it afterwards. The high standards and good practices of entry personnel, clerks and postal workers ensured the accuracy on each record.?

Many of us suffer under the delusion that our information still receives this level of care, as if we are living in the world portrayed in a Norman Rockwell painting. Unfortunately, we need to become much more cynical and realize that we live in a world of keystroke counts and workflows that rarely value the quality of the information entered.

Companies are leaving data entry up to the customer: sign up for our membership, register your appliance, fill out the satisfaction survey, review the product, fill in this form.? The consumer who fails to enter correct information may lose their warranty, never receive their service, be overcharged or turned down for a loan or find erroneous information in their credit reports.?

Meanwhile bots or data scraping programs promulgating inaccurate data circulate across the online world.? Tech corporations and their bricks and mortar counterparts are retreating from any responsibility for the quality of the data they retain, especially because the sale of our data remains a profitable business. Once the consumer checks that box of pages of tiny print they finds themselves left with the overwhelming task of ensuring their information across the web and in a thousand and one databases is both accurate and current. This same customer also finds themselves abandoned to deal with increasing infestations of spam, bots, fake accounts and outright scams on social media all by themselves.

Even the Social Security Administration charges retirees when their own software has miscalculated and over payed. They just dock future payments to claw back the funds rather than write-off their mistake. The resounding message you should be hearing is that the quality of your data has become solely your responsibility.

For this new and personal model of Data Quality and Data Governance to succeed requires us to create new arguments and rationalizations for the importance of good data and why we need to convince the organizations retaining our data to care about not just its security, but its quality.? It is time for a new paradigm, fully engaged in our modern reality, that makes data “ownership” simple and provides owners with effective tools to observe, maintain, and retain their information.??

I will explore some of these ideas in upcoming posts