What are the tools and techniques that you use for data quality improvement methods in data governance?

There’s little mystery to fields like name, address, age, and date. And you’ll rarely profile data blindly because vendors should supply documentation like coverage, currency, layouts with data types and definitions, reference tables, counts, and field populations. If they don’t, they’re the wrong vendor. (Scraping is a different story; so are internal data sources.) Use consistent, descriptive, succinct, field names and definitions. With words, not codes (and don't use ChatGPT :). No one should have to reference a separate table to figure out that PNL_CRGT_R_VNL means a paneled, corrugated vinyl roof. Or even worse see some alphanumeric. Professionally print your Data Dictionaries and use them as sales collateral. P.S., Use spell-check.

Use visualizations to make your discovery and analysis more understandable. It is much easier to see outliers and anomalies in your data using graphs than in a table format.

Cleanse and de-dupe all you like but keep the bathwater. You never know when a project or product will require that data. Not to mention lawsuits and audits. I’m a big fan of the “belt and suspenders” approach: Process, store, and platform both unparsed fields and parsed ones; formatted and unformatted. Storage is cheap. Keep them all, be ready to platform them all. I once had a seven-figure opportunity to license a legacy dataset that had been purged; deemed unusable. It was found in a cardboard box in a closet, stored on a series of unlabeled Iomega ZIP disks. (We bought the drive on eBay and the deal closed.) Commit to a central library for all data and documents that can survive change of ownership and the evolution of technology.

Just because data passes validation does not mean it’s useful much less used. Memories are short. Data can be processed religiously for years without a hiccup but if no one on the business side knows the data is production-ready, it might as well not exist in the Universe. You’ll keep validating. Vendors will keep invoicing. And Accounts Payable will keep paying. Testing the data? Ensuring that it’s fit for purpose? Unless you’re launching a new product, model, or report, those things should be established well in advance of contract execution. In the end don't let an errant validation failure hold you back. If, say, the demographic file fails validation, well, proceed to load the 29 other files that passed.

I question performance and quality as measures applied to data. The #1 thing to monitor for? If the file actually showed up. And always actually read the README file. You can monitor your data but you can't monitor for stupid. Blame your execs when they opt not to license critical data due to budget; fail to stay abreast of the data marketplace; or choose to ignore customer feedback. Finally, you can't monitor your clients' warehouses. Clients will blame providers when in fact they did not load the files. Or loaded them out of sequence.

Encourage cross-departmental collaboration. Sharing insights and data monitoring responsibilities can lead to a more cohesive and effective approach to data management.

DMBoK, for example, is really about stating the obvious. Large companies almost always plan to implement frameworks and comply with standards… eventually. Or maybe they rolled out an enterprise-wide project and training but it never took hold with one rogue but huge division. And does anyone really expect innovators / startups to buy into ISO 8000? They might as well be staring a caveman in the face. Their very purpose is to shake things up. I give them a pass on participation in standards committees umpteen times a year.

Data Governance Step to be involved before Data Monitoring. Data Governance involves establishing processes, policies and standards to ensure that master data is accurate, consistent and secure across various systems and applications within the organisation. Data Governance roles, responsibilities, policies, security, privacy, availability & usability should be elements for governance.