How do you ensure data quality?

Before doing anything else, it is important to conduct a data quality assessment for the organization. Choose selected data sets from each subject area / function / domain and evaluate the quality of the existing data (AND METADATA) against business and technical expectations. Many of these expectations will not be documented (yet) so document them as you discover them. The evaluation and simple analysis of the chosen data sets can help guide development of data quality rules and more formal expectations. Data set assessment should be done by teams of data stewards (for business) and relevant technical team members for a balance between technical data quality and business data quality. Document all results - and plan the next steps.

Data Entry Controls Data entry controls play a crucial role in ensuring data quality by systematically enforcing standardized formats, validating accuracy, and preventing errors during the initial data input process, thereby minimizing inaccuracies, inconsistencies, and the need for subsequent data cleaning, leading to a foundation of reliable and high-quality data for effective decision-making and organizational processes.

Before defining the quality dimensions of our data, it is important for the organization to understand what Data Governance is, its importance, and its benefits. This will enable the key Data Governance roles, such as a Data Owner, Data Steward, etc., to be actively involved. It is essential for them to have a clear understanding of the different quality dimensions so that we can jointly define the most appropriate ones for our data.

Most of the data quality dimensions are easy to understand by any business leaders. It does not mean that no definition needed to start a data quality program. A formal definition of all the dimensions should be documented and clearly explained with examples. Such definitions should be easy referenceable by dynamic data community within the organization.

In general quality is based on factors that the business usually derives. So first, I would understand the Business Needs: Collaborate with stakeholders to understand what quality means for them in the context of their specific needs. Create Data Quality Framework: Define clear quality standards, including (but not limited to) the following dimensions: accuracy, consistency, completeness, relevance, and timeliness. Develop Metrics: Establish measurable qualitative quality metrics to gauge compliance with the defined standards (as set by/in partnership with the business)

In my experience, the best people to define Data Quality rules are the data owners themselves, or their appointed stewards. Only they have the required knowledge to truly grasp what rules are needed to ensure high quality data.

Data Governance Policies: Create and enforce data governance policies that guide how data is collected, stored, processed, and used that serve as the backbone of the data quality measurements. Data Stewardship: Assign data stewards who will be responsible for ensuring that data quality standards are maintained within their areas and in compliance with the data governance standards set. These rules/standards are not to be assumed do once and you are done, but rather something to be viewed in alignment with how business/systems/processes evolve and will need to be continuously improved/updated - so having a change control process overseen by a Data Governance Review Committee will be key.

Document all data rules - format, definitions, sources, and level of data quality expected for each critical data element. Note that not all data elements are "critical". Data stewards and data owners are the best sources for business data quality rules - and business metadata; technical staff working with the data stewards are good sources for technical data and metadata quality

Data quality checks can be implemented through a combination of manual processes - performed by data stewards and the technical teams - and through automation. They should be part of every new project and every maintenance effort to ensure that DQ rules are followed consistently.

Collecting all that information about data quality is futile without analyzing the results and acting upon those results. Continuous improvement is only possible from consistent analysis and attention to the results and the remediation activities.

Communication of processes for DQ, analysis of DQ rules, the use of the rules, and the results of the DQ efforts - all are part of reviewing DQ performance. Measuring compliance, presentations with data visualization, and establishing data quality benchmarks for critical data in each domain / subject area / function are essential.

Have Data literacy across the organization. Ensuring Data quality is everyone's job. Be it source systems owners, platform engineer, database developers, Analysts or Leadership. So long as Data has existed, so has the phrase - garbage in garbage out. Since Data flows through entire eco system it is imperative that everyone who touches Data in some ways understands the risk of bad data.

Maintaining a consistent data definition across an organization is vital to ensure meaningful data quality checks. Without uniform definitions, the same data element might be interpreted differently across various departments, leading to discrepancies and inconsistencies that can dramatically affect data quality. For instance, if "customer" means a paying client in one department but includes prospective clients in another, the resulting analyses could be vastly different. Such inconsistencies can cause confusion, mistrust in data, and potentially erroneous decision-making.

A good framework for ensuring data quality is the Shewart cycle - Plan - Do - Check - Act. Planning involves understanding what quality means to the business and defining data standards and acceptable metrics. The Doing phase involves communicating the defined metrics and data standards with the people, enforcing validation rules on systems, and ensuring processes align with the defined standards. The checking phase entails monitoring and review of data to confirm alignment with the established quality standards. This is where gaps are identified. In the Act phase, the gaps identified are analyzed, root cause analysis is done and the cycle is repeated by planning how the identified data quality gaps will be fixed.