How can a data architecture leader ensure data quality?

Data Quality use cases are heavily influenced by domain-specific Use cases that impact corresponding downstream KPIs. E.g., for an e-commerce company, we cannot accept null or non-numeric values for pin codes. Inconsistency in Pin code information will hinder business leaders from visualizing deliveries in a particular area. This will, in turn, impact capital allocation for infrastructure planning (number of warehouses, delivery partners, etc.) Hence, one should understand KPIs & KRA's for all businesses before defining DQ dimensions.

Ensuring data quality involves addressing accuracy, completeness, consistency, timeliness, relevance, validity, reliability, integrity, accessibility, duplication, conformity, precision, uniqueness, consolidation, and clarity. Architects must uphold these dimensions to enhance decision-making and business processes, employing regular assessments for sustained quality.

I always liked the Four-Cs of data quality Completeness: no missing values Consistency: uniform throughout the column Accuracy: Correctness and precision Currency: Timeliness To define the dimensions or facets of data quality (Even though Accuracy starts with A)

Neben den fehlenden Werten oder NULL Werten ist die Verwendung der -für den Verwendungszweck- korrekten Datentypen für die Genauigkeit der Ergebnisse immens wichtig. Gerade bei dezimalen Werten entstehen hier h?ufig, ungewollt, ungenaue Ergebnisse oder unbeabsichtigte Rundungsdifferenzen. Diese in einem fertigen Produkt nach der Umsetzung zu identifizieren ist h?ufig mit sehr viel Aufwand verbunden. Daher sollten schon bei der Definition der Dimensionen m?gliche Effekte auf die Ergebnisse festgehalten werden.

Major points are been covered already but based my experience the business sometimes doesn’t have a clear forward process to handle and reprocess flagged data. -> during design you should consider Is your business can afford to quarantine the bad data? -> What is your strategy to reprocess the quarantined data.

Establishing data quality rules and policies is crucial for maintaining accurate and reliable data and ensuring its usability and trustworthiness. We need to establish a robust Data profiling mechanism to understand the characteristics of our data. DQ analysts should specify conditions that define acceptable data & outline procedures for data validation & monitoring. We should also be in sync with business stakeholders to gather insights & ensure DQ rules align with the needs & expectations of end users, decision-makers & regulatory bodies. This enhances decision-making processes and reduces the risk of errors stemming from inconsistent data. Trustworthy data fosters confidence in decision-making & fosters strategic planning.

Once you have defined your data quality dimensions, you need to establish the rules and policies that will govern how your data is collected, processed, stored, and distributed. These rules and policies should be aligned with your business objectives, regulatory compliance, and best practices. They should also be documented, communicated, and enforced across your data architecture. For example, you could use data quality rules to specify the acceptable range, format, and source of your data values, and data quality policies to define the roles, responsibilities, and procedures for data quality management.

In the end, the success of a DQ program resides in our ability ti implement rules & controls. I have seen many projects fail as we do not have a clearly defined implementation methodlogy along with RACI. DQ analysts must work with businesses to understand KPIs. They then need to educate business users to define the same in the tool & then work closely with the data engineering team to marry the rules in the existing tech stack across all steps in the data lifecycle.

Ensuring good data quality goes beyond just creating rules; you must integrate them into your data system. This involves incorporating checks and controls throughout the data process, from intake to delivery. Actions like validating, cleaning, and auditing data should happen at different stages. Tools like data quality software, scripts, workflows, or APIs can be employed for tasks like deduplication, enrichment, and maintaining data consistency. In essence, practical implementation is crucial to uphold data quality standards in your system.

I would say that data managers need to act like quality control managers in traditional manufacturing sense. Quality Assurance, or the policies and guidelines need to be in place as well as quality control measures where scripts are being used in order to actually sample data to make sure that it is in compliance. This two-pronged approach allows for not only a properly run and well-defined program on a qualitative basis, but also a comprehensive data sampling plan to ensure compliance on a quantitative basis.

In order to ensure the accuracy of your data, it is not sufficient to simply establish rules; it is crucial to seamlessly integrate them into your data system. This involves incorporating checks and controls at every stage of the data process, starting from intake and continuing until delivery. It is essential to perform actions such as data validation, cleaning, and auditing at various points in the process. By utilizing tools like data quality software, scripts, workflows, or APIs, you can greatly facilitate tasks such as deduplication, enrichment, and ensuring data consistency. Ultimately, the practical implementation of these measures is imperative in maintaining high data quality standards within your system.

Defining and documenting data quality rules and policies is not enough, you also need to implement them in your data architecture. This means that you need to design and deploy data quality checks and controls at various stages of your data lifecycle, such as data ingestion, transformation, storage, and delivery. These checks and controls could include data validation, cleansing, standardization, deduplication, enrichment, and auditing. You could use various tools and techniques to perform these tasks, such as data quality software, scripts, workflows, or APIs.

Ensuring data quality goes beyond setting up checks; it involves monitoring and reporting. Regularly collect and analyze metrics like data quality scores, using tools like dashboards and alerts. Share insights with stakeholders through reports and feedback. Effective implementation requires utilizing data quality software and visualization techniques to track and communicate the health of your data.

With data quality like in physics - an object observed is changed. Monitoring the quality dimensions will highlight the data that has quality issues. This is put into the right hands will make the change needed. Data will be the worst on the first day it is monitored and will improve each day.

Implementing data quality checks and controls is not enough, you also need to monitor and report their results and outcomes. This means that you need to collect and analyze data quality metrics and issues at regular intervals, and share them with your stakeholders, customers, and users. These metrics and issues could include data quality scores, dashboards, alerts, logs, or feedback. You could use various tools and techniques to measure and visualize these metrics and issues, such as data quality software, dashboards, reports, or surveys.

Root cause analysis of data quality issues will come down to 3 elements in my view: 1. Lack of controls in the capture system 2. Lack of training in the people who are capturing the data (data entry) 3. Lack of will from management to make changes in points one and two. Excepting that the above will occur in any system (and has occurred in all historic systems and data) will leave requirements for cleansing and remediation. And keep data engineering teams busy for the future.

To improve data quality, first it is important to understand what is the best fit for the organization. This responsibility of describing what can be defined as?good lies with the business. Data and analytics?leaders need to have periodic discussions with business stakeholders to capture their expectations. Different lines of business using the same data, for example, customer master data, may have different standards and therefore different expectations for the data quality improvement program.?

Automated testing and, whenever possible, correction are crucial. Given the current high volume and speed of data, relying on manual control for data quality is impractical. At my organization, we have integrated standard pyspark data pipelines that validate incoming data against different expectations and assumptions. These pipelines identify inconsistent data elements and halt the process if the severity of the violation warrants it. Additionally, we have a separate component that corrects data based on predefined procedures.

Monitoring and reporting data quality metrics and issues is not enough, you also need to act on them and improve and maintain your data quality continuously. This means that you need to identify and prioritize the root causes of your data quality problems, and implement corrective and preventive actions to resolve them. These actions could include data cleansing, correction, enhancement, integration, or migration. You could also use various tools and techniques to support these actions, such as data quality software, workflows, or automation.

To foster a data quality culture and mindset in an organization, it's crucial to intertwine practices seamlessly with daily operations. This begins by emphasizing the strategic importance of data quality by conveying how it directly impacts decision-making, operational efficiency, and overall business success. We should integrate data quality measures into existing processes, such as data entry, validation, and reporting. Automated tools and systems need to be implemented for continuous monitoring and proactive error detection. This not only ensures accuracy but also reduces the burden on manual interventions. Data quality stewards should enable a culture of openness, where issues are acknowledged and rectified without blame.

The important concern to close the loop is to have a data fixation strategy. Extracting dimensions by running DQ rules is the smallest piece of puzzle and is often confused as the end of Data Quality program. DQ initiative needs a focused approach for Data Fixation program and a workflow which might impact several development teams. The DQ dashboard should measure the data fixation initiatives after every release. The stewards need to have a well defined accountability and the steering committee should have a monthly retrospective through the DQ dashboard measuring impact of Data fixation. Note that the DQ dimensions will simply enable the retrospective through reporting but not fix the DQ issues.

It is essential to consider Data Quality from both a bottom up and a top down perspective. A bottom up approach may to use techniques such as Data Profiling at Dataset levels to suggest Data Quality rules. From a Top Down perspective its important to consider Critical Data Elements, Policies and leverage Data Quality rules to measure and monitor these Critical Data Elements holistically.

Develop Enterprise-wide data models to align with business processes/models by identifying and mapping business assets with data assets with maximizing the use of data catalog and data lineage.

Establish a feedback loop where the data quality issues identified by end user are reported and addressed promptly. Encouraging open communication to improve the overall quality issues .