How do you prioritize data quality for different domains?

In considering data quality dimensions, it's crucial to recognize that data isn't just a company asset; it's also a product, serving the needs of customers/users, facilitating decision-making. From a decision-making perspective, we can distinguish between indispensable and nice-to-have dimensions. Eg., data used to halt a suspicious money transfer must be delivered faster than the transfer itself, meaning timelessness is a must-have. Conversely, data for decisions on launching a new product may tolerate delays of a few weeks. By applying this rationale to each dimension, we can ascertain their essentiality for users. This approach empowers us to prioritize dimensions that contribute most to decision-making, thereby creating value.

Prioritizing data quality is essential to sound data management. I would start with fully understanding the end game. For regulatory compliance in most domains, the accepted practice is to have the highest level of DQ. Similarly, marketing pitch books to sell financial instruments to prospective institutional clients must be highly accurate for regulatory and reputational purposes. However, for use cases and business objectives, where accuracy and completeness are accepted within a certain range, it is critical to understand how much effort a team needs to exert. I have seen teams pursuing data quality on elements that are not highly prioritized or do not need to be highly accurate. This takes away their focus from other priorities.

The data quality dimensions also depends on the data attributes within the domain. Not all dimensions finalized for the domain will be applicable for all the dat attributes within the domain

In my experience, the first step to prioritize a domain or a data element is to understand the business problem e.g. If a company is not able to cross-sell or up-sell, they might have a issues in effectively segmenting, categorizing and/or classifying their customer data. So the quality of data being maintained in the CRM or other applications is suboptimal. Therefore, the first step is identifying the critical data elements that impact business processes. This understanding serves as a fundamental prerequisite before the company embarks on a data quality initiative. It underscores that Data Governance is key for the successful management and maintenance of data quality

Data quality is to be one of the most important instruments to get value from data, it is a prerequisite. If data not ready, the value may be less than what is expected. Data value may come on form of more insights about customers, efficiency of operations, .. To get data quality you have to work on both sides : IT Systems that nurture this data by putting controls in entered data (combo box, lists, auto complete, ... ) & by cleansing existing sets of data : IA may help at this stage.

It's essential first to identify what you're looking to improve or sustain. Once you've tailored your metrics, go beyond measurement and aim for actionable insights. For example, if data accuracy is below a set threshold, understand the root cause. Tools like Alteryx or PowerBI can provide dashboards with real-time metrics, serving as a continuous monitoring mechanism. Look into automated solutions that correct errors and log them for data cleansing. Consider adopting a balanced scorecard approach, incorporating both quantitative and qualitative metrics, to provide a more holistic view of data quality. Periodically review your metrics in relation to changing business objectives, compliance requirements, or technology landscapes.

Collect cases with poor Data Quality which cause real business impact. Explain the impact and provide a meaningful picture of what should be monitored and tracked. Visibility is the key to get business support for a DQ initiative. Make this visibility clear, regular with defined action items and tangible outcomes.

In order to define Data Quality metrics, talk to the ground level employees who actually face the issues in their day-to-day working. By asking the right kind of questions, you'll be able to identify the DQ dimensions that should be prioritized for the specific business group. Do not aim to measure all dimensions at the beginning. Start with the most critical ones. This ensures faster ROI for the business and helps them understand the importance of investing time and money in the Data Governance activities.

All data quality tools measure data quality post facto or not real time making it difficult to provide timely alerts to the data owner

Metrics give life to abstract data quality dimensions, turning them into actionable insights. By benchmarking and consistently measuring these metrics, businesses can keep a pulse on their data's health. It's not just about having data; it's about ensuring it's robust and reliable. Tools and methodologies streamline this process, but the ultimate goal remains: derive value and trustworthiness from our data. Remember: "If you can't measure it, you can't improve it." - Peter Drucker

An important part of this exercise is also setting up proper data ownership. Data ownership in many firms is a federated process where the engineers send the log events, data engineering team ingests them into data lake, subsequently modelled by the BI Team and eventually used by analysts and data scientists across the board. The process gets heavily broken down due to different teams involvement, lack of ownership etc. I think the BI and Analytics team need to own these definitions and establish the data quality rules which also evolve as the business does. For eg a company may not be offering a COD payment at its inception and may offer it later. The data quality checks on the payment mode field needs to be appropriately changed then.

One of the 4 guiding principles of Practical Data Migration (PDM) since its first publication in 2006 was that "No organisation needs, wants, or will pay for perfect quality data". Central to the management of this is the the Data Quality Rules (DQR) module now in its 3rd full revision (PDMv3). This brings together all the Key Data Stakeholders (KDSH) to make the call on prioritising remediation. KDSH can be internal, or external to the organisation (e.g. regulators). I always say that it may be expedient to substitute some other practice for some of the modules during the tailoring process at the outset of a project, but if you are not using the DQR process and artefacts you aren't doing PDM.

I often picture data quality rules as the traffic signals directing your data's journey. These rules set the standards, defining the expected quality levels for each domain. They act like guardians, making sure your data aligns with business requirements and user expectations. Whether it's at the collection, transformation, storage, or delivery stage of the data lifecycle, these rules—enforced through techniques like verification, standardization, enrichment, and correction—keep your data journey smooth, safe, and in tune with the desired quality.

Data quality rules set clear benchmarks for maintaining data integrity across its lifecycle. Essential for aligning with business and user needs, these rules can be enforced during data collection, transformation, storage, and delivery. Key techniques include data verification against standards, ensuring uniformity via standardization, enhancing data through enrichment, and rectifying discrepancies with correction. In essence, these rules ensure data remains consistent, reliable, and fit for purpose, paving the way for informed decisions.

Identifying the right node to implement data quality rules can have huge impact on the quality and timeliness of data, and the effort spent in maintaining the good data. Going for a proactive approach wherein the rules are enforced at the point of data entry ensure good quality data being injected into the downstream systems. As opposed to a reactive approach where data needs to be "fixed" at various levels in the data pipeline.

"What you see, is what you believe"! Monitoring and reacting to the data quality trends is not only relevant to improve the quality but also to showcase the quality of processes within the organization. Domain leaders should use these trends to identify areas of process improvement and automation .

Imagine data quality processes as the gears that power your data engine. These are the dynamic workflows ensuring top-notch data quality in every domain. They're not just routines; they're your co-pilots, aligning data quality goals with business strategies and user expectations. Think of it like fine-tuning your engine for optimal performance. Use proven practices, frameworks like the Data Quality Assessment Framework (DQAF) or the Data Quality Management Method (DQMM), and insights from audits and evaluations. It's like having a GPS for your data quality journey, constantly guiding you towards better outcomes.

DQMM provides a structured approach to identifying, measuring, and improving data quality by defining standards, processes, and responsibilities. By following the DQMM, businesses can enhance data reliability, accuracy, and usefulness, ultimately leading to better decision-making and operational efficiency.

Monitoring data quality trends offers insights into data's evolving reliability. These trends help in gauging the success of data quality rules, pinpointing issues' root causes, and planning corrective actions. Using tools like dashboards for real-time visuals, reports for periodic evaluations, and alerts for deviations aids this process. Benchmarking against industry standards or competitors offers a comparative view. Transparent communication of these trends to stakeholders not only builds trust but also ensures that data remains a reliable asset for decision-making.

Not all data is created equal, nor is it used equally across domains and use cases. the level of data quality required can vary significantly, For instance, exploratory and analytical use cases may not require the same level of data quality as regulatory and financial use cases. Data Quality objective is to ensure data is fit for its purpose. To ensure that data quality aligns with the requirement, consider these key steps: 1.Engage Your Business Stakeholders 2. Define Data Quality Goals for the domain and usage 3.Tailor Checks and Reports to the specific Use Cases: 4 .Implement a Data Quality Certification Process: Depending on the required level of quality, establish a certification process for different data usages and purposes.

The term 'garbage-in, garbage-out' is one of my favorites and will span all time. No matter the domain, prioritizing strong data quality including validation, cleansing and ongoing monitoring is crucial for a strong foundation. Ensuring data accuracy, consistency, completeness, security, timeliness, reliability are all critical success factors in delivering value from your data assets. In life sciences, misdiagnosis or errors could harm patients. In banking, increases in fraud or regulatory non-compliance can occur with poor data. In retail, incorrect recommendations, pricing errors, or supply chain inefficiencies. Regardless of the domain, ensuring data quality and applying these principles is a continous process and high priority.

I would think SOX caused CFO to be very vigilant. We can borrow that scenario of well governed financial data assets as to why that worked. After all, data is a business asset, not a technical asset coz at the layer of business it’s knowledge that’s camouflaged as data. If we treat that as going-in position, the viewpoints will change.

Quality data means quality service delivery. Hence, maintaining a consistent quality of data is the hallmark of robust data governance policy. But, it is crucial to understand that quality of data will different for each domain. So these steps may be taken for domain specific data quality: 1. Define clear objectives for your domain service area, so can streamline data collection and quality. 2. Collect relevant data necessary for your services in the domain area. 3. Conduct regular data audits to track your collected data. 4. Dispose of unnecessary backlog to maintain a clean process and ensure data quality.

Prioritizing data quality is crucial in any domain to ensure that stakeholders and users can rely on the data for making informed decisions. First and foremost is understand Stakeholder Needs: Engage with Stakeholders: Regularly communicate with stakeholders to understand their specific data requirements, challenges, and expectations. Define Clear Objectives: Clearly define the goals and objectives of data collection and analysis in collaboration with stakeholders. This will help prioritize the domains, for example, Master Data Quality is always a priority before we take up other domains as it has a downstream impact.