How do you prioritize data quality when resources are limited?

It is hard to prioritise DQ when resources limited. The best way is to develop a holistic view, identify very specific business area/outcome that may be impacted due to bad data quality, and start working there.

Focus on the pipelines producing the highest value for the business. To paraphrase Shewhart "All data is noisy/dirty, and some data contains a signal under the noise." To get the highest ROI, put your signal baseline discovery and cleansing efforts where the business depends on the signal.

The approach that I highly recommend is to treat DQ as a "product." Start with proper research/discovery and identify the minimum viable product (MVP) with direct business impact. In short, identify key high-value models(ML) or reports and start with the health of its data. Validate, monitor, and report on that, eventually automating the DQ workflows. Iteratively build and expand from there ensuring that high-value data assets are always accurate.

DQ for me is 2 way approach and any one can be picked depending on what is pressing business need and needs immediate attention. 1. Use case driven - Where you know north star and challenges faces and to map DQ rule to resolve it to get ROI asap 2. CDE/KBE driven - Where there is ambiguity and we need to hunt the issue by profiling and creating rules to bring the value out.

Focus on completeness, accuracy and uniqueness. Don't boil the ocean and place controls on every data point. Identify data fields that, if wrong, negatively impacted business decisions and maintain the integrity of those .

The key of adding a control, is that a control is to counteract, mitigate or "Control" the risk of something going wrong. By managing an comprehensive list and understanding of data quality risks, and how those risks occur, and how likely they are to occur you are more effective in your control design. Using risk management vernacular significantly helps in data quality control design. Controls can be split into - detective controls (data quality rules exceptions), preventative controls (validation rules in an ingestion framework). Each time an issue or break is reported from a detective control, there should be a control remediation (ie, adding a preventative control). These could be as simple as drop down lists or field restrictions.

Data quality means different to different audience as a measure of quality may be relevant to one group of users and not others. Key is to find which quality measures as the max impact and focus on those like a quality of an airplane engine part is far more critical then the quality measure for a seat recliner.

One thing about Data quality control or check is to start with Critical Data elements for any particular Reports. Another way is to start with simple Dw dimensions such as Completeness or Format related Dq checks. You can defined the specific Dq thresholds for every Dq control and follow Data quality issue management process for the DQ checks which falls under the defined threshold.

Start simple with data quality and build modular tests. For example, if it's significant when a record is missing a field (country) or when one field is present without another (city/state), create an independent test for each. Over time, you'll build unit tests for data quality that will scale independently.

Indeed limited resources is a biggest blocker for implementation as to build comprehensive tool might cost a lot, especially if you are planning to have not only automatized detection, but also automatic rectification of inconsistencies. Therefore, I see a need in really strong business case along with leadership buy-in based on the prioritization of business impact. In general, automation is vital as manual maintenance (detection and rectification) might be driving a disengagement in the team, not mentioning that performing one time cleanups, from my pov, is unsustainable approach as is, especially in less controllable environment in terms of access.

The key to monitoring of data quality is presenting the data in a way that stakeholders understand business impact, and enough contextual information to take action. Monitoring should lead to the top, so that the behaviours are reinforced by management to remediate data quality issues.

There is much more to data quality than cleansing and remediation of data. Identifying root causes and correcting the source eliminates the need for continued rework. Understanding what data is fit for what purpose and defining minimum requirements for data are also essential to prioritise efforts.

1. Find the business area/process/system where you can make the most impact. Use your story-telling skills with facts to make a compelling case. 2. Have really good Standards, Guidelines, Controls and Procedures. Weak fundamentals can quickly reduce the trust on your ability. 3. If you don’t have the best tools, use what is available and use it smartly to deliver output. Automate as much as possible. 4. Get the technology teams' buy in to your journey. 5. Use only necessary metrics to measure DQ progress. 6. Tie DQ to a risk if possible. 7. Prevention is better than Detection. Embed DQ controls within business or technology processes where possible. 8. Show tangible quick wins & gradual improvement to sustain interest and sponsorship.

It would be better to build data quality controls into your processes to eliminate the high chances of data contamination. If all the processes include a data quality measure/control, monitoring and evaluation becomes easy and requires less resources. Where processes are personnel-dependent quality training should be part of their skills improvement. Including quality control/measures in the planning phases of each business task will help mitigate data problems and will improve data usefulness.

I'd put in a caveat that different use cases call for different metrics and standards for "data quality." Easy to understand are the different weights (certainly some are more important to others), but more difficult to grapple is the reality that "bad data" is in the eye of the beholder. Late data may be acceptable to some, abhorrent to others, and even desirable still to others (if, say, it means more "acceptable" data). Ditto for stale data.

Data quality matters when your stakeholders work with you to define the shape of the data they want. When they recognize that data is improving, you both win.

We found that once you have a way to monitor and report on data quality, you are most of the way to having a continuous improvement process. We had a monthly meeting with all teams responsible for data quality. Using the visibility of the monitoring and reports, we collated and discussed the most severe data quality incidents. Together we came up with action items that would prevent similar issues from occurring in the future, or at least minimize severity. If your teams collaborate well and consistently implement their action items, you should see your data quality reports improve over time.

In my experience it is important to highlight the impact incorrect data can have on business decisions to be made. Based on this kind of mapping, it is also possible for the organisation to decide which data quality checkpoints need to be put in place and where data quality checkpoints might not necessarily be needed as they would not influence decision making process.

In management reporting, if I oversimplify the data flow: -database-business logic-reports -Automated validation points can be implemented in between, then any mismatch will stop wrong data to flow through in the first place -By sharing validation rules and error logging, this will help data team to identify root cause.

It is essential to create an open dialogue between all stakeholders - data owners, producers, consumers, and users. The establishment of clear data quality policies, roles, and responsibilities creates a structured approach. Furthermore, implementing effective feedback mechanisms allows for the quick identification and resolution of data issues. Regular training and education sessions can assist in establishing a strong data quality culture across the organization, ensuring that everyone understands their role in maintaining data integrity.

One thing I have found helpful is to factor business engagement into our prioritisation. Yes, importance of data is part of the equation, but if business units are not ready to engage and provide resource to support data quality, then lots of effort can be spent following up on required SME input and owner sign off. Sometimes doing the less 'important' measurement with and engaged unit first and building momentum can produce benefit in the long run. Fast delivery builds momentum and momentum builds engagement and engagement builds fast delivery.

Prioritizing data quality issues is vital for driving operational efficiency. Drawing from the DAMA DMBOK, consider business impact to rank issues. Focus on those affecting critical processes, KPI's and compliance for the best results. This approach ensures that your resources are directed where they matter most, aligning with industry best practices. Additionally, I would also recommend identifying and tier critical data elements that underpin key processes / KPI's. Focus on those affecting these critical elements and compliance as a priority to demonstrate value proposition for your investment in data governance or data quality as a practice within your organization.

Before applying data quality on tables, prioritize the tables by the impact. List your tables, apply numeric priorities, and begin with analyzing the 1st priority tables. If your data quality score is sufficient, move to the next priority level. Typically, your most important fact tables are the 1st priority. The dimension tables and lesser-used fact tables are your 2nd priority. Reserve the 3rd priority for other tables. You can use the 4th priority for all other tables. Usually, just four priority levels are enough to cleanse data incrementally.

Based on your comprehensive data quality assessment, control, monitoring, and communication strategies, it is essential to focus on the most critical data quality issues. Using root cause analysis can help to understand underlying issues, enabling more effective solutions. It is essential to develop effective data quality plans and implement targeted improvement projects. It is also beneficial to continuously update and adhere to the best practices and standards of data quality, ensuring that the organization's data assets remain robust and reliable.

One of the critical factors in DQ improvement is prioritization of the CDEs. We should always classify the CDEs (Critical Data Elements) according to their impact on daily business users. - There's very high chances that the DQ has to be improved across multiple domains, across business units. We should try to prioritize the most critical ones before moving on to the next. - Starting with multiple CDEs might lead to fatigue since the benefits of seeing the actual data quality improvement might be delayed if multiple items are considered together

I cannot stress the continuous review of data quality enough. Data quality is never a "one and done" process, old data having been given a once-over (even if it was looked at by top-of-the-line systems) is not good enough for eternity. New data points and new factors may give new context to old data points and factors.

Continuous adaptation and evolution are at the heart of any successful data quality strategy. As your business and its ecosystem evolve, so must your approach to data quality. The key is to foster a culture of learning and improvement, where data quality reviews become opportunities for growth, and to drive data maturity. Think of data quality as a journey rather than a destination. It's not just about the numbers; it's about the insights, decisions, and impact those numbers have on your business. So, organization should continue to review, adapt, and keep data quality journey alive and thriving to be more impactful and relevant.

I strongly support the need for periodic data quality reviews! It's crucial to: ? Evaluate and adapt: Regularly evaluate and adjust your data quality strategy to meet the evolving business needs. ? Feedback and Improvement: Provide feedback and lessons learned for continuous improvement. ? Stay Agile: Adapted to changes in technology, regulations, and business environments. Regular updates keep data quality relevant and valuable

Only focus on the data that relates back to actual business value. Fix the bare minimum that is required to help unlock or improve that value. By doing this incrementally you can increase overall data health and trust whilst at the same time delivering value to business. Try and fix it at source and optimize the process that creates or curates the data by embedding data quality by design. Too often we try to go big bang and fix everything, the challenge is you will always be constrained by resources, platforms etc. Perhaps something as simple as a bit of user training and awareness will already increase the quality of the data being captured in some areas.

Apply the LLC method of prioritization - Leverage What You Have: Maximize the use of your existing resources and technologies. Sometimes, it's about making the most of what you already have. Lean on Automation: Automation tools can be your best friend. They help you do more with less. Automate data validation, cleansing, and monitoring processes. Clean as You Go: Implement practices that ensure data cleanliness from the start. Don't wait for issues to pile up. Prevent bad data from entering your systems in the first place.

It is hard to prioritise data quality in the case when resources limited. The best way is to develop a holistic view, identify very specific business area/outcome based on the business requirements that may be impacted due to bad data quality, and start working there. This will be accompanied with a business case also.

1. Streamlining Data Quality Checks: - The automation of data quality controls significantly minimizes the need for manual intervention, enhancing efficiency. 2. Curating a Centralized Data Catalog: - Maintaining a comprehensive catalog of available data assets promotes reusability and prevents redundant copies, ultimately reducing the effort required for data management. 3. Defining Ownership and Appointing Data Stewards: - Above all, clear data ownership and the appointment of proficient data stewards are paramount. Without designated owners to address identified quality issues, the value of controls and checks remains unrealized.

To my experience getting the required recourses can be influenced by clearly presenting to your CFO what bad data quality may cause as well in company as to your customers. Make it visible, feelable, understandable in terms of costs and risks for your CFO. I often used the example of buying a brand new car, where the garage forgot to supply oil in the engine. Nothing seems wrong the moment the CFO leaves the garage, but after a few hundred meters, or even earlier the motor will have very serious damage. Tell him that providing high data quality is comparable to the function of that motoroil. You are responsible to make sure it’s present. Your department is the oil. The CFO does not notice it, but it’s there and the car will run smoothly.