What are the most effective ways to communicate data quality issues in BI data cleaning?

The first and foremost action for any Data Analyst is to check the Data Accuracy because 1 incorrect data examples and take off the trust of the stakeholders. Redoing the Data project, Analysis or Dashboard is easier but developing that Trust again is always challenging - thoughts completely based on my experience working for last 17 years So a few things to be done in strict timelines - Check data and clearly state that you data is XX% accurate (so you know monthly revenue is 1M, but data you have is 990K, so mention this is 99%) - Don't work if data is <90% accurate or take stakeholder's approval - Clearly mention if you know that say City, Product names etc aren't clean but rest data is ok

Data Quality (DQ) is a founding principle for consistent and reliable BI, rather than a one time project. Implement a structured, holistic approach for best results -: 1/Identify the most critical BI elements for the business (vs. boiling the ocean). 2/Define rules, metrics and standards to measure DQ e.g. completeness, accuracy, validity etc.. 3/Assess DQ using scorecards and monitor trends in defect rates and materiality across time periods to highlight improvements or deterioration to leadership. 4/Remediate DQ issues in order of magnitude of materiality and impact to the business, fixing the root cause of issues systematically (rather than the symptoms). 5/Sustain DQ over time by continuously iterating through steps 1 to 4.

Data Integrity checks is an important task to start with while processing raw data for reports and dashboards. Some key metrics are to find outliers, NULLs/blanks and duplicate records 1) Defining incorrect/default end dates(Year 2100) for Sale contracts would impact forecasting accuracy of future revenues.Conversion rate/Human error sometimes causes inflated numbers. Outlier detection techniques help identify these records to rectify any possible skewed analysis. 2) Larger volume data ruins user experience due to high load times. Neglecting Null/Blank/Zero valued records or Aggregation techniques are some of the best practices. Any data omitted needs to be communicated to client/stakeholders when it’s of significant volume.

In my career I have seen people run a report once and they throw it on a PowerPoint and interpret their findings through cognitive bias. If you have new data let it settle before you make it gospel. Defining what you need is good but sometimes the data will teach you.

Firstly you need to agree what makes data valid. There will be two main types Publicly referenced (ISO countries, addresses etc.) and internally referenced e.g. the steps in the business process, receive order, accept order, provision order etc. These processes may not align in all sytems. Profiling it and working through the results with the data owners is a good start. For reference data and lookups define the acceptable values e.g. list of countries / currencies Then work on the less obvious reference data used in the business processes. Take it step by step and just keep pushing for great data. Once all the data conforms to agreed reference values then you can start working out if the values are correct. How many orders a month?

Especially while working as a Consultant and delivering your services to 3rd party clients, use the below methodology - Share the monthly Revenue and Orders that you have access to - Provide details of Null, Min, Max, Mode, Median of all Numeric fields - Provide Null, Unique values, % Data available of all Alphanumeric fields - Similarly ask for Grouping of data for either of the fields you deem fit - Clearly list of the data variables that are completely blank or have just 1/2 values - Highlight all fields that are complex to segregate (like Complete Address) Document all these things, share with the client and take the sign off before you start working on any analysis

I find using Jira or similar and treating data quality issues as bugs to be a good way to document them. Then alter the ETL/transforms to remove the issues. Write test cases etc. Why not treat it as development? Report on progress from Jira/DevOps. Ask your customers (if able) to access and update the issues. You can extract Jira etc data and use as final documentation of data requirements. You have the user story, developed features and the final testing. The data quality report from JIRA becomes your shared status document. If the customer is not happy using Jira then the PM/BA can update it with email exchanges.

Purpose of resolving data quality issues could be multifold, improving the accuracy of analytics value chain (Diagnostic, Descriptive, Predictive,Prescriptive) and also Improving process, Driving organizational behavior, Adoption of tools and processes etc. While data cleaning is important, it's also important to understand the qualitative insight and feedback for such data gaps. Those when looked at from a summarized level will help uncover many trends, gaps, problems, improvement scopes that are highly valuable.

Generate data quality reports that highlight key metrics related to data quality, such as missing values, outliers, and inconsistencies. These reports can be regularly shared with stakeholders to keep them informed about the state of the data.

The best approach to get the attention of the Stakeholders on the Data related issues is to present them rather than showing only values eg: - Show a heatmap of Pincodes/ cities/ states and ask if you find high concentration in just 2-3 geos - Show bar chart for all Delivery type options if you think there should be 5-7 but actual data has 25+ field - Show a trend line for Revenue, orders, customers to show monthly business trend and highlight any outlier - Use AI to highlight an Outlier automatically and use colors like "Red" so that attracts the attention

Where you don't have an agreed automated strategy to correct data then you need to make the issue visible. As always remember your audience. The department head wants 80% of 20,000 records automatically fixed as agreed. They then know 4000 need either manual touching or more automated development. The data owner needs to know roughly why each group of records is wrong so they can decide what are the next steps The Data steward needs to know the breakdown of the issues so they can touch each one or suggest a strategy. The board need an estimate on what the bad data is costing and why. All need to know how we are progressing so they can be confident of our efforts.

This is the most crtical element of the data cleaning exercise. It's not the data department that can silo itself to clean the data, you need extensive support from your SCM teams to coordinate with brands and vendors to clean the data. Educate your senior leadership on data quality needs and take their support. The KPI that you communicate will not only ensure data is being enriched, it will put a pressure on SCM teams to ensure they provide the right collaboration in this task. Once again, break down the cleaning exercise in segments, if you are a retailer, by categories, by suppliers, and show KPI trends according.

Having an automated framework to measure non compliance, summarize, report and bringing visibility is extremely important. The most useful thing would be to have when we can show and bring visibility to the impact of such data quality issues in overall organizational goals. Actionability and ability to drill down to details and take corrective action is crucial. Finally simplicity and frugality is key. Less is always more.

Create data quality dashboards that provide a visual representation of data quality metrics. Dashboards make it easier for stakeholders to quickly grasp the overall data quality status and identify areas that may require attention.

"Don't Assume" Many a times Data Enthusiasts have a problem of assuming a lot of Data fields and thus impacting the quality of the data further. The best approach is to collaborate with the stakeholders or Data Quality/ Engineering people, spend considerable time with them and talk about all Data Quality issues - Ask if a particular field is even relevant to spend time and effort in cleaning it - Ask for all Business definitions and fields to be used to derive the metrics like whether Repeat rate should be at daily, weekly, monthly, quarterly or annual level - Confirm your assumptions if any - Ask if a field can be dropped because it has blanks or just 1-2 values - Re-confirm if some other data variables can be added initially

As earlier using Jira/DevOps as the basis and then creating reports that are available to the project team both the developers and the data owners etc. Using Jira to document tasks and assign responsibility allows people to focus on what is assigned and see how far we are on the journey. Having such clarity allows you to keep everyone updated.

Again I would say Empathy is key here. To be able to collaborate, first listen to your stakeholder. Understand the pain point. Oftentimes such open ended discussion could help come up with a better solution to help the overall business goal. Remember having 100% data fidelity may not move the needle if it does not solve the business purpose. Measuring, reporting, escalating the anomalies and exceptions are important but more importantly the ability to discover the underlying issues will help to come up with innovative data solutions that will help your stakeholders. Which eventually will to drive the overall big picture goal than just moving a metric from red to green.

Clearly communicate data quality issues in a transparent manner. Use plain language and avoid jargon to ensure that all stakeholders, including those without a technical background, can understand the issues.

If you start a data cleaning project, prepare a rule engine that CV an work for both, data within the system and new loads. All rules must be applied to the new data sets and strict adherence must be maintained irrespective of the pressure from business. A parallel exercise to clean historical data has to be managed. This ensures your gateway is protected before you start the cleaning. Else the exercise will never complete. Also, prioritize the data. If there is no stock, no sales or no moments for certain period of months, deactivate those data sets or deprioritize the data sets. Last but not the least, use MDM tools with strong workflows.

I cant stress more about the importance of simplicity and frugality. It's complex to build simple solutions. Define and build metrics that are not complicated math problems but simple to comprehend and internalize. Build visualization that helps to drive key take away and not the deep analysis that helped to arrive at the key take away.

As always - how the data would be used is most important to determine the quality attributes. What is fine for an estimation exercise is not fine for a management update for example. How many layers of questions the data needs to answer would determine the levels of data quality and audit checks to be acceptable

Craft "Data Detective" newsletters, sharing intriguing stories of resolved data mysteries and highlighting the importance of vigilant data cleaning practices. Develop a playful "Data Blooper Reel" showcasing amusing but instructive examples of data quality issues to engage and educate BI users. Implement a gamified "Clean Data Champion" program, recognizing and rewarding individuals who contribute significantly to data quality improvement efforts. Host interactive "Data Spa Days" where teams collaboratively address and cleanse specific data sets, fostering a communal approach to maintaining data integrity in BI.

I believe MDM Master Data Management will add value to any data. Second but not less important is the collaboration among different silos of application to transform data into information. Information is the new currency.