How can you use data cleaning to promote data literacy and empower stakeholders?

This data mashup can lead to a bigger project since we digger into the data. Hence, from the beginning of the project, we need to define the goal and scope. It is important to get things done rather than keeping iterating. In the business world, efficient communication with the stakeholders is critical for our success. We keep talking with the stakeholders to help clarify the request and define what they really need at this moment. The awareness of opportunities and risks has to be noticed, which will help us keep the goals and scopes in the mind always.

Three key ways in which data cleaning promotes data literacy and empowers stakeholders: Reliable Insights: Data cleaning ensures accurate data, cultivating confidence and reliability in decision-making. Enhanced Literacy: Stakeholders gain better understanding of data structures through hands-on data cleaning. Collaboration and Continuous Improvement: Data cleaning fosters collaboration between teams, encouraging communication and knowledge sharing among stakeholders. Establishing a feedback loop for stakeholders to contribute to data quality creates a culture of continuous improvement and active involvement in maintaining accurate information.

A cleaner data gives a clearer a picture of the data which contributes to time efficiency, resource efficiency and improved data quality. This means more meaningful insights for the stake holders.

Data preparation is a crucial and often undermined step in the data analytics process. Many times analysts do not realize the importance of data cleaning. Preparing data by cleaning contributes to 70% of the whole Exploratory Data Analysis process. If the underlying data is correct, the analysis yields accurate results, but. Erroneous may lead to errors in decision making and thereby incurring heavy losses for the business. It gives great power to the decision makers if analysts come up with business insights based upon correct and error free data which further helps in taking right decisions for the business.

By meticulously cleaning and preparing datasets, analysts enhance the overall quality and reliability of the information presented to stakeholders, facilitating a clearer understanding of the data. This process involves addressing missing values, correcting inconsistencies, and ensuring data accuracy. Transparently documenting the data cleaning steps also aids in demystifying the analytical process for stakeholders, fostering a greater appreciation for the reliability of the results. Additionally, providing clear explanations and visualizations of the impact of data cleaning on the final analysis helps stakeholders grasp the nuances of the data, fostering a more informed and data-literate decision-making environment.

Li um paper da Cornel University que chama Datasheets for Datasets que acho extremamente útil quando leio esta etapa 2. A comunidade de machine learning n?o possui, atualmente, um processo padronizado para documentar conjuntos de dados. Pensando em resolver essa lacuna, os pesquisadores trouxeram uma ‘ficha técnica’ para os conjuntos de dados, descrevendo suas características operacionais, resultados de testes, usos recomendados e outras informa??es - aqui vale a inclus?o de limita??es e problemas de qualidade dos dados. Se cada dataset tiver documentando sua motiva??o, composi??o, processo de coleta e usos recomendados, a comunica??o entre os criadores do algoritmo e os consumidores fica mais objetiva.

Depending on the scope/size of the project, I would highly recommend some form of a RAID log. Share this with your decision-makers so that they can make decisions and determine priority. Some data quality issues may be higher impact than others! Be smart with your time and work on the high value, low effort stuff first.

I would like to share a practical example from my experience dealing with data standardization. I encountered a situation when working with a database of quotes (spanning over 30 years) where different teams maintained their own format (no standard). For example, one had "Article" in its header (or "Designation"), columns appeared grouped, while they were single in another quote... I found you can separe columns based on different names, pull-out trailing spaces, symbols, convert to lowercase, match synonym columns.. via PowerBI with python, with the help of regular expressions ***. The key lesson I learnt was that even seemingly disparate data formats often have common patterns that can be found when you carefully analyze.

Data quality issues are any problems that affect the accuracy, completeness, consistency, or relevance of data. These issues can have a significant impact on businesses and organizations, leading to poor decision-making, wasted resources, and reputational damage. By taking a proactive approach to data quality, businesses and organizations can improve the accuracy, completeness, consistency, and relevance of their data, which can lead to better decision-making, increased efficiency, and reduced risk.

Documentation can be really helpful with some stakeholders in that it can help prevent misunderstandings and improve clarity when working with complex data structures

What I'm guilty of is partially cleaning data during my early parts of my career. When data sets go beyond hundreds of thousands of records, it is easier to just hope and pray it won't skew the results. Apparently, my poor habit haunted me and costed me a lot of back and forth with stakeholders. This was a lesson learned that data cleansing is in fact crucial for data analysis.

Testing the data processing process is important and is often ignored. Just as we design tests for other software to prevent errors, we need tests for data processing codes. Data can often receive unwanted biases in the processing and will still be validated by interested parties. It is also important that tests are designed before code development and if possible by other developers, to avoid any confirmation bias

As an entreprise architect, I strive to give managers the bigger picture, like the GIEC is advising country leaders on the global climate change. And I do this by showing a landscape of applications or functions colored for a given analysis. The missing or incorrect data are easily visible, like a population worldmap withtout North-Korea nor Zimbabwe. That way the local managers realise the impact of data quality onto the overview (previously only seen by the top managers). Not only the data mindset is promoted and the data gets cleansed, but often the root cause is corrected, which leads to a "better world".

A picture is worth a thousand words. the techniques that i usually follow for effective data storytelling can be summarized in 6 steps: 1- Define a Clear Purpose by determining the key message or insight you want to convey. 2- Identify Your Audience. 3- Craft a Compelling Narrative. structure your data story with a clear beginning, middle, and end. 4- Visualize Data Effectively by using charts, graphs, and other visuals to present data in an engaging and easy-to-understand manner. 5- Use Simple Language. 6- Practice your data story to ensure it flows smoothly and delivers the intended message.

Data Cleaning is the Catalyst for Stakeholder Empowerment and data literacy 1. Build Trust: data cleaning instills confidence, allowing stakeholders to rely on data-driven insights for informed decisions. 2. Simplify Complexity:cleaning educates stakeholders, enhancing data literacy by breaking down complex concepts. 3. Encourage Participation:Active stakeholder involvement in cleaning cultivates ownership, improving data literacy and fostering a shared understanding. 4. Showcase Impact:Link cleaning to tangible improvements, reinforcing the importance of data literacy and encouraging stakeholder investment. 5. Continuous Learning:Establishing a culture of continuous learning through sessions and newsletters.

Effective communication and education through data storytelling play a pivotal role in ensuring that stakeholders not only grasp the technicalities of data cleaning but also appreciate its impact on business outcomes.

A 10-step approach towards a clean data 1. Conduct the workshop with business to understand the current challenges and the expectation. 2. Understand and identify the sources 3. Filter out the scope and start small (Example: Customer master) 4. Identify the Critical data elements and sources for the Customer master. 5. Run profiling to identify the quality of the attributes across the source system. 6. Discuss the results with the business and document the business rules. 7. Create a Quality Dashboards 8. Run the rules and validate the Quality score. 9. Refine and scale. 10. Educate and Empower Business users for continuous monitoring & improvement.

Data cleaning enhances data literacy by ensuring accuracy and reliability. Stakeholders can confidently engage with trustworthy, well-documented data, fostering informed decision-making and a deeper understanding of insights.

Reading uncleaned data is like reading an unedited book. The errors can be distracting and will impact the your ability to tell a story with the data. This is why data hygiene is essential to communicate insights.