Coping with incomplete or conflicting data: How do you do it?

Handling incomplete or conflicting data is like piecing together a puzzle with missing pieces. My strategy starts with pinpointing where the data comes from and what it is. Think of it as detective work. Is it numbers or words? Is it firsthand info or something someone else gathered? I even consider whether it's current or if it's been chilling in the data fridge for a while.

One thing that i have learned is not to rely on single source of data rather make multiple data points and to corroborate with market intelligence available and then benchmark with companies in same segment and region. This has always helped me in giving directionally accurate insights/analysis.

I believe that another tip can be added, that is to highlight that perceiving that the available data / information is incomplete (if this is the case) is a very crucial step to reach the most accurate conclusion and decision in an efficient manner. in other words being deceived by the level of completeness of the available data shall lead to unfavorable consequences that would even reach the level of mal-trust in the person that had processed such type of data.

I believe we can also add the fact of whether you have all the required information to connect this data with your problem statement. Idea being that if you need A to do B, but you are getting C somewhat related to A, it still won't help even if it can give direction

CSU Chico came up with a great source evaluation model: the CRAAP test. (Currency, Relevance, Authority, Accuracy, Purpose) Not only does it give a good picture the quality and reliability of your sources, but it's a fun mnemonic - and if it's fun we're likely to remember it, and if we remember it we're more likely to use it.

I disagree with the flow here. The process needs to start with . Identifying the problem statement . Identifying what data is required to get insights on the problem statements . Understand what is available and how it can be used . Figure out what data point is missing and how to identify a proxy or alternative data source

An example of this is might be: Evaluating a software company's recent email campaign. Problem might be to measure impact on user engagement and conversions. Question you are trying to answer: Did the campaign boost engagement and conversions? Ultimate Business Goal: Enhance engagement and conversions via effective emails. Boundaries and Criteria: Analyze past month's data for registered users, focusing on open rates, clicks, site visits, and sign-ups. Variables: Campaign metrics, user segments, content types. This targeted approach ensures relevant insights and informed decision-making. Lastly, outline the analytical methods and techniques that will be employed to address the defined problem, question, or goal.

Think of it as figuring out the game plan. What problem are we tackling? What question needs an answer? Where are we aiming? It's like setting GPS coordinates. I draw lines around the analysis—defining the boundaries, audience, and goals. It's all about avoiding data overload and staying on track. Imagine you're on a treasure hunt. You're not chasing after every shiny thing you find; you're hunting for that hidden gem.

I agree with Anjali on the flow. Assuming the first step is clarifying the purpose & scope of analysis, there are methods to be effective & efficient. From my experience, focus on effectiveness first, then efficiency. Why? You can efficiently do the wrong thing. So, make sure you are effective first. This is why purpose is the most important. It defines WHY you are analyzing. THEN define the scope for efficiency to focus on WHAT & HOW.

I like to know my purpose of the project and the definition of the scope first before identifying the nature of the available data so that I can remain focused.

Agreed. Identifying the alternate source or proxy data can help with directional decision making. For ex: in case of a customer centric problem statement, a quick survey to get the pulse of a sample set of customer may help as a proxy for network wide customers

Often, we don't get the data the way we want, so additional or alternative data is the next-best solution. Although this is risky, it is less risky than relying on faulty or no data. The caution here is to be careful about alternative data. Make sure it is a reasonable proxy and not just something to fill a dataset without ensuring a plausible connection.

In addition to the pivotal points mentioned by other peers, if easilt available - Search academic literature, industry reports, archives for supplementary data. Given modern times, consider crowdsourcing or public engagement to obtain new datasets although this may be too optimistic/adventurous.

When dealing with missing data, consider imputation techniques to estimate or fill in the missing values. However, exercise caution and use statistical methods appropriate for your data type to minimize bias. If possible, cross-reference data from multiple sources. Divergent data from independent sources can help identify and resolve conflicts. This approach adds robustness to your analysis.

Many think about this as the analysis phase. From my perspective, this is a precursor to analysis, and I would add another phase after this called analysis. Evaluation & comparison is looking at the data elements and values to determine validity, internal consistency, and anomalies. Doing this prior to analysis enables a greater degree of confidence in the results of analysis. THEN, analyze the data for patterns, trends, relationships, or differences.

In my experience complex analysis becomes more difficult when biases are possible or likely. Consider the evaluation of a politically favored or maligned hi-tech startup. Proponents tout the benefits, protractors the nepotism. Where do you go? Out side your comfort zone. In such a case a scientific journal outlined per unit cost of production and implementation and the product’s impracticality. The business failed in due course and as expected. A political boondoggle.

Agreed. I find heat maps to be a pivotal source of identifying the key areas to be focused on. It helps with creating a custom solution basis the patterns identifying for a group of customers or a set of pre-defined locations.

I recommend to develop data quality metrics and criteria specific to your project. This will help you quantify the degree of incompleteness or conflict and prioritize which issues to address first. In addition, a data cleaning process that includes standardizing formats, resolving inconsistencies, and addressing missing values will also contribute. Seek input from subject matter experts or colleagues who may have insights into the data's context may help clarify discrepancies or suggest resolutions. Finally, performing a sensitivity analysis to understand how different interpretations or resolutions of the conflicts may impact your conclusions can help quantify uncertainty.

Be a listener first and speaker later. Communication is a crucial part of the analysis process. A candidate without this skill won’t able to deliver a solution that worth. Initial stage communication: The first step of a analyst is to understand the problem statement by communicating with stake holders. Work-in-progress communication: After getting clear on problem statement, analyst may require more discussion to enriching understanding of dataset/information collected. Presentation communication: Analyst requires to write or speak about presentation: what, why, where, when, and how. I would say communication is the oil/lubricant for the analysis engine. In the end, Communication must be two-way.

I agree with Mayur. Listen first, speak later. When working with incomplete or conflicting data, it helps to acknowledge the incompleteness or conflict. This allows the audience to trust the process you followed and trust YOU. It is also an opportunity for feedback, possibly leading to enhancing the analysis with follow-up. If you know your audience before you are presenting results, involve them in the process earlier. They may guide you to alternative sources, for example.

This i believe is the most difficult part because all the above work done should be presented/articulated in a way that it is understandable for the audience in the room. I have always found it helping when you take the audience through the slide/data first as to what is presented where and then start the articulation so that they know where to look for what, furthermore always be ready to take questions between the articulation and if it is related to the part you are on address it, if not, assure the questioner that this will be clarified in the next part.

Structure analysis into logical sections with executive summary, findings, recommendations. Acknowledge limitations and other considerations transparently. Tailor messaging and medium to resonate with target audience.

Clearly communicate the limitations and uncertainties associated with incomplete or conflicting data to stakeholders or decision-makers is crucial. Transparency is essential to manage expectations. While communicating you'll need to be prepared to iterate. Sometimes, resolving data issues is an ongoing process. As you gather more information or better understand the context, you can revisit and refine your analysis. Also, explore the option of using AI techniques for pattern recognition and predictive modelling, which can help identify trends or fill gaps in data. Consider to evaluate the potential risks and consequences of making decisions based on incomplete or conflicting data. Consider fallback plans or mitigation strategies.

I believe it is also important to tap into the collective wisdom of the expert network to ensure the conclusions are rigorous enough to stand up to scrutiny. Firstly, establish the expert network. These could be a group of individuals that are highly skilled in relevant fields or possess strong critical thinking to question the findings. Next, the findings will also benefit from being scrutinized by a diverse group so you can mitigate biases and the influence of herd thinking. New sources of data or new ways of analysis could also be unveiled during the process.

Conflicting data is normal - even ideal - for analysis. If all the data says the same thing or points in the same direction, there's not much need for analysis. The real value of analysis is sorting through and making sense of incomplete and conflicting data, not just compiling and repackaging information.

“One root does not make a tree” is a personal saying of mine. Curate from others’ perspectives; collaborate and observe the solutions trickle down.

Identifying the use cases basis the customers who will use the data is also an important step while generating your insights. It is necessary to know how to create multiple cuts of your data basis the stakeholder and their requirements.

I believe it is more alarming to have conflicting data than having incomplete data, the idea is conflicting data may pose a halt to major decision the organisation needs to take or make due to lack clarity of what the taken decision will affect. while incomplete data is considered a limitation rather than an issue. A starting point toward fixing or coping with this issue, is to identify where the issue is coming from or what kind of data is having this issue, then start on " cleansing " plan towards eliminating this issue, day by day until the level of confidence in the data is high and can be used to make decisions. Coping with incomplete or conflicting data: How do you do it