Data analysis is a crucial skill for researchers, but it also comes with many challenges and pitfalls that can affect the quality and validity of your results. In this article, we will discuss some of the common issues that data analysts face and how you can avoid or overcome them.
One of the first challenges of data analysis is ensuring that your data is accurate, reliable, and relevant for your research question. Poor data quality can lead to biased, misleading, or erroneous conclusions. To avoid this, you should follow some best practices for data collection, such as defining clear and consistent criteria, using valid and reliable instruments, and checking for errors and outliers. You should also document your data sources, methods, and assumptions, and perform data cleaning and validation before analysis.
Three questions come before discussions of data, quality, or analysis methods: “What is the research question?“ “What are the hypotheses?” “What data will validate or negate each hypothesis?“ We’ve become very powerful with our growing mountains of data and our clever means of processing, but complex data analysis risks suffering from the same ivory-tower distance from reality as theoretical physics. Don’t get me wrong, I’m glad we have theoretical physics—because the published abstracted ideas trickle into math and engineering. But it’s hard to see how abstracted corporate data analysis will ever get translated down to reality. More likely the abstracted work will just get shelved when someone thinks of something useful to do.
I agree, but when defining "clear and consistent criteria" be weary of sample size and adversely increasing Type 1 errors - and incorrectly reject the null.
Poor Data Quality. Bias and Sampling Issues Overfitting and Overgeneralization Lack of Context Data Privacy and Security. Interpretation Bias. Lack of Communication and Documentation. Lack of Domain Expertise. By being aware of these challenges and employing best practices, such as thorough data cleaning, unbiased analysis, context-aware interpretation, and effective communication, you can mitigate the pitfalls and improve the quality and reliability of your data analysis.
"Bad data" or data from data camps! This could look like "asjfbjw" as an answer for a short question, or an answer that appears to have no thought behind it. One way to get around this, if collecting data via survey, is to ask a terminology-based question. For example, showing a picture of an eggplant and asking the participant what it is called. If the study calls for participants based in the U.S, then the short answers may involve "eggplant" rather than "bringle".
It was mentioned to "document" the methods for data collection, but this is rather vague & can easily be overlooked & underestimated. I would suggest taking the time to make sure all methods are not only well known, but also practiced by anyone involved to ensure they're executed properly & consistently. Depending on what's being collected & how, an inexperienced individual can show incompetence when collecting data causing discrepancies among all of the final data collected.
Another challenge of data analysis is dealing with the complexity and diversity of data sources, formats, and types. Data can come from various sources, such as surveys, interviews, experiments, or online platforms, and each one may have different characteristics, such as structured or unstructured, quantitative or qualitative, or static or dynamic. To cope with this, you should use appropriate tools and techniques for data integration, transformation, and visualization, such as databases, software, or libraries. You should also be aware of the strengths and limitations of each data type and format, and choose the ones that best suit your research purpose and question.
In cybersecurity, it’s tough to really validate the source of data or information due to collaborative platforms like GitHub or just the open sharing nature of OS software and libraries. GitHub has been helpful in helping us crack some analysis but do we know who’s behind the usernames and if they are a reputable source or not? No. But the data obviously proved useful. Not a huge issue now but in the future it could be…
In addition to choosing the best format for your research purpose, one should also bear in mind what is most practical for those in that field. For instance, in exercise, body fat % is BEST recorded using a large tank for hydrostatic weighing. While this will provide the best quality data, it is very unlikely anyone in a practical exercise environment will have access to such a tank and as a result, will not be able to compare the results to their capabilities. Keeping things practical when selecting the format of data can sometimes be very useful.
A third challenge of data analysis is selecting and applying the right data analysis methods for your research question and data. Data analysis methods can range from simple descriptive statistics to complex inferential statistics, machine learning, or network analysis, and each one has its own assumptions, requirements, and interpretations. To avoid confusion or misuse, you should have a clear understanding of the logic, principles, and applications of each method, and consult relevant literature, guides, or experts for guidance. You should also test and validate your methods, and report your results with transparency and honesty.
There are three rules that a good analyst considers when picking data analysis methods: 1) Think of what the stakeholders actually need: Communication is an underrated skill in analytics. Speak with the people who will be receiving your deliverables. Ask them questions and make suggestions. 2) Think about your process and end result: Figure out how a product will look. This will allow you to create steps for your task. A common workflow includes extracting data, transforming that data for your use case, loading the data, and representing that data in a unique way. 3) Keep things simple: Do not over complicate the process. If you can solve a problem with a Simple Linear Regression, why would you use a more time consuming method?
A fourth challenge of data analysis is interpreting and communicating your results in a meaningful and relevant way. Data interpretation involves making sense of your findings, identifying patterns, trends, or relationships, and drawing conclusions or implications for your research question. To do this effectively, you should use appropriate visual aids, such as graphs, charts, or tables, and use clear and concise language, avoiding jargon or technical terms. You should also acknowledge the limitations, uncertainties, or biases of your data and analysis, and avoid overgeneralizing or exaggerating your claims.
One great & simple tip to follow when presenting your data is always ask yourself the question: "So What?" This is what your audience will be asking if you haven't interpreted your data well. This question will challenge the practicality of your data as well as the degree of its impact.
A fifth challenge of data analysis is ensuring that your data and analysis are ethical and respectful of the rights and interests of the data subjects, stakeholders, and users. Data ethics involves following the principles of consent, confidentiality, privacy, security, and integrity, and complying with the relevant laws, regulations, and standards. To achieve this, you should obtain informed consent from your data subjects, protect their identity and information, and use secure and encrypted platforms for data storage and transfer. You should also avoid manipulating, fabricating, or plagiarizing data or results, and give proper credit and citation to your data sources and collaborators.
To expand on this - analysts should consider blending the three main approaches to ethics: virtue (try to do the right thing), consequentialism (judge how you did), and deontological (build processes based on results). I give a lecture on ethics in prescriptive analytics at universities and companies. Here's a copy from WPI during the pandemic times. https://youtu.be/K179woxSr2M
A sixth challenge of data analysis is developing and maintaining the skills and competencies required for data analysis. Data analysis is a dynamic and evolving field that requires constant learning and updating of knowledge and skills. To keep up with this, you should seek opportunities for training, education, or certification in data analysis, such as courses, workshops, webinars, or online platforms. You should also network and collaborate with other data analysts, researchers, or professionals, and join communities or forums where you can share your experiences, insights, or feedback.
Data doesn’t exist in a theoretical vacuum. Data is not independent of theory. What is worth researching and cataloging as data depends on a valuation criteria - an entirely subjective theory. Ask a communist and a libertarian what are the key data in the context of government and they will give to you sharply contrasting data sets. Add to that a semiotic lawyer - imagine one of the two is French speaking and thinking, and the other is German. Our language impacts our understanding of the world. Data simply isn’t neutral as we will love to believe.