What are some of the most common data analysis challenges or pitfalls that researchers face?

In business, people often focus on the data they want instead of the decision they are trying to inform. While there is often excitement (and pressure) to get to the answers you shouldn't rush through definition phase. You want to collect data for a clear reason! I spend a lot of time clarifying the question and probing into why past results aren’t ‘sufficient’.

Tunnel vision & getting tied to a narrative - in my experience, one of the most common pitfalls for any research endeavour, be it setting up your hypothesis, analysing your data, or sharing your future recommendations, is tunnel vision. We are all subject to confirmation bias, so try and approach your work by letting the data show you the way, rather than being dragged around by your narrative, the results you want to see or are expected to output. Stay true to the scientific method as much as possible. Peer review is a great way to combat this, as is looking at your research and your data through different perspectives (upside down, back-to-front, or even using inverted logic to try and prove the opposite hypothesis to your original idea).

From the product lens, understanding the importance of data analysis in research and recognizing its business value can be challenging. Many stakeholders have personal hunches and narratives that may not match your own. Defining and understanding the value of a clear and specific research question before collecting and analyzing data is the most important stage of data collection and is the focus of any up and coming product leader. Using your knowledge as a data expert and business leader is often the hardest part of being in product as a result. Building a solid data collection plan will make your life easier. Building relationships and inserting your guidance based on these principles will lead to success for you and your stakeholders.

Some challenges in quantitative data collection, include the lack of skills and knowledge about the methods, political crisis, pandemics or epidemics, and limited funding, but a major challenge in data collection could be the researchers own beliefs and values known as axiology (Guba & Lincoln, 1994). This can result in poor relationship gatekeepers, assumed (confirmation bias) knowledge about the setting, population and characteristics of the population. I mitigated this challenge in my Doctorate, spent 14 months for data collection (observation, informal conversations, interviews), but the first three months was my integration into the place, people and processes, including gatekeepers and 250 people with mental health challenges.

In my field of work where we frequently want to understand causality (with research questions like "does x cause y" or "how much y does x cause?"), a critical early decision to make in data analysis projects is whether to run experiments to actively collect data to answer the question, or use observational methods that rely on data that already exists. We usually consider experiments to be the gold standard, but it's often difficult/impossible to design an experiment for the precise cause-effect relationship we want to understand; in that case, observational methods can be better.

Too frequently investigators proceed to analyze their data with a pre-chosen plan before the data have been inspected for conformation to the assumptions of the plan. At best this produces uninterpretable results and at worst invalid results which compromise the reliability and generalizability of the inappropriate analysis.

In my experience, most researchers studying internet forums have published incomplete stories by relying on a single method. Reading posts on social media, for instance, is not enough on its own. There is a lot behind each post. We should remember that while the people online are a representation of real people, their conduct online and offline sometimes differ. I have advised many researchers to always sample a few comments and identify and contact the owners to ask further questions and compare them with their initial findings. It really helps especially when studying online data for a community you don’t know. Background to issues affects how people react to topical issues. I found this strategy useful during my PhD.

For high data quality, we always ask open end verbatim for key questions that explains "why" for the "what" (rating, response selection). Geo-tagging, Straight lining of quant questions can be detected using tools, but verbatim analysis provides quality index of customer responses.

This is especially true when data is provided by third parties, who often don’t fully understand what they are providing. It’s necessary to carry out a full data validation/acceptance analysis to make sure that the data is suitable for the proposed work.

A detailed analysis is imperative. However, that does not need to be presented. The data needs to be read completed but only the story should be told. That is when the key message gets communicated, interests the audience and opens a dialogue. Having too much data in the presentations is distracting and not value adding!

In a phrase: research informatics. Presentations never capture all data, thus having a system in place where data can be easily stored, queried, filtered and compared is critical to research operation scalability, knowledge retention, and agile decision-making. Such processes are so often a second-thought exercise, and happen after research operations have far out-scaled effective communication in loose file formats like Excel and PowerPoint. Building foundations for data organization, management, and retrieval early pays untold dividends as programs accelerate and advance toward the clinic.

When communicating results, context matters. Consider your audience and convey your results accordingly. Avoid industry jargon and use simple language if your audience is the general public.

The most effective graphs and charts communicate your results in a clear and simple way. Complex infographics add visual interest to a presentation but may need a text caption for your audience to fully appreciate and understand.

Take time to craft your figures and graphs and choose the most appropriate and visually appealing visual. Your graphs are an integral part of the overall story and having a great visual will help your story stick.

Realize that bias can come from anywhere, especially from what questions are being searched for and what data is used or not used. Always try to disprove what you believe.

Having a diverse team is another important factor to help eliminate analysis bias. If you’re working alone, then have others review your results and provide feedback on potential bias before publishing.