How can you design experiments that are valid for data science applications?

As I get never bored to tell my team : Context is the king in statistics and data science. This is also true for the subfield of statistical experiments. Though there are numerous approaches to define a statistical test driven problem, best way might be to ask first "how not to define this problem ?". For instance, an AB testing problem can be ruined from the beginning if it would be impossible to attain the necessary sample size etc.

The allure of diving straight into data is undeniably strong. However, even in these early stages of my career, experience has taught me that the bedrock of any meaningful data-driven project is a well-defined problem statement. It's like setting out on a voyage; no matter how advanced my ship is, without a clear destination, I'd be aimlessly adrift. Every time I've taken the time to meticulously define the problem, by performing hypothesis formation, variable selection, or metric determination – everything falls into place. In contrast, the few times I've bypassed this crucial step, I've found myself confused. The act of defining the problem effectively in Data Science is the most potent tool you can have.

Nature of some problems are so intuitive and self-explanatory regarding the choice of statistical test methods. Certain some might allow, or even force you to choose unorthodox methods. I advise everyone working with statistical testing to consider checking multiarm bandit and statistical experimentation design with them.

In an early project, I analyzed the effect of a new UI on our mobile app's engagement. Opting for a quasi-experimental design, the next step was data collection and preparation. I curated a user sample that mirrored our overall user base, balancing demographics and past interactions. Diving into the data, I encountered challenges. Missing values, inconsistencies in user activity logs... This developed in a rigorous data cleaning phase which made me realize the importance of meticulous data collection and preparation as well as ensuring that every data point is accurate, consistent, and truly reflective of the problem at hand. The quality of insights is directly proportional to the quality of the data feeding into the analysis.

After prepping your data, it's time to delve into analysis and interpretation. This means employing the right statistical tests—be it t-tests, ANOVA, or chi-square—while noting significance levels, effect sizes, and confidence intervals. Visualization tools, like graphs or charts, can spotlight key findings. Always anchor your interpretations to your initial problem statement and research query. Additionally, address the experiment's limitations and broader implications.

You would be surprised to know how many numerous statistical testing results I have read just include the basic independence and normality assumptions along with basic tests valid under these ideal circumstances. Though, the invalidity of these assumptions were starkly obvious given their context. Thus, statisticians need to invest a lot of time and effort to scrutinize distributions and the nature of the use case to design more sophisticated experiments.