How can you apply hypothesis testing in OR decision-making?

Hypotheses are of great importance when choosing a strategy. It is important to have a complete understanding of the problem from a business point of view in order to determine in which part of the space the optimal solution is located.

In my experience, one key aspect that I have found helpful in defining the problem for hypothesis testing in Operations Research (OR) decision-making is to ensure a thorough understanding of the underlying business context. Often, the true challenge lies not only in identifying variables and parameters but also in grasping the intricacies of the operational landscape.

One must ask 'why' and 'what till do not have further why/what. List what you already know and write or define in words. The discuss the details and define important areas to include and construct an agreed problem statement. Keep the discussed points and areas as info.

Crafting hypotheses is not a mere formality; it's a strategic move in the chess game of decision-making. It necessitates a balance between statistical rigor and practical relevance. As such, my approach involves continuous refinement in collaboration with subject matter experts, ensuring that hypotheses resonate not just with statistical significance but with operational significance. This synergy between statistical acumen and domain expertise lays a robust foundation for the subsequent stages of hypothesis testing, enhancing the likelihood of uncovering actionable insights.

Have defined subjects with probable or predicted outcomes, list out different variables. The outcome should be binary i.e. should conclude whether the hypothesis is true or not and must be reproducible. Also keep it in mind the null hypothesis.

The quality of the hypothesis is determined by special tests on different datasets of input data. It is necessary to conduct a sufficient number of tests for most possible situations to determine which of the hypotheses has the best numerical indicators: time and quality of the solution.

I would add that whatever statistical test is chosen should be clearly pre-specified in advance of any data collection. For clinical trials we often do this in a data analysis plan but a short document outlining the endpoint being measured, timing of measurement, expected effect size, desired statistical power and resulting sample size should be prepared. This process step can help minimize “p-hacking” or “cherry picking” that often occurs when ad hoc statistical tests are applied to collected data after the fact.

In my journey through Operations Research (OR) scenarios, I've come to realize the critical importance of marrying hypothesis formulation with the appropriate statistical method. Choosing the correct statistical test is comparable to selecting the right tool from a diverse toolkit. It requires a deep understanding of the data's characteristics, its distribution, and the intricacies inherent in the formulated hypotheses. My experience has frequently highlighted the value of close collaboration with statisticians or data scientists at the outset of the process. Their insights into the subtleties of different tests prove invaluable, ensuring that the analytical approach aligns seamlessly with the intricacies of the given problem.

I would add that one should also look at the results within the context of pre-specified data analysis plan. Specifically, was the final test presented the one that was planned for? Was the collected data sample size adequate to test your hypothesis? How many total hypothesis tests were performed, in addition to what results are presented. If many hypothesis tests were performed was there any adjustment for multiplicity? Knowing the answers to these and other questions allows one put the results into context.

While test statistics and p-values offer quantitative measures, their significance transcends mere numerical thresholds. They act as guideposts, directing us toward a decision. However, it's crucial to pair these outcomes with contextual understanding. A low p-value might indicate statistical significance, but its practical significance hinges on the business context. Also, view hypothesis testing results as pieces of a larger puzzle. Instead of treating them in isolation, I contextualize them within the broader operational landscape. This involves scrutinizing not just the statistical significance but also considering the magnitude of the effect and its practical implications.