How do you distinguish between statistical significance and practical relevance using p-values?

Although p-value (statistical significance) informs about "possibility" of an event occurring, if you consider a really rare effect by nature; it may become not that informative, even misleading. One should always take into account the number of observations and number of multiple tests; correct p-values based on those if necessary, and never assume a p-value<0.05 means "significant change". For some observations, p<0.1 could already inform about the "significance" whereas for some others even p<10e-5 might not be that important. This problem actually highlights the importance of being involved in the questions asked and assumptions premade for the specific test and statistical question.

As stated by the other contributors, the p-value is the probability of observing a value (use mean for simplicity) as extreme as the observed mean. The threshold of p<0.05 was instituted by Sir Ronald Fisher and used almost exclusively in regulatory submission for clinical trials. Other significance levels can also be useful, depending on the risk of error. What is not mentioned is that p-values can be seriously misleading if not connected to the desired effect size that should be linked to a meaningful difference. When done right, there is no difference between statistical and practical difference, because a p-value used to make a decision should be based on a study design that has practical significance at its very core.

Understanding p-values is essential for discerning statistical significance from practical relevance. A p-value indicates the likelihood of observing an effect as extreme as the one in your data if the null hypothesis—suggesting no effect—is true. Typically set at 0.05, a p-value below this threshold leads to rejecting the null hypothesis, indicating statistical significance. However, it's crucial to note that p-values solely measure probability, not effect size or importance. Hence, a statistically significant result may lack practical significance, emphasizing the importance of considering both statistical and practical implications in statistics.

You can think of a p-value as a goodness-of-fit statistic, where the model whose fit you are judging is the null.

Statistical significance alone doesn't guarantee real-world significance. A significant p-value may stem from a large sample size, amplifying even insignificant effects. Thus, while a small p-value indicates that an effect likely isn't due to chance, it doesn't convey its practical importance. To grasp the full implications of findings, both statistical and practical significance must be assessed in statistics.

Statistical significance is not the same as real world significance. For instance, we may run a linear regression and our main variable is statistically significance, but the R squared may equal to 10%. This shows that the model has failed, although the variable is statistically significant, because it fails to coreelate with the dependent variable. In the above example, a data transformation to a non - linear model, or the addition of more variables may improve the results.

Understanding and interpreting the p-value is key for running statistical analyses and using the outcomes and outputs. I constantly see that authors are reporting p=0.000 which is a rounding error of stats programs. It correctly means p<.001, which must be reported like that. I hope more authors will accordingly correct their reports and such errors do not occur any more!

Practical relevance, unlike statistical significance, gauges the actual impact of findings in real-world contexts. While statistical significance hinges on p-values and sample size, practical relevance is subjective and context-dependent. For instance, a minor enhancement in a medical treatment might be statistically significant but lacks practical relevance if it doesn't notably enhance patient outcomes. It's essential to evaluate the practical implications of findings to ensure they extend beyond statistical metrics in statistics.

The major problem in case of academic research is that many of these researches have little practical or economic implications/significance. Statistics works on number, it's possible that based on your data you get results that holds statistical significance but the real question is are the result relevant in real world? What is their significance and impact in real world scenario? These two are totally different ideas. A good research should not only be proven statistically significant but also should hold practical implications. Interpretation of results to understand it's practical implications is the responsibility of researcher.

Good statistical practice, as an essential component of good scientific practice. Statistical significance is not equivalent to scientific, human, or economic significance. Smaller p-values do not necessarily imply the presence of larger or more important effects, and larger p-values do not imply a lack of importance or even lack of effect. Any effect, no matter how tiny, can produce a small p-value if the sample size or measurement precision is high enough. Ronald L. Wasserstein & Nicole A. Lazar (2016)

Practical relevance hinges directly on the practical significance of a deviation from a baseline and indirectly on the baseline itself.

p-harking is a big problem, and providing effect sizes with practical interpretation is a must. Accordingly, journals asking for a "practical relevance" section or sth comparable can really help to not only run excellent analyses and generate important results but also contribute to enhancing our understanding of the world and what needs to be done to improve it.

Sample size is a fundamental issue when we are dealing with statistical problems. Usually if we get a small sample there are two possible sources of problems. Firstly, a small size does not always provide enough data to make a reliable analysis, and secondly, the Data provided may be for instance very large or very small in regards to the population, so the conclusions at which we may arrive may not be accurate. For instance, if you have a class and you take only the boys playing in the basketball team, measuring their height and weight, would this sample represent the whole class? It would be great to compare it with basketball teams of other classes, but not for students in general.

Another way to put it: large sample sizes allow you to pick up fainter signals. But whether these signals are useful will depend on the context.

Two free online two-page articles deal with aspects of context in statistical hypothesis testing. Search "ASTM statistical testing in context" to find "Statistical Testing in Context: How Analysis Provides Perspective on Results and Data". Also search "ASTM 2016 Data Significance" to find "Data Significance: Understanding Statistical and Practical Significance". These are from past issues of ASTM Datapoints a bimonthly column dealing with statistical applications and/or ASTM statistical standards. Many additional useful perspectives for common statistical subjects from multiple authors within the standards communities are provided. Search "ASTM DataPoints".

Researchers should recognize that a p-value without context or other evidence provides limited information. For example, a p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis. Likewise, a relatively large p-value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data. For these reasons, data analysis should not end with the calculation of a p-value when other approaches are appropriate and feasible. Ronald L. Wasserstein & Nicole A. Lazar (2016)

Dissemination and practical implications need sound statistical analyses and reliable results. So do not forget to translate your findings to different stakeholders and make use of co-creation/co-design/participation. If you need more ideas and guidance with that, pls check out the resources we provide with our Health Cascade project.

Sorry, the question could have been worded better. You know the practical relevance of your estimations when you see the assumptions of your respective model(s). So the relevance comes prior to the estimations, it precedes them. It is another matter that the estimations can prompt revisions of our models - yes, they can. Let me quote a comic relief from a well-known Econometrics textbook: "6. Thou shalt look long and hard at thy results. Corollary: Thou shalt apply the laugh test".

I would say that when interpreting p-values, it is important to consider both statistical significance and practical relevance. A statistically significant result may be practically relevant if the effect size is large and has important practical implications. Conversely, a statistically non-significant result may still be practically relevant if the effect size is large and has important practical implications.