What are the common misconceptions about statistical significance testing?

P value is the probability of something happening if null hypothesis is true. If p value is low it means null hypothesis is rejected

Common misconceptions about statistical significance testing include: * Confusing statistical significance with practical significance (a small effect can be statistically significant but not meaningful). * Believing that a non-significant result means no effect exists (it may just mean the study was underpowered). * Focusing solely on p-values and ignoring other important aspects like effect size and confidence intervals. * Misinterpreting p-values as the probability of the null hypothesis being true (it's actually the probability of observing the result by chance).

Statistical significance testing is widely used without a proper understanding of this concept. Many statistical tests have underlying assumptions (e.g., normality, homogeneity of variances) that, if violated, can lead to incorrect conclusions. It is important to check and, if necessary, address these assumptions.

Even if the p value is low, it still means there is a probability. However, the importance of that value depends on wich phenomena are we talking about. For some cases, occurence is more likely to happen because the related factors are determinated or predicted. For other cases, related factors are more random, and probability here will be just about random occurence.

Statistical significance can happen due to some other effects happening simultaneously. It is important to know conceptual/ business reason for some relationship to understand if statistical significance is truly meaningful.

I think that the statistical test should include factors and measurements that are close to the real complex of a phenomena. In other words, we should put values that make the result significantly close to the effect size. This way, we can be more sure that the significant result will give a close idea on effect size.

Significance tests are often used for arguments, although common sense is enough to see an effect. In my opinion, you don't need a test to determine that food fills you up. For me personally, you don't even need statistical tests anymore to see if the climate is warming. That was perhaps really necessary in the nineties. At the moment, unfortunately, it is enough to go outside and look at the dying forests.

The common misconception about significance threshold is that there is some set benchmark p-value. In reality what should be the p-value of your study depends on the type of study. If cost of rejecting a true hypothesis is high then p-value will be low. E.g in a pharmaceutical research our null hypothesis will be that a new vaccine is not safe for human trials. Let's say the company claims that the vaccine is safe for human and can be introduced in market. By doing this the company is rejecting the null hypothesis. Here the cost of rejecting a true hypothesis ( Type 1 error) is very high (cost of human life, bad company image) hence the company should be extra careful while rejecting a true null hypothesis and so will choose a low p-value.

I have always worked with relatively large samples of data and on every occasion, statistics were present for me to test and confirm some hypotheses. For instance, once I needed to test whether all my clients who traded a specific stock on the Ibovespa had more than x reais in their bank account. The significance level used was 0.01, as I wanted the most accurate representation possible of what I could consider for the sample used

A common misconception about statistical significance testing is misinterpreting p-values as probabilities of the null hypothesis being true. In reality, p-values indicate the probability of observing the data or more extreme data, assuming the null hypothesis is true. This misunderstanding can lead to erroneous conclusions and misunderstandings about research findings. Correct interpretation of p-values as measures of evidence against the null hypothesis is essential for accurate statistical inference. Emphasizing the distinction between p-values and hypothesis probabilities is crucial for promoting rigorous statistical analysis.

Read about Type I and Type II errors. When considering increasing or decreasing your significance threshold, also take into account how it will affect the power of your test. Your interpretation of the results should not only focus on whether or not you achieved statistical significance, but also whether the results come from a setting with low or high power.

Higher p value means there is higher probability of something happening if null hypothesis is true. However, it may not always mean that there is no impact, as there could be other factors at play such as size of population, other factors which are not controllable and measurable. Also, you maybe ignoring some factors like transaction cost in trading for example

It might also mean that there are sources of variation that aren’t controlled. Perhaps a more efficient experimental design is required (blocking structures, covariates, etc.)

Relying solely on statistical significance overlooks crucial aspects of data analysis. Effect size quantifies the magnitude of an effect, while confidence intervals provide a range for the true effect. Both metrics offer valuable insights into practical implications and should accompany p-values in result interpretation. Statistical significance is just one facet; integrating it with other measures like effect size and confidence intervals provides a comprehensive understanding of research findings.

P value by its own is not enough. For example in regression, we need to look at many other statistical tests such as adjusted R square, stationarity. Autocorrelation etc. Moreover, business intuition of the relationship is also needed

Another problem is that so-called p-hacking. You do a study and then retrospectively adjust the value to confirm your own desired statement or prejudices. This is methodologically incorrect. You have to agree on a threshold before the experiment and cannot "fake" it afterwards

Relying solely on statistical significance for decision-making can be deceptive. While a significant p-value may seem like a clear signal to act, it overlooks critical factors such as data context, biases, and measurement reliability. Decisions must consider a holistic view of evidence, incorporating statistical significance, practical significance, and broader research context. A nuanced approach that balances these factors leads to more informed and effective decision-making.

Understand type 1 and type 2 errors. Understand the distribution of the data. Understand the assumptions under which p value test are valid. Understand the maths behind the formula such as t test

If you test many hypotheses, consider using a correction for multiple comparisons (e.g., Holm-Bonferroni which is more powerful than Bonferroni; in a linear regression, this might be a joint F-test). If you have a lot of hypotheses to test, it might be wise to control the false discovery rate (FDR) using, for example, the Benjamini-Hochberg procedure.

# Example illustrating the importance of considering practical significance alongside statistical significance if p_value < 0.05: print("Statistically significant result detected!") if effect_size > 0.5: print("Practically significant effect size observed!") else: print("Effect size may not be practically significant.") else: print("No statistically significant result detected.") A holistic approach, considering statistical, practical significance, and broader context, enhances decision-making beyond statistical significance alone. This code snippet illustrates integrating both statistical and practical significance for comprehensive analysis and informed decision-making.