How can you determine if experiment results are statistically significant?

In HEP, a result that has a statistical significance of five sigma means the almost certain likelihood that a bump in the data is caused by a new phenomenon, rather than a statistical fluctuation. Scientists calculate this by measuring the signal against the expected fluctuations in the background noise across the whole range.?For some results, whose anomalies could lie in either direction above or below the expected value, a significance of five sigma is the 0.00006% chance the data is fluctuation. For other results, like the Higgs boson discovery, a five-sigma significance is the 0.00003% likelihood of a statistical fluctuation, as scientists look for data that exceeds the five-sigma value on?one half of the normal distribution graph.

Keep in mind that different thresdholds apply to different circumstances. With a lot of repeated statistical tests, you might want to set your thresholds much lower. With 100 statistical tests, on average 5 will be "positive" due to sheer chance. On the other hand, in a lot of business alpha of 0.1 or even 0.2 is frequently used - especially in marketing. The reason for that is that in business (compared to academia), we tend to focus more on multi-method approaches with larger volumes of evidence. If historically we keep getting alpha of 0.1 or 0.2 across many research projects with large effect-sizes, we can often confirm our hypothesis.

Statistical significance is determined by the p-value, which measures the probability that the observed results occurred by chance. If the p-value is less than the threshold (commonly 0.05), results are considered statistically significant, suggesting the observed effect is likely not due to chance. This determination often involves hypothesis testing and confidence intervals. For complex experiments, statistical software or tests like t-tests, ANOVAs, or chi-square tests are used.

Note that statistical significance describes the chance of the effect happening AT RANDOM due to sampling - that's it. It also describes this chance for a test, not for an entire experiment. If you're performing 20 different tests on a dataset, it's likely that one of them is a false-positive. Similarily, if you keep observing an phenomena that does not have perfect statistical significance (i.e. p = 0.15) but keeps appearing across many tests, its likely that the effect is a reality in your population.

Before touching any technology, make sure you have a great grasp of fundamental stastics. Do run ANOVA or regression or correlation tests with advanced software. First, do them by hand, on a piece of paper (or in excel!). This will let you understand WHY you're seeing certain numbers and what influences them.