What does p-value really tell you in statistical testing?

Consider an example of coin toss, after tossing a coin 100 times you get 80 times head. what can you say about the coin, is it fair or not? Ho (null hypothesis becomes) : Coin is fair Ha (Alternate Hypothesis) : Coin is not fair suppose after using appropriate test, in this case we will use binomial dist. suppose you get P-value=0.02 ,this will mean that there are only 2% chance of happening which is very less and still the event had occurred ,so we will reject null hypothesis and conclude coin is not fair, here p-value is probability of happening of particular event given that Ho is true.

A p-value helps in deciding whether to reject the null hypothesis. If the p-value is less than or equal to a predetermined significance level (usually 0.05), it suggests that the observed data is unlikely to have occurred by random chance, and thus, one may reject the null hypothesis in favor of the alternative hypothesis.

In statistical testing, the p-value represents the probability of obtaining the observed results (or more extreme results) if the null hypothesis is true. A low p-value (typically less than the chosen significance level, such as 0.05) suggests that the observed data is unlikely to have occurred by chance under the null hypothesis, leading to the rejection of the null hypothesis in favor of the alternative hypothesis. The p-value does not directly tell you the magnitude or importance of the effect, but rather the likelihood of obtaining the observed results if the null hypothesis is true.

In statistical significance testing, the p-value is the probability of obtaining a test statistic result at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. We reject the null hypothesis when the p-value turns out to be less than a predetermined significance level, usually 0.05

We can reject the null hypothesis (H0) if the significance of Probability is below 0.05. We have to keep model H0, if p-value is above 0.05. But the core logic of hypothesis testing that my team maintains: 1. We state a hypothesis about a population, it is usually based on existing research, on observation, or through reasoning; 2. We collect data through a random sample from the population; 3. We compare the sample data with the prediction that was made from the hypothesis. If consistent, the hypothesis is reasonable; if not, the hypothesis is wrong; 4. We always consider the context and connect all dotes (test results, settings, meaningfulness).

Also the significance level such as 0.05 is not determined by the p-value, rather the significance level is decided by the person conducting the experiment. The p-value does not indicate the size or importance of the observed effect, the larger the effect, the smaller sample size will be required to get a significant p-value

The significance level must be fixed in the design phase of the trial/study/experiment. It’s the false positive risk. The risk depends on the consequences of making a type 1 error. In no way is 5% or 0.05 a norm.

The smaller the p-value, the larger the significance because it tells you that the hypothesis under consideration may not adequately explain the observation so your hypothesis is rejected if any of these probabilities is less than or equal to a small, fixed but arbitrarily pre-defined threshold value

Typically we look at 95% confidence intervals, and sometimes 99% confidence intervals, but they all have a similar interpretation: they are limits constructed such that for a certain percentage of the time (be that 95% or 99%) the true value of the population mean will fall within these limits. The basic idea behind confidence intervals is to construct a range of values within which we think the population value falls.

Note that the p-value must be carefully interpreted based on the study design, sample size, comparability of study groups, and appropriateness of statistical tests, statistically significant p-value should not be the sole criteria for accepting or rejecting the conclusions of any report or publication.

In order to interpret p-values, one must make sure that the study is sufficiently powered. That is an adequate sample size was determined at the planning stage of the experiment. Large sample sizes may yield small p-values that have no practical significance and conversely. The magnitude of the p-values has nothing to do with the magnitude of the detected effect size.