How do p-values interact with confidence intervals in statistical analysis?

The p-value is a fundamental concept in statistical hypothesis testing. It measures the probability of obtaining test results at least as extreme as the observed results, assuming that the null hypothesis is true.

P-value is the probability of observing a sample statistic as extreme or more extreme than the one observed under the assumption that the null hypothesis is true. Suppose you are testing whether a new drug is more effective than a placebo.The null hypothesis is that there is no difference in effectiveness. P-value: If you get a p-value of 0.03, you would reject the null hypothesis at the 0.05 significance level, suggesting the drug is significantly more effective than the placebo. Confidence Interval Suppose the 95% CI for the difference in effectiveness is (1.2, 4.5). Non-zero, indicating that the difference is statistically significant and suggesting that the true difference in effectiveness likely lies between 1.2 and 4.5 units.

A confidence interval at confidence 100(1-a)% for a parameter T is a RANDOM interval (l(X), u(X)) such that P[l(X) < T < u(X)] = 1 - a. Things to note: * Every sample of data X yields some confidence interval (l(X),u(X)). Since the sampled data X is random, the formed confidence interval is random. * The interpretation of a confidence interval is NOT: T belongs to the formed confidence interval with probability 1-a. This doesn't make sense because T is fixed, so it doesn't have a probability distribution. Instead, the interpretation is: If one were to generate a large number of samples X, the proportion of intervals containing the true value of T is about 1-a.

Confidence intervals give you a range within which you can be confident the true coefficient value lies.A confidence interval is defined by two limits: the lower bound and the upper bound. These limits are calculated based on the sample data, the desired confidence level, and the variability in the data.

Do you know that p-value and confidence interval are about the "process", but not about any particular observation? This topic is very nuanced and many people make mistakes here. 95% CI means, if you follow the process 100 times, for 95 of them you will observe sample means fall into this interval. Why it matters? Because given another observation, anything can happen (extreme things happen with lower chance, but are still possible). You just CANNOT make any claims about the confidence interval when you have only one observation. This is fundamental to a frequentist view of the world and really speak to if the person truly understand the first principles.

I’d also add that more and more, researchers are coming away from just reporting p-values and confidence intervals. A measure of effect size, like cohen’s d, which provides a measure of effect in terms of standard deviations, is generally expected and preferred, at least in social research. Effect sizes let us assess if an effect is practically important, even in the presence of a significant p-value. P-values are influenced by sample size and should not be relied on as the only measure, even when coupled with confidence intervals, when attempting to interpret research findings.

You should not only give the p-value, but other relevant quantities that go into computing the p-value, such as standard error and sample size. This allows your audience to contextualize and also understand the magnitude of the quantities you are measuring.

When a confidence interval includes the value of no difference, it means the experimental intervention is not significantly different from the control intervention. In such instances, the p-level (probability) does not support rejecting the null hypothesis.

When conducting logistic regression, it is importance you remember what value you are interpreting for the 95% CI. For example, if you are interpreting the log odds, if your 95% CI included 0 it is not statistically significant. But if you are interpreting the odds ratio, it would not be significant if the value 1 is included.