How to build a Hypothesis Test?

One of the problems to be solved by statistical inference is to test Hypotheses. A statistical hypothesis assumes a given population parameter, such as mean, standard deviation, correlation coefficient, etc. 

Hypothesis testing is a procedure for deciding on the veracity or falsehood of a given hypothesis.

For a statistical hypothesis to be validated or rejected with certainty, it would be necessary to examine the entire population - which in practice is unfeasible. 

Alternatively, a random sample of the population of interest is extra. Because the decision is made based on the sample selection, errors may occur:

• reject a hypothesis when it is true 
• don't reject a hypothesis when it's false

A Statistical Hypothesis Test is a decision procedure that allows us to decide between H° (null hypothesis) or Ha (alternative hypothesis), based on the information contained in the sample.

H°— Initial claim or Status Quo

The Null Hypothesis states that a population parameter (such as mean, standard deviation, and so on) is equal to a hypothetical value. The Null Hypothesis H° is often an initial claim based on previous analyses or expertise.

H° is what we take as a result of research through a sample; that is, we reach a value based on sample analysis, tabulate the answers, and come to a product - a hypothesis, since, in practice, we do not consult the entire population.

Ha— Alternative Hypothesis

The alternative hypothesis states that a population parameter is smaller, higher, or different from the hypothetical value in the null hypothesis. The alternative hypothesis is one that we believe can be true or hope to prove to be true. 

For example, based on a satisfaction survey result, 95% of customers are satisfied with the service. We want to challenge the Research Hypothesis - the Status Quo.

Therefore, we launched another hypothesis (alternative), contesting that the level of satisfaction is higher, lower, or different than 95%. Consequently, we performed a hypothesis test to reject or not the null hypothesis.

Types of errors

Because we are analyzing sample data and not population data, errors can occur:

• Type 1 error is the probability of rejecting the null hypothesis when it is true. False—Positive
• Type 2 error is the probability of rejecting the alternative hypothesis when it is effectively true.

The definition of hypotheses is one of the most critical points in a hypothesis test. In practice, we have a business problem, and we have to interpret it and start from that to define the null Hypothesis and the Alternative Hypothesis. 

A wrong definition compromises all future work. The definition of this hypothesis is a business problem, where we interpret a scenario and define hypotheses.

Ex. 1

A researcher has exam results for a sample of students who have taken a training course for a national exam. The researcher wants to know if the students who were students scored above the national average of 82. 

In this case, an alternative hypothesis can be used because the researcher is specifically raising the hypothesis that the scores for trained students are higher than the national average. 

In the penultimate paragraph, we have that the national average for an exam is 82. However, a researcher wants to verify whether or not students who complete the training have a population above this average. Based on this, we could define the hypotheses:

• H°: μ = 82 - status quo
• Ha: μ > 82 - alternative hypothesis

Based on this, we can now choose which hypothesis test we're going to work on:

• Unilateral Hypothesis Test
• Bilateral Hypothesis Test

In this case, we're working on the right one-sided hypothesis test. This is a definition of hypotheses: interpreting the business problem, understanding what is being requested, and defining what H° is and what is Ha. 

If we reverse that definition, we will probably apply the Hypothesis Test, but our conclusions will be completely different.

How to build a Hypothesis Test?

• Step 1 is to define the hypotheses (null and alternative). We should keep in mind that the only reason we're testing the null hypothesis is that we think it's wrong. We state what we believe is wrong about the Null Hypothesis in an alternative hypothesis. 
• Step 2 is to define the criteria for the decision. To define the criteria for a decision, we declare the significance level for the test. It could be 0.5%, 1% or 5%. Based on the significance level, we decided to reject the null hypothesis or not. It is based on business requirements; that is, the definition of the level of significance depends on the business area with which we are working - a hypothesis test for the health area should have a minimum margin of error.
• Step 3 is to calculate statistics and probability. Higher probability has enough evidence not to reject the null hypothesis. 
• Step 4 is to decide. Here, we compared the p-value with the predefined significance level, and if it is less than the significance level, we reject the null hypothesis. By deciding to reject the Null Hypothesis, we can make mistakes because we are looking at a sample and not an entire population.

Therefore, we first formulate the null and alternative hypotheses from the understanding of the business problem. We then collected a sample size n and calculated the sample mean, considering that the mean is the parameter we are studying. 

We traced the sample mean on the x-axis of the sample distribution, and we chose an alpha significance level based on the severity of the type I error. 

Next, we calculate statistics, critical values, and critical regions. And then we make the decisions. If the sample average is in the allowed area of the chart, we do not reject the null hypothesis. If the sample average is in one of the tails, we reject the null hypothesis.

Unilateral Hypothesis Test

We have two kinds of hypothesis tests. The One-Sided or One-tailed Test is used when the alternative hypothesis is expressed as: < or >

Ex: We have two definite hypotheses. H° is the average of any study equal to 1.8, and the alternative hypothesis Ha indicates that the mean is less than 1.8. 

In this case, we have a lower tail test or left unilateral test. The same reasoning fits for the upper tail or right unilateral test; the only difference is that we change the position ode analysis within the graph.

If the mean is within the white region of the chart, we do not reject the null hypothesis; otherwise, we reject it.

Ex. 2

A school has a group of students (population) considered obese. The probability distribution of the weight of students in this school between 12 and 17 years is normal, with an average of 80 kilos and a standard deviation of 10 kilos. 

The school principal proposes a medically monitored treatment campaign to combat obesity. 44The treatment will consist of diets, physical exercises, and a change of eating habits. The doctor states that the result of the treatment will be presented in 4 months. And those students will have their weights decreased in these periods.

• H°: μ = 80 - status quo
• Ha: μ < 80 - an alternative hypothesis

Where: μ = average of students' weights after four months. 

So, what we want is to challenge the status quo. The principal says that the moment he starts a weight reduction campaign, he will decrease the average weight of students. In this case, we will use the mean less than 80, i.e., H. μ < 80, a left one-sided test.

Bilateral Hypothesis Test

A bilateral hypothesis test is used whenever the alternative hypothesis is expressed as "different from"; that is, we are not concerned with whether the alternative hypothesis is greater or lower than a given value; we want to know if the Alternative Hypothesis is different a given value.

We have H°, setting the average to 1.8, and we have the alternative hypothesis Ha with the average different from 1.8. If the average is different from 1.8, it can be greater or lesser than the value. Because of this, we need two rejection areas on the chart. 

The curve above represents the sampling distribution of the average broadband utilization. It is assumed that the population average is 1.8GB, according to the null hypothesis H°: μ = 1.8 - status quo.

Ex. 3

A cookie factory packs boxes weighing 500 grams. Weight is monitored periodically. 

The quality department has established that we should maintain the weight at 500 grams. What is the condition for the quality department to stop the production of the biscuits?

• H°: μ = 500- status quo
• Ha: μ ≠ 500 - an alternative hypothesis

The null hypothesis indicates that each box weighs 500 grams. However, we want to pass to quality control a check to change the weight of the package to stop production. It doesn't matter if the box is 499g or 501g - if it's different from 500, we stop production. In this case, we apply a bilateral test to result in one of the two tails.

Type I and Type II errors

The purpose of the hypothesis test is to verify the validity of a statement about a parameter of the population based on sampling. 

As we are taking a sample as a basis, we are exposed to the risk of wrong conclusions about the population due to sampling errors. 

The Null hypothesis may be true if we have collected a sample that is not representative of the population or is very small. 

To test the Null Hypothesis, H°, defining a decision rule to establish a rejection zone of the hypothesis is necessary to determine a significance level, α, with the most common values 0.10, 0.05, and 0.01.

We have a level of confidence, each associated with a level of significance. According to the value of the α significance level that we define in the Hypothesis Test, we can increase or decrease the level of confidence with which we reject or not the Null Hypothesis. 

Suppose the value of the population parameter, defended by null hypothesis H°, falls in the rejection zone. In that case, this value is doubtful to be the actual value of the population, and null hypothesis H° will be rejected in favor of the alternative hypothesis Ha.

Eventually, although rejected based on data from a sample, the Null Hypothesis is true. In that case, we'd be making a mistake in deciding. This error is called A Type I Error, the probability of which depends on the level of significance α chosen.

According to the business problem, we can use one value or another of α. So, we will increase the degree of confidence or not - The Hypothesis Test is a business tool, helping the decision taker. 

When the value defended by Null Hypothesis H° falls outside the rejection zone, we consider that there is no evidence to reject H° to the detriment of the Alternative Hypothesis. But here, we may also be making a mistake if the Altercation Hypothesis, although discarded by the data we have at hand, is, in fact, true - this error is called Type II.

Ex. 4 

After one year, the effectiveness of a particular vaccine is 25% (i.e., the immune effect extends for more than one year in only 25% of people taking it). A new vaccine develops, more expensive, and one wishes to know if this is, in fact, better. 

• H°: p = 0.25 - status quo
• Ha: p > 0.25 - alternative hypothesis

We want to challenge the Null Hypothesis by verifying that the p-value is greater than 25%.

• Type I error: approve the vaccine when, in reality, it has no effects more significant than that of the current vaccine.
• Type II error: reject the new vaccine when it is, in fact, better than the current vaccine.

To adjust the two errors, we depend on the significance level of alpha. We will increase or decrease the alpha value to increase or decrease confidence when rejecting the Null Hypothesis - the choice is the data scientist.

The probability of making a Type I Error depends on the values of the population parameters and is called α = significance level.

We then say that the significance α of a test is the maximum probability we want to run the risk of a Type I Error.

The alpha value is typically predetermined, and common choices are α= 0.05 and α = 0.01. The probability of making a Type II error is called β.

Confidence Intervals and Statistical Significance

Confidence Interval is a range of values that are likely to contain the actual value of the population. Note that in the confidence interval definition, a probability is associated. At this probability, we call it:

• Confidence Level
• Degree of Trust
• Confidence Coefficient

These probabilities can come from common choices of the degree of confidence that one wishes to achieve, among the most common we have:

A Confidence Interval acts as an indicator of the accuracy of your measurement. And it indicates how stable your estimate is, which can be calculated to determine how close we are to our original estimate when we perform one or more experiments. Therefore, the confidence interval is associated with a degree of confidence that measures our certainty that the gap contains the population parameter.

What is the p-value?

The p-value helps us interpret the results of a Hypothesis Test. The method of constructing a Hypothesis Test is part of setting the level of significance α. This procedure can reject the null hypothesis for an α value and the non-rejection to a lower value. 

So, let's assume we have a business problem where:

• we define hypotheses H° and Ha;
• we collect the data sample;
• we calculate the statistics and set the value of α.

We verified that the null hypothesis H° fell into the rejection area! If we keep everything the same and change the value of α, it may be that with this new value, the H° hypothesis does not fall into the rejection area. After all, what's right? 

Therefore, we need something more to help interpret whether or not we should reject H°, that is, to increase our Degree of Trust ...

Another way to proceed is to present the probability of significance (p-value) or descriptive level. This probability is the value on which we base our decision, so statisticians give that probability a particular name, p-value, or "plausibility value." 

This indicates the probability of more extreme statistical values than that observed under the hypothesis that Null Hypothesis H° is true.

p-value is another indicator that helps make the right decision on rejecting or not H°.

The p-value is a probability of 0 to 1, where 0 indicates impossible, while 1 indicates absolute certainty. Therefore, if we have a p-value of 0.001, which indicates a chance in a thousand, it is improbable the occurrence of a given phenomenon.

The p-value represents the chance or probability of the effect (or difference) observed between treatments/categories due to chance and not to the factors being studied. The p-value helps us to increase a little in our level of accuracy.

Let's assume that one researcher tested the efficiency of two treatments and observed that the mean treatment "A" was higher than the average of treatment "B." After performing the appropriate statistical analyses, the researcher found a p-value = 0.3.

This means that the chance of this difference between the averages is due to chance (and not an effect of the treatments) is 30%. After all, the p-value is a probability of significance. If the researcher states that the differences between the means occurred because of the treatments, he has a 30% chance of being mistaken. 

Analyzing another point of view, that of probability, if the researcher performs the same experiment 100 times, he will find similar results in 70 investigations. After all, he will have a chance to be mistaken since the p-value is equal to 0.3.

How to Interpret the P-Value?

In classical statistics, the p-value (also called descriptive level or probability of significance) is the probability of obtaining a test statistic equal to or more extreme than that observed in a sample from the perspective of the null hypothesis.

For example, in hypothesis tests, we can reject the null hypothesis if the p-value is less than 5%. Thus, another interpretation for the p-value is that this is the lowest probability of significance with which we would reject the null hypothesis. In general terms, a small p-value means that the likelihood of obtaining a test statistic value as observed is improbable, thus leading to the rejection of the null hypothesis.

The p-value is the lowest probability of significance with which the null hypothesis would be rejected.

The p-value is the low value for probability in the blue region of the y-axis; that is, it is the most negligible probability with which we would reject the null hypothesis. 

In general terms, a small p-value means that the probability of obtaining a test statistic value as observed is improbable, thus leading to the rejection of the null hypothesis. In short, we found no evidence in the data so that we can reject the null hypothesis.

Why take a Hypothesis Test?

We conducted the Hypotheses Test to challenge the Status quo, challenging what we have today by rejecting the Null Hypothesis H°. 

? low p-value

A low p-value says that the data we observed would be very unlikely if our null hypothesis were true; that is, the null hypothesis has a low "plausibility."

We started with a model, and now that same model informs us that the data we have is unlikely to happen. That's surprising. In this case, the model and the data are in conflict with each other, so we have to make a choice: the null hypothesis is correct, and we have just seen something remarkable, or the null hypothesis, that is, the model is wrong. 

Suppose we believe in data more than assumptions, then, given this choice. In that case, when we are faced with a low p-value, we should reject the null hypothesis and consider the alternative hypothesis because we do not find evidence to support the current null hypothesis.

? high p-value

In that case, we didn't see anything unlikely or surprising. The data are consistent with the null hypothesis model, and we have no reason to reject the null hypothesis. Events that have a high probability of happening happen all the time.

Does a high p-value mean that H° is true?

No! We know that many other similar hypotheses may also explain the data we have seen. The most we can say is that it doesn't seem to be fake. Formally, we say that "we no longer reject" the null hypothesis. That may seem like a pretty weak conclusion, but that's all we can tell when the p-value isn't low enough. All this means that the data is consistent with the model with which we started. 

• Low p-value, we reject H° in favor of Ha
• high p-value, we do not reject H°, and the test will be inconclusive.

Common mistakes interpreting the p-value

• The p-value is not the probability that the null hypothesis is true but rather a probability of significance to verify whether or not we have evidence within the data to reject the current hypothesis. 
• The p-value is not the probability that the null hypothesis has been deceptively rejected. 
• The magnitude of the p-value does not indicate the size or importance of an observed effect. For example, in clinical research where two treatments are compared, a relatively small p-value is not an indicator that there is a significant difference between the treatments' effects.
• The p-value and significance level are not synonymous. The p-value is obtained from a sample, while the significance level is usually set before data collection.

And there we have it. I hope you have found this helpful. Thank you for reading. ??