Statistical Significance & Practical Significance
The understanding of Statistical Significance and Practical Significance are very critical for any analytical professional. The term Statistical Significance is more from the mathematical world focused on the test statistic that we are evaluating and Practical Significance is mapped to the real world.
A test statistic is a sample parameter which is used to make assumptions about the population parameter with some confidence. Typically, we use a 95% confidence interval when we make assumptions on the population parameter based on the test statistic.
Now that we understand that Statistical Significance in related to a test statistic, let us try to understand Statistical Significance. Assume that our test statistic denotes the mean difference of 2 samples. We calculate the test statistic from the sample and make assumptions about the population mean differences.
Statistical Significance tells us if the mean differences may have been due to chance or there is really some correlation in the population parameter.
Typically, we use hypothesis testing and calculate the Statistical Significance based on the p value (Area of the Probability Distribution Function).
H0: Null hypothesis which normally assumes the difference is due to chance
H1: Alternative Hypothesis assumes differences have statistical significance.
Let me explain the above curve in terms of statistical significance. In the above normal distribution say z or t distribution the p value is the area of the darkened curve. The darkened tails are regions which are statistically significance in other words the differences in test statistic in not due to random chance.
Normally we assume a 5% significance level and if the p value calculated based on the test statistic is less than 0.05 then we reject the null hypothesis and accept the alternative hypothesis. Basically we say that there is statistical significance.
Now we move to Practical Significance which is focused on the real world where we with costs, time, objectives etc. To put in other words practical significance deals with taking a decisions based on the outcome of the test statistic factoring in all real work challenges and constraints.
It is easy to argue that analytical professional should limit themselves to statistical significance and leave the practical significance decision to the business folks. Though to a large extent this is correct it will do a lot of good to analytical professionals to have a better understanding on Practical Significance in terms of objectives as well as decisions that will be taken.
I would like to highlight some points to show to highlight the importance of analytical professionals to understand practical significance and not just limit to statistical significance of any test.
- In analytical projects a key success factor is to clearly understand the objectives in terms of the decisions that will be taken
- Help in clearly identifying the Null and Alternative hypothesis as well as the confidence interval
- Helps better collaboration between Analytical team and Business team as it is not perceived as just a IT project
- Both Statistical Significance and Practical Significance are equally importance together
- Having a large Sample size can help in achieving statistical significance but in order to effectively take decisions there should be practical significance
Now to explain with an example.
Suppose there are 2 sets of random samples A and B having data of primary school going kids.
Sample A are kids from school providing noon meals
Sample B are kids from school that are not provided noon meals. The sample data have 2 variables weight and % of attendance of school kids.
Given the data, as analytical professional performs a statistical test of mean difference of 2 samples. The statistic was the mean weight difference of the students.
H0: Mean difference of population is 0
H1: Mean Difference of population > 0
The results of the statistical test say that there is statistical significance with 95% confidence. Now the question is this practically significant, the answer is based on the objectives of the test and the decision that will be taken.
Suppose the objective is to validate the impact of the noon meal scheme on the student attendance. The decision is to decide whether to identify if there is any strong correlation between noon meal scheme and student attendance.
Suppose the test shows a 5% mean difference between A & B samples, and is statistically significant the govt may decide if it is practical significance based on the on the cost, timeline and critical. Government may feel 5% is not practically significant for the government to roll out the noon meal scheme immediately in a hurry.
However, say there is a 30% mean difference in attendance and is statistically significant, the government may provide a priority to roll out the noon meal scheme across schools as it has a huge impact on the attendance of the students and in-turn have positive impact on literacy rate.
Associate Director at Virtusa Chennai
8 年Interesting Article !
Solutions Architect at BrickSimple LLC
8 年Rajesh, I read through this artice, I like it... Just a few thoughts on this... From the example you cited in this article, I believe that the Practical Significance should strongly follow the Statistical Significance. However, the criteria that decides the Practical Significance will be based on the Objective of the Practice... Let me elaborate... If the Government's objective is to increase literacy among kids through increasing their attendance and if through statistical analysis we determined a strong correlation between attendance and the provision of noon meals, then yes, if there is only a 5% statistical significance, it would be a waste of time, money and effort on the Government's part to put the noon-meals into practice unless the statistical significance may be say as high as 30% as you cited. So they ought to follow the statistical significance when deciding to put the noon-meals into practice for achieving their objective of increasing literacy. But if you are arguing that the Government ought to put the noon-meals into practice even if the statistical significance is low since it is only right to make sure that all kids are fed, then I would say that the Government's objective at that point is no longer ensuring that there is high literacy, but it is ensuring that all kids are fed. Under this objective, their Practical significance criteria that would trigger putting this into practice might be 5% to begin with, which coincides with the statistical significance that was determined. Or did I understand this completely wrong...