Validity Co-Efficients for Recruitment Leaders

Recruitment Leaders! Are you sick of a Hiring Managers arbitrary biases being put ahead of your data driven insights? Here's how to measure if an interview ACTUALLY predicts future success ??

Nerd Alert : I'm about to bang on about Validity Co-Efficients...

We've all been there. You want to set up a really good technical & behavioural interview. Questions linked to specific success criteria. Review guidelines all written up. Interviewers with the right training.

Then the hiring managers says :

"but only if they have 10 years experience and got a 1st from a good college"

Then you cry inside... knowing that this offhand comment all but guarantees an increased cost and time to hire, as well as kill any progress towards your diversity goals.

What's more, you have little negotiating power. This is how they've done it for ten years, how can you fight against that?

Here's the good news. People have been studying how well different interview formats and criteria predict future success for decades. There are hundreds of studies and multiple meta-analyses on this topic and the results are entirely in our favour.

So, how can we quantify how well something predicts future success? Validity coefficients.

What is a Validity Co-Efficient?

A validity coefficient is a statistical measure that indicates the degree to which a test or assessment accurately predicts a specific criterion.

It shows how well a particular assessment measures what it is intended to measure.

The validity coefficient is typically represented as a correlation coefficient, ranging from -1.0 to +1.0.

+1.0 = a strong positive relationship, meaning the test is a good predictor of the criterion.

-1.0 = a strong negative relationship.

Around 0 suggests no relationship, indicating that the test does not predict the criterion effectively.

In the context of recruitment, a high validity coefficient for a selection method (like an interview or test) suggests that it is a good predictor of job performance. This is important for making informed hiring decisions, ensuring that the candidates selected are likely to perform well in the job.

Validity Co-Efficients of Common Assessment Formats

Let's explore the effectiveness of years of experience, college degrees, behavioural interviews, and in-person skills tests in predicting job success:

Years of Experience

The predictive validity of years of experience for job performance is relatively low, with a validity coefficient of 0.18. This suggests that relying heavily on years of experience in recruitment can be misleading.

Overemphasis on years of experience might also introduce biases and overlook candidates with high potential but less experience.

College Degree

Interestingly, the validity coefficient for years of education is only 0.10. This indicates that educational qualifications, such as possessing a college degree, are not strong predictors of job performance (Massive caveat here depending on specific role. Completing a medicine degree... HUGE indicator of success for doctors of course).

This finding challenges the traditional emphasis on formal education in recruitment processes.

Behavioural Interviews

Behavioural interviews, where candidates are asked to describe past behaviour and experiences, are a common part of many recruitment processes. While specific validity coefficients for behavioural interviews aren't as available as I'd like, we know that structured interviews, which often include behavioural questions, have a higher validity coefficient of around 0.51.

This suggests that behavioural interviews, especially when structured, can be effective in predicting job performance.

Situational Skills Tests

Skills tests, which can include practical tasks like pair programming, or simulations relevant to the job, have a high predictive validity.

Work sample tests have a validity coefficient of 0.54, making them one of the more effective methods in predicting job performance. These tests provide a direct assessment of a candidate's ability to perform job-related tasks.

This is data from multiple studies and sites i'll reference below, but it's worth noting that you can measure the validity coefficient of interviews in your own business using criterion-related validation.

Calculating your own Company's Validity Co-Efficients

This process requires correlating the scores from your interviews and assessments with an objective measure of job performance.

1. Determine the Criterion (Job Performance Measure): First, you need to define what constitutes successful job performance in your organisation. This could be sales figures for a sales role, quality of coding for a software developer, or customer satisfaction scores for a customer service position.
2. Collect Assessment Data: Gather data from your interviews and assessments. This could include interview scores, test results, or ratings from assessment centres.
3. Collect Performance Data: After a certain period (e.g., six months or a year), collect performance data for the same set of employees. This should be quantifiable data that reflects their job performance, aligned with the criteria you defined earlier.
4. Correlate the Two Sets of Data: Assuming Column A has your assessment scores and Column B has your job performance data. Click on the cell where you want the correlation coefficient to appear.Enter the formula: =CORREL(A:A, B:B)EG, if your data is from row 1 to row 100 in both columns, the formula would be: =CORREL(A1:A100, B1:B100)
5. Interpreting the Coefficient: A high positive coefficient (close to +1) means a strong relationship between the assessment scores and job performance, indicating your assessments are good predictors of job success. A low or negative coefficient suggests that the assessments are not effective predictors of job performance.
6. Consider the Context: Remember that the validity of an assessment can vary based on job roles, the organisation's culture, and other situational factors. It's beneficial to conduct this analysis separately for different roles or departments.
7. Legal and Ethical Considerations: Ensure that your methodology and the use of performance data comply with relevant employment laws and ethical standards.
8. Continuous Improvement: Use the findings to refine your assessment processes. If the validity is low, consider revising your interview questions, assessment methods, or the criteria used for measuring job performance.

This approach requires a good understanding of statistical analysis and additionally, the reliability of this process depends on having a decent sample size and accurate, unbiased performance data.

So, that was a pretty heavy one. But if you can take this information and use it to shape your interviews and success criteria to have the highest possible chance of predicting future success, your business will have fewer probation fails, higher retention, lower cost per hire. its huge.

References

1. Recruiting Toolbox. (n.d.). Validity of Interviewing and Selection Methods. Retrieved from recruitingtoolbox.com
2. Society for Human Resource Management (SHRM). (2020, February 27). How to Evaluate Hiring Assessments. Retrieved from shrm.org
3. TestGorilla. (n.d.). Revisiting the validity of different hiring tools: New insights into what works best - Part 2. Retrieved from testgorilla.com
4. SHL. (n.d.). Guidance for the Interpretation of Validity Coefficients. Retrieved from shl.com
5. NSW Public Service Commission. (n.d.). Predictive Validity of Assessment Methods. Retrieved from psc.nsw.gov.au
6. Testgrid. (2021, June 15). Psychometric Assessments: Can They Really Predict Job Performance? Retrieved from testgrid.com
7. Alva Labs. (n.d.). Why Judging Candidates on Years of Experience Could be Increasing Bias in Your Recruitment. Retrieved from alvalabs.io