Taking references in selection and recruitment, are the validity low, or?
Patrik Reman
Bitr?dande kommundirekt?r | F?rfattare | F?rel?sare | Specialist inom AI, Digitalisering och Framtidens HR & Ledarskap
In a recruitment process, many varied selection tools are necessary to ensure the process's good validity. One of the most common selection steps is the reference check, often used at the end of the recruitment process. If you work with recruitment, I think you have seen the picture where you compare various parts in a standard recruitment process and how the validity is for the other parts. Here, reference taking has a validity of 0.26 compared to the structured interview, which has 0.51. But is the validity of the reference check so low?
The Hunter and Schmidt (1998; 2016) are commonly misinterpreted regarding the reference check. According to Frank L Schmidt, the "reference" in the work presented is not the kind of references we do in the nordic and most European countries. References here are a recommendation letter, "a letter of reference".
To get a more correct validity for the kind of references where you contact people, usually from a previous job, you should check the "Peer ratings" in the Schmidt & Hunter reports, which has a much higher validity, but that are if you as an organisation decide what references you want to ask.
And if you do digital reference checks instead of reference checks over the phone, you get an even higher validity. I have had a conversation with Frank Schmidt, who made the analyzes of validity in the meta-analysis, where the last update was 2016. A large part of the included studies' references is not referenced in the way we often do in Sweden. The references are usually a letter of recommendation. "A letter of reference", or a simple "standard" template with 5-8 questions for the candidates. If you do, for example, competency-based interviews and then they reflect the questions and make a competency-based reference, the figures will be much higher. If you also add the competence-based reference questions digitally instead, you will remove BIAS from the person who interprets what is said at the other end of the phone. When you take references digitally, you get a good set-up of the answers from the references. For example, if you take three references and all 3 have consistent answers, you have higher validity. It is essential here that you take references from people who have worked with the person recently, not 10-15 years ago. The best thing is if you choose the references you want for my organisation we choose one former boss, from the previous job, one colleague and if you are a manager, an subordinate staff.
A factor that in research has been shown to have a significant effect on the usefulness of using reference check to predict success at work is the degree of structure. Reference checks where the interviewer can ask questions freely have lower reliability and lower validity in predicting work performance compared to standardized questions.
Reference checks where the interviewer himself intuitively interprets "between the lines" have lower reliability and validity in predicting work performance. To get the highest possible validity, the questions to be asked must be decided in advance to ensure that all references receive the same questions.
Then, how should I do reference checks?
Determine the purpose of the reference. Do you want to check already collected information, or should the reference provide additional information about the candidate or a combination of both purposes?
Standardize the reference check!
Have three or more references. The more, the better.
Do the reference check digitally to remove a lot of BIAS.
References:
Comparative Analysis of Reliability of Job Performance Ratings
October 1996Journal of Applied Psychology 81(5):557-574
Chockalingam (Vish) ViswesvaranChockalingam (Vish) ViswesvaranDeniz S. OnesDeniz S. OnesFrank L. SchmidtFrank L. Schmidt
Aamodt, M. G., Bryan, D. A., & Whitcomb, A. J. (1993). Predicting performance with lettersof recommendation. Public Personnel Management, 22, 81–90.
Browning, R. C. (1968). Validity of reference ratings from previous employers. Personnel Psychology, 21, 389–393.
Carroll, S. J., & Nash, A. N. (1972). Effectiveness of a forced-choice reference check.
Personnel Administration, 35, 42–46.
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied
Psychology, 79(4), 599-616.
Daniel, D. (1990). Validity of a standardized reference checklist. Applied H.R.M. Research, 1, 51–66.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job
performance. Psychological Bulletin, 96, 72–98.
ISO10667 (2021). Bed?mningstj?nster i arbetslivet – Processer och metoder
f?r bed?mning av m?nniskor i organisationer. Del 1: Krav p? uppdragsgivare.
Del 2: Krav p? leverant?r, 2021 (No. 10667.1.).
McCarthy, J. M., & Goffin, R. D. (2001). Improving the validity of letters of
recommendation: An investigation of three standardized reference forms. Military
Psychology, 13, 199–222.
Validity of different types of references. Personnel Psychology, 12, 469–477.
Oh, I.-S., Wang, G., & Mount, M. K. (2010).
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in
personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.
Taylor, P.J., Pajo, K Gordon W. Cheung, G.W., & Stringfield, P. (2004). Dimensionality and validity of a structured telephone reference check procedure. Personnel Psychology 57, 754–772.
HR-specialist | Expert p? Onboarding / Medarbetarintroduktion | Projektledare | Design av kund och kandidatresor |
3 年Sanna Fjellman
Associate Professor and Deputy Head of MU Research Institute
3 年When we reviewed the research literature about reference checks a couple of years ago, it becomes obvious that in practice, reference checks can be regarded as a differentiated non-standardized method that differs between situations and organizations.? We found it impossible to generalize the results of the research in order to be able to say in general how reliable and valid a the reference check is.?For example, it is much easier to argue that intelligence predict work performance because researchers largely agree on how intelligence should be defined and assessed in a selection context (and it is a lot of empirical support). However, there are some general things to keep in mind when it comes to increasing the reliability and validity of reference checks: Determine the purpose of the reference, should it be a check of information already collected or should the reference contribute additional information about the candidate, or a combination of both purposes? Carry out a thorough work analysis. ? Decide what the reference check should predict (e.g., Contextual or Task Performance) Standardize the reference check (same questions and same answer alternatives) Check your reliability and validity before you say anything about your measurements reliability and validity (we need som numbers here) The last point also applies to peer review and job interviews (in Sweden called kompetensbaserade intervjuer), and I think it's time to stop quoting Schmidt & Hunter (1998), and instead to do your own validation studies.? https://psychometrics.se/2017/06/22/dags-att-sluta-citera-schmidt-f-l-hunter-j-e-1998/
Disrupting reference check | Co-Owner at Refapp
3 年Thanks Patrik Reman for sharing this. I would like to add one thing to increase validity in reference checks; Influence the request of references - as a recruiter you should ask the candidate for the references to choose, based on what Patrik wrote about first determine the purpose of the reference checks. Do not just accept the selected references from the candidate, this is to provide the problem that the restriction of range (sv. beskuren spridning) is a problem when candidate select references. The candidates of course just select those he/she think would be positive. A specific selection and, as Patrik mention, more references (at least three) gives us a greater restriction of range which increase reliability and validity.
Total Reward Expert at SKF Group
3 年I am not familiar with the particular meta analysis that the table is taken from, but that work samples have a high predictive validity has been known for a long time. That's why I have always included a work sample test when recruiting into my own teams. The predictive validity of selection methods can also vary with the type of role recruited for. I once analysed the predictive validity of the methods used in an assessment centre for apprentices. The best predictors were the school leaving grades in maths and native language. None of the assessor based methods had any predictive validity for later job performance at all.
ML Engineer @ Modulai
3 年I would be hesitant to take the coefficient of 0.49 for peer ratings and apply it to Scandinavian reference checks at face value. As the authors of the original article where this coefficient was estimated put it (Hunter & Hunter 1984, page 85): Peer ratings have high validity, but it should be noted that they can be used only in very special contexts. First, the applicant must already be trained for the job. Second, the applicant must work in a context in which his or her performance can be observed by others.? This is because the studies were done in a context that is different from the reference checks we are talking about here. What are your thoughts on this?