You are Assessing Talent the Wrong Way. This is How to Avoid it (2/2)

Hey, we are Ramón Rodrigá?ez Riesco and Andrea Marino , co-founders at Nova, the Global Top Talent Network.

Welcome to Talent First, our monthly newsletter where those who believe that talent is the most important resource in the Economy get together to learn and discuss about attracting, hiring, developing, and retaining talent.

If you prefer to receive the email in your inbox, feel free to subscribe here .

Summary of this edition

1. Talent Assessment 2/2 - why most selection processes (particularly “massive” ones with hundreds of candidates) are wrong and how Holistic Assessments are 10x better
2. Novas open for a change (scroll to the end) - The high potentials of Nova who are now open to a career change but don't disclose it elsewhere. You can contact them directly for free.

? Talent Assessment (2/2)

In our last issue, Talent Assessment (1/2) , we started our discussion about talent assessment and introduced the concept of Holistic Talent Assessment as an alternative to traditional waterfall processes.

TL;DR? Quick summary about Talent Assessment (1/2)

1. Waterfall Assessments, the traditional way

• As a company, you always look for the best candidate overall, not the one with the best CV or the best IQ test. This means the person who has the right experiences (E), the right capacity for hard skills (H), the right competencies and soft skills (S), and is the right cultural match (C).
• Therefore, the way to assess talent must be some kind of weighted average with potentially some thresholds. For instance, if you believe all are equally important (which is unlikely), your calculation of the final score of a candidate would be: 25% x E + 25% x H + 25% x S + 25% x C
• However, most selection processes, particularly massive ones, follow a funnel or “waterfall” process by which candidates only get to the next step if they are successful in the previous. Typically, companies have 3 steps: CV screening, testing, and interview

2. Why Waterfall Assessments are fundamentally wrong

These waterfall processes look at each measure independently generating both:

• False positives: candidates in the final stages that are not among the best
• False negatives: candidates among the best that are rejected early in the process, typically because of a poor CV (E)

3. What the science says about assessments

Science proves that, as we had anticipated, the best selection processes (those that predict job performance more accurately) are holistic, in the sense that they assess several aspects of the candidate at the same time.

4. Introducing a Holistic Talent Assessment

To create a “Holistic Talent Assessment”, organizations need to design a process where candidates go through multiple stages of the selection process before being discarded, like a resume (i.e. background and experiences), testing / task-homes (i.e. IQ and or hard skills) and interviews (i.e. competences and cultural fit). Although it may look costly, technology is here to help as we will see in this week’s newsletter.

5. The math around Holistic Talent Assessment

At this point, you are most likely convinced that assessing talent in a holistic way is more accurate, but you probably still believe it is not practical and time-consuming. We want to prove you wrong. What if we told you that a holistic assessment can be both more accurate AND less costly? Let us do the math together.

Let’s imagine the following case: “Corporate Inc.” is assessing talent for a Graduate Program, where they look for great young talent without much prior experience. They typically receive around 500 candidates and hire 10.

“Corporate Inc.” has created a scoring model which gives the following weight to the different aspects of the profile:

• Right experiences (E) - 20% (measured through extracurricular activities and internships)
• Right capacity and hard skills (H) - 20% (measured through university grades and tests)
• Right competencies and soft skills (S) - 30% (measured through behavioral questions in an interview)
• Cultural fit (C) - 30% (measured in an interview)

We will analyze 3 alternatives of the selection process that “Corporate Inc.” could use and compare:

• Their cost
• The candidates they choose (i.e. their accuracy)

5.1 The selection processes available

A. The waterfall process:

The classical waterfall process will be composed of the following steps:

1. CV screening (500 candidates). Accounting for processing them, scoring, etc., let’s assume this takes 3 mins / CV. About 50% are sent to the test phase.
2. Testing (250 candidates). Let’s assume it takes another 3 mins to process and review the grades and a test. The best 40% are sent to the HR interview phase.
3. HR Interview (100 candidates). It’s a 30 min-long interview in which the candidate actually speaks for about 50-60% of the time. With setup and processing, it takes at least 45 min/candidate.
4. Interviews with the team (50 candidates). 2x 45min interviews with the best candidates (to the second interview only the best candidates arrive). The 10 offers are decided here, taking about 1h/candidate/interview.

B. The pure holistic process

We want to consider a “pure” holistic assessment that still maintains the final interviews with the team, but where we substitute the HR interview for a video interview, which can be done asynchronously (thus improving the candidate experience). If we combine that video interview with the CV screening and a test, the new “pure” holistic process will consist of only 2 steps:

1. Holistic first step. All candidates would do:

• CV Screening (same 3 mins per candidate)
• Testing (same 3 mins per candidate)
• Video interview. We look for the same things we did in the HR interview in the waterfall process. Reviewing the candidate's answers takes a maximum of 10-15 minutes.

1. Interviews with the team (same as in waterfall).

C. An alternative “semi-holistic” assessment

Finally, we want to analyze a second version of the holistic assessment a bit in between the “pure” version and a typical waterfall assessment. The steps would be:

1. Holistic first step. All candidates would do:

• CV Screening (same 3 mins per candidate)
• Testing (same 3 mins per candidate)
• A video interview (same as in pure holistic). In this case, we will save time by only reviewing the best 250 from the CV screening (candidates would have taken the interview, but we do not look at it).

1. Interviews with the team (same as in waterfall, but only sending the best 30 candidates as we trust this is a better measure of talent).

5.2 Cost Analysis - can holistic assessment be similar to waterfall in terms of cost?

The image below shows the classic waterfall interview process and the estimated cost of assessing those candidates. With very conservative estimates, the cost of this process is 12,625€ and the likelihood of both false positives and negatives is quite high as we will see later.

A “pure holistic” assessment in which we keep the 2 live interviews of the team to the same volume of candidates (the best 50) and simply exchange the initial waterfall for a holistic assessment with a 15-minute video interview would yield better results in terms of accuracy with just ~25% extra investment.

Finally, the “semi-holistic” selection process proposed would reduce the cost of the waterfall assessment by 16% at a very low accuracy cost as we will soon discover:

5.3 Accuracy Analysis - would I choose different candidates with waterfall vs. holistic assessments?

We have just proven that a holistic assessment can be approximately in the range of cost of a traditional waterfall assessment (from -16% to +25% of the cost). We already know science proves it’s more accurate. However, how much more accurate? Do you not end up choosing the same candidates either way?

If we can prove that waterfall and holistic assessment deliver different candidates with approximately the same cost, as we know the latter is a better predictor for job performance, there should be no reason why companies would not massively shift to this approach.

To prove it, we generated a spreadsheet with 500 candidates at random with different scores across the 4 dimensions valued by “Corporate Inc.” (E - experiences and background, H - intellectual capacity and hard skills, S - soft skills, and C - cultural fit). We assessed which candidates arrived at final interviews with the team following the 3 methods described above:

A. Waterfall assessment:

B. Holistic assessment (+25% of cost vs. waterfall)

C. Semi-holistic assessment (-16% of the cost as waterfall)

We will consider the Holistic Assessment the baseline, as its measure is the most predictive of job performance, and compare the candidates delivered by the other processes against it. In the end, we cannot know if those candidates are “the best”, but we can know if they would have been chosen with the most accurate selection process.

At the end of this experiment, by comparing which candidates get to the final stage, we can compare the processes on the following key metrics:

• % difference of average score of final candidates vs. holistic process: we want to get the highest quality of candidates possible to the final round. The higher this number, the less accurate a selection process is.
• % of final round candidates are different vs. holistic process: we want to have ideally the same final candidates in the final round as those which the Holistic Assessment has (as those are the “best”). The higher this number, the less accurate a selection process is.
• % of the top 10 candidates of holistic assessment not in the final round: we want the best 10 candidates (who should be the ones hired if our process works well) to arrive at the final live interviews. The higher this number, the less accurate the selection process is, as we would not be giving the opportunity of the live interview to the best candidates.

We ran this experiment 1.000 times and calculated the 3 metrics across the different selection processes. The results show how Waterfall processes are wrong and deliver very different outcomes:

* Note that these numbers change with the weights and volumes. For instance, if the % of weight to E increases, the Waterfall process does better and vice-versa. Let us know if you want the spreadsheet model to run your own iterations in the comments.

6. Conclusion

The math we have seen shows that, if you are still doing Waterfall assessments, you could be both:

• Wasting your time with 43% of the candidates you send to the final rounds, as they are not among the best.
• Missing 17% of the best candidates, who you are not passing on to the final round because you are just analyzing their CV or their CV and a test.

Holistic processes are more accurate and not that much more expensive (+25% of the cost) thanks to technology and video interviews. If your budget is limited, you can even reduce cost in a “semi-holistic” selection process, which will deliver almost the same accuracy candidates (below 5% difference in finalists and almost all the really top candidates) with 16% savings on cost.

Now that we hopefully convinced you, if your company wants to implement a holistic assessment for a massive selection process and you need help with the technology or a calibrated team who can help with reviewing candidates, reach out here .

?? Novas open for a change

And now, we want to introduce you to +50 Novas (i.e. top talented, pre-vetted individuals) who are passively looking for new job opportunities. You can reach out to them with your open vacancies for free!

Meet +50 Novas

PS: At Nova, our mission is to become the Talent Agents of the most talented people in Business and Tech. We have created a merit-based access community of+18.000 pre-vetted, high-potential individuals who trust us to help them achieve their full potential through networking, development, and career acceleration opportunities.

Every month, we share the anonymized profiles of Novas who are now open for a change so that HR Managers and Founders can contact them directly and avoid wasting time on spamming candidates who are not ready for a change on LinkedIn.

Thanks for reading Talent First. If you liked this issue, don't forget to Share Talent First by Nova with other people who might like it or who are looking to hire top talent.