Training Evaluation Models

The need to evaluate the effectiveness of knowledge transfer predates the Kirkpatrick Four Levels by at least 20 years. From the need to observe workers performing tasks well after instruction, Ralph Tyler's 1930s eight-year study of curriculum evaluation to Raymond Katzell's papers on responsive evaluation; the interest on evaluating learning results has been there. However, the most known model is Donald Kirkpatrick's (1924-2014) Four Levels of Training Evaluation. The evolutions of that model by Jack Phillips ROI Framework along with the New World Kirkpatrick model by James Kirkpatrick and Wendy Kayser give better guidance today on what to consider in training evaluations.

Pre 1959

In 1959, Donald Kirkpatrick publishes a series of four articles for the American Society of Training Directors (ASTD, now ATD) addressing each of the steps aka Levels of evaluation. Prior to Kirkpatrick's articles, there is evidence of systemic curricular evaluation (Tyler, 1930), and the Training Within Industry (1945) program evaluation from the World War II labor movement. There's also the research of Raymond Katzell published a paper in 1948 calling for responsive evaluations as part of a training program evaluation i.e. gathering feedback from participants

Kirkpatrick's Four Levels of Training Evaluation

Kirkpatrick states in his book and some interviews that he never meant for the four evaluation steps: Reaction, Learning, Behavior and Results to be considered as "levels"; this is probably due to how practitioners tend to change terms or what usually happens in the industry i.e. Bloom's Taxonomy used as a metaphor for the learning process or ADDIE just considered as an acronym.

Reaction

Participant reactions are collected through surveys, focus groups or interviews. When done strategically, reactions can point to inconsistencies between trainers in their delivery and also the learners disposition to use the new skills back at the job.

Learning

Collection of cognitive knowledge transfer and skill acquisition by testing or other quantifiable assessment.

Behavior

Change of observable behavior at the workplace, in most cases, corporate training contributes the most to the adoption of new behaviors.This level focuses on capturing how people apply what they learn back on the job.

Results

At this level, you are looking to validate the training intervention's impact on the organization by meeting the established business needs i.e. if you were cutting ramp up time for a sales team, did you achieve that? How well? Did it meet the expected results?

Phillips Five Levels ROI Framework

Dr. Jack Phillips was one of the first practitioners to take on Kirkpatrick's original.work and expand it to include Return-on-Investment (ROI) determination as the fifth level. The interview below has some great insights about how he became involved in improving the model.

Criticisms of the Four Levels

There have been several critics of Kirkpatrick's original work. After all, it seems the four original articles on. Behavior and Results lacked empirical evidence and this is part of why we see the improvements not only made by Dr. Phillips but also Dr. James Kirkpatrick (Donald's son) and Wendy Kayser (James wife) on the creation of the The New World Kirkpatrick model.

Some academics also noted Kirkpatrick's Four Level to focus on what had already happened, and some overlap on the levels (Watkins, et al., 1998) which led to Roger Kaufman five levels and the Kirkpatrick Plus model . A more recent and salient criticism can be found in several articles by Will Thalheimer, which seemed to be motivated by the release of his own model known as the Learning Transfer and Evaluation Model or LTEM.

The IPISD Does Not Reference Kirkpatrick

On episode 7 of the podcast, we explored the Interservice Procedures of Instructional Systems Design (IPISD) model and the Control (Evaluation) phase seems to have not used Kirpatrick's as a reference. Instead, the guidance is to conduct Internal (team, design, program development) and External (test performance and learner assessments based on objectives) evaluations that would inform any needed system revisions.

Wrap Up

As always, there's much more to know and consider on how you may approach the evaluation of your training initiatives. The patterns that can be observed are:

• Regardless of model, an evaluation program cannot exist without initial goals and identified needs to resolve.
• Evaluations should be strategic with the purpose of not only evaluating observable skills but also the training design and staff performance.
• Although nice to have, learner reaction surveys although popular, may not be as essential as one thought of. The evaluation program should focus on performance and business impact as the result of a training or performance support effort.

Listen to more details on this week episode of StyleLearn Origins

About Alex

Alexander Salas is a US Navy veteran, award-winning Learning Experience Designer, Storyline expert and creator of?StyleLearn Courses ?the online upskill platform for corporate learning. Alex specializes in the integration of eLearning design with popular authoring tools, custom web design, game design, virtual and augmented realities for L&D purposes. Among his professional achievements, Alex is CTT+, Agile Certified ScrumMaster, and a few publications in the L&D space. He's is a regular speaker and contributor at major ATD, eLearning Guild and Training Magazine conferences and publications.

Sources

Kirkpatrick, D.(2011) Long Time ASTD Member Donald Kirkpatrick Shares His Story

Kirkpatrick, J., Kayser, W. (2021) An Introduction to the New World Kirkpatrick Model

Phillips, J. (1997). The Handbook of Training Evaluations and Measurement Methods.

Thalheimer, W. (2018). The Learning Transfer of Evaluation Model

Thalheimer, W. (2018). Donald Kirkpatrick was not the Originator of the Four-Level Model of Learning Evaluation

Thalheimer, W. (2015). The Kirkpatrick Model Good or Bad? The Epic Mega Battle

Watkins, et al. (1998). Kirkpatrick plus: Evaluation and continuous improvement with a community focus