Global Standard for L&D Metrics

I recently perused the first edition of ISO/TS 30437, entitled 'Human Resource Management - Learning and Development Metrics', published by the International Organization for Standardization (ISO). Follow this link to obtain a preview copy.

As stated in the introduction, "this document provides a framework and the concepts, metrics, descriptions and guidance necessary to create a basic measurement and reporting strategy." The recommended metrics and descriptions of these metrics comprise most of the 42 page document.

On reading the standard, I was immediately struck by two aspects that are contrary to my research and methodology for designing and delivering effective training. My comments are intended to add value to anyone who is interested in the evaluation of training effectiveness.

Program Defined

According to ISO/TS 30437, a ‘programme’ is defined as a “course or series of courses with similar learning objectives designed to accomplish an organizational objective or need”.

?Consider the above definition in relation to training programs. The obvious problem is that it portrays training as an event, or series of events, and it indicates that all the necessary learning happens during the course or courses.

In a previous article (The Learning x Transfer Equation is Wrong!), I explain that effective training programs have two fundamental phases relating to two types of learning:

• Off-the-job learning, typically via courses, which can take different forms, such as e-learning, virtual facilitated classes, face-to-face facilitated classes, or some blended combination.
• On-the-job learning from application, or learning from transfer. I call it 'self-regulated experiential learning'. This is a form of learning from work[ing], which is also known as workflow learning or learning in the flow of work.

The important point is that learning happens in both phases, not just during courses. And the post-course learning that takes place on the job is crucial to skills development and the achievement of competence and proficiency.

If, as stated above, a program is “designed to accomplish an organizational objective or need”, then it must address the off-the-job AND the on-the job learning necessary to achieve the desired performance outcomes.

Since the second phase of learning is key to achieving program objectives, it seems obvious that this phase should be an integral part of training programs. Unfortunately, it is more the exception than the rule.

There are numerous reasons why off-the-job learning does not become the foundation of on-the-job learning. One of the most significant, yet possibly least understood, is the design gap that exists because learning from application is not treated as a fundamental part of the learning journey.

Levels of Effectiveness

The ISO standard identifies three categories of metrics (efficiency, effectiveness and outcome) and has adopted the following five levels from the Kirkpatrick [1] and Phillips [2] models:

In their book 'Measurement Demystified', Vance & Parskey [3] make it clear that each of the above levels relate to effectiveness. Later, they explain that "Level 4 is so important ... we have called it out from other effectiveness measures and made it a separate category called outcome." The ISO/TS 30437 standard does the same, so the effectiveness metrics provided only relate to Levels 1, 2, 3 and 5.

I believe that we have a language problem in L&D. Time and again I encounter words being used to mean different things, and different words to mean the same thing! I am also acutely aware how the labels we use can influence understanding and mental models.

My concern with the above framework is that it is misleading. Consequently, where off-the-job training is concerned, it is common that Level 1 (Reaction) and Level 2 (Learning) are associated with courses, and Level 3 (Application) and Level 4 (Outcome) are associated with post-course 'transfer'.

Now consider the following:

• Reactions are also relevant to the on-the-job learning (e.g. participant reactions to the support they receive, and manager reactions to the support provided by L&D) and provide valuable data regarding important drivers of training success.
• The labels used for Level 2 (Learning) and Level 3 (Application) imply that learning happens before application and that application comes after learning. But, as explained above, on-the-job application is another phase of learning that is essential to skills development and achieving desired performance standards and outcomes.
• Application and learning go hand-in-hand. With the appropriate guidance and support, application (new work behaviour) leads to learning and the learning influences future application. It's a dynamic, interlinked process that takes place in the flow of work.

My Approach

I differentiate between a course and a program. Courses only provide off-the-job learning; programs also incorporate on-the-job learning. I avoid using terms like pre-training and post-training when referring to courses.

I put more weight on metrics associated with the on-the-job learning (see Lens 5 in the Six Lenses Evaluation Model). These include reaction, learning process, and proficiency/performance measures. It is important to assess the extent to which skill/ability has increased.

My experience is that most managers do not set a very high bar for evidence of success; they just want sufficient data to be assured that the training is producing the desired performance and impact. The appropriate evidence inevitably relates to post-course application and associated learning.

Finally, it's important to keep in mind that (a) positioning on-the-job application as a fundamental part of training, and (b) measuring the process, support and results of learning from application, sends powerful signals to participants and their managers.

References

1. Kirkpatrick, J. and Kirkpatrick, W. Kirkpatrick’s Four Levels of Training Evaluation. Alexandria, VA: ATD Press, 2016.
2. Phillips, J. and Phillips, P. The Handbook of Training Evaluation and Measurement Methods, 4th ed. New York: Routledge, 2016.
3. Vance, D. and Parskey, P. Measurement Demystified:?Creating Your L&D Measurement, Analytics, and Reporting Strategy. Alexandria, VA: ATD Press, 2021. [Preview] The quoted text is from 'Level 4: Results and Impact' in Chapter 4.

Author

Throughout my L&D career I have continuously researched and experimented with ways to increase learning effectiveness. Along the way I have immersed myself in the 'science of learning' and the 'science of instruction' and have learned from successes and failures.

I know from experience that training can be very powerful if appropriate and implemented properly, which means addressing the off-the-job and the on-the-job phases of learning. I have personally been involved in designing and implementing training programs that consistently produced an ROI in excess of 100%.

Over the last 20+ years I have successfully delivered many professional development programs for learning specialists. I have also created and delivered programs to help managers get better results from employee development.

If you would like to arrange a chat or want more info about my research, please email me at [email protected].