Learning is Overrated

I love Learning

First, I want to say that I really do love learning. To demonstrate this point, I want to share four facts about myself:

1. I hold a master's degree in “Instructional Psychology and Technology” (A.K.A. Learning)
2. I’ve been studying and designing learning experiences for 17 years.
3. My current job title is “Learning Content Developer.”
4. In my personal life I read or reread a book almost every week.

I could go on, and I think you understand: I love learning.

Learning IS Overrated

So why write an article that discounts learning?

In the vast majority of learning situations that I’ve been involved with, learning has been enthroned as the supreme goal, while applying what was learned has been forgotten or put aside with minimal attention.

This is why I say that learning is overrated. Learning shouldn’t be the central focus. Learning can be valuable, and it is a means to an end. You can see that I’m a functionalist. Happily. And as one who makes regular student loan payments, I’m always asking: What is the ROI for the learner?

Unfortunately, there is zero ROI for learning. There is only ROI for application.

How Learning is Overrated

Let’s cut to the chase with four examples of how learning can be a waste of time and money. These examples are hypothetical examples based on my four real life facts above:

1. If I obtain a degree and graduate not having the skills to function in an entry level position in my field I just bought a very expensive piece of paper.
2. If I do a needs analysis and then design, develop, and implement a course, measuring how much the participants recalled one week later, real world impact is not necessarily realized.
3. If I create and deliver a workshop for participants to learn about leadership skills, and I don’t help them develop the leadership skills, I’m leaving most of the opportunity on the table, as I abandon participants for the most difficult and most important component of the training: behavior change.
4. If I read an exciting book and I can’t figure out how to apply what I learned, the book only holds entertainment value (which is great, as long as that was the only expectation).

Learning and entertainment are fun. I do put them in the same bucket. After all, if learning isn’t entertaining, it might as well be called “torture.”

Learning can help us relax or even increase awareness. And if it increases awareness it fulfills its valid purpose as the prerequisite for application.

You might have had a teacher or trainer who told a lot of stories and you may have wondered what that had to do with the learning objectives. A well chosen, and well told story can give actionable information about learning objectives.

However, you might notice the last syllable of the word “actionable.” It leaves something to be done by the participant. It pushes the accomplishment of the learning objectives into the learner’s realm of responsibility. This is appropriate in some situations. Just not that many that I can think of. It does a huge disservice to the learner, as application is usually the most difficult and most important component of training and development. We can do better.

In most contexts the learner needs additional help - not just a well told story (or even an explanation) in order to develop the desired skill. Here are some key examples of where explanation or demonstrative storytelling is clearly not enough by itself:

• Sports
• Musical instruments
• Math, science, and writing
• Dancing, photography, and other expressive art forms
• Public speaking
• Leadership development

Clearly this list could go on and probably extend into most of the skills involved in your job and mine. It begs the question - how are we supposed to apply what we learn if there is no structure for application provided as part of the learning experience?

Kirkpatrick, a program evaluation guru, created such a structure, which is how I mentally frame all-things-learning. Viewing this dilemma through Kirkpatrick’s 4 levels unlocks several secrets.

Kirkpatrick’s 4 Levels of Evaluation

For you corporate learning professionals out there, Kirkpatrick’s 4 Levels of Evaluation are either something you “know/knew about” or something you actually use on a daily basis.

Either way, what you need to know is that Kirkpatrick’s 4 levels are considered by tens of thousands of learning professionals, scholars, trainers, and otherwise to be the gold standard for evaluating the effectiveness of learning experiences. The question is: Did you measure a positive impact on all four levels?

A quick review of each level will provide understanding of how learning is overrated, and what we can do about it.

Level 1: Reaction

The degree to which participants find the training favorable, engaging and relevant to their jobs.

If a learning experience is evaluated, this is where it usually ends. It can be very gratifying to hear from a learner (or stakeholder) something like: “Hey, I really enjoyed the training you facilitated last week. You did a really good job with that." From personal experience, I know this type of praise feels good, as it does to receive high satisfaction scores or when clients are happy with the product. Unfortunately, it doesn’t mean much in terms of impact.

Level 2: Learning

The degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training.

Notice that Kirkpatrick’s first level doesn’t even imply learning. It’s here, in Level 2 that learning is measured. The two major elements of learning are knowledge and skill. Typically when we talk about measuring learning, we're speaking strictly of the knowledge element.

In the knowledge category, we have multiple choice tests, papers, and activities where the teacher/trainer takes inventory of what people are understanding about the material. Even Bloom’s Taxonomy includes such layers as “application.” However, we often accept an isolated mental effort of connecting dots as “application” to be the same as skill. This, however, may be an over-estimation of the knowledge element. Consider the following examples of what could be considered application within Bloom’s Taxonomy, yet clearly there is a division of knowing versus doing.

Knowing (Knowledge)

• Explain the steps of the scientific method for testing a new toothpaste...
• Tell how the company’s mission statement relates to your job role...
• Describe how the steps of ADDIE could be used to manage a hypothetical project...

Doing (Skills)

• Carry out an experiment using the steps of the scientific method and turn in a report.
• Tell of time when, given a difficult situation, you exemplified a principle in the company’s mission statement. What was the situation, what was your behavior, and what was the impact?
• Complete a project using the ADDIE model and another project using the Agile methodology. Discuss differences in how decisions, mistakes, and mid-project changes and deadlines were handled.

The difference that I see is – if a person wants to develop a skill, they actually have to do the thing they're trying to learn. It's more than just talking and writing about a topic. There is a huge opportunity for skill development within Level 2. This value is generally left on the table throughout the adult learning world.

This limitation (the over-focus on knowledge alone) is also a large roadblock to breaking into Level 3.

Level 3: Behavior

The degree to which participants apply what they learned during training when they are back on the job.

I feel that this level is self explanatory, yet it desires a second read: The degree to which participants apply what they learned during training when they are back on the job.

It’s vastly different than Level 2 – even the best Level 2 measurements. The best Level 2 measures include evaluating whether the participant can demonstrate the skill in a simulated environment. Level 3 measures how well participants practice the skill on the job. It's where the real value begins to happen. This is where people get better at their jobs.

This is where things become very interesting. Which of the following do you think your stakeholders would value most as a result of a development or training experience?

1. Participants enjoyed themselves (Level 1).
2. Participants learned something (Level 2).
3. Participants improved specific, targeted work behaviors (Level 3).

The reason we think training and development experiences will be valuable is typically because we think they will change work behavior. If we believed anything less, we probably wouldn't invest so much time and money in helping people get better.

However, there is a concrete wall 1000 feet thick and 1000 feet tall between Kirkpatrick Level 2 and Kirkpatrick Level 3. It’s too high and too wide to throw a rock or jump over. No ladder is tall enough. We will probably never arrive at Level 3 via lecture, storytelling, group discussion, exploration activities, visual aids, video segments, assigned readings, memorization, brain storming, or fire walking (unless the skill being learned is one of those activities – you have to practice fire walking in order to become a great fire walker).

New content, new visual aids, and better packaging have little hope of passing over or through this wall. To overcome this hazardously large stumbling block, we need a very deliberate, advanced strategy. We need to develop and deploy entirely different methods (not just better/new/different content).

Another piece of this puzzle is that our expectations of on-the-job application are very fuzzy. I find the following questions helpful to correctly set up a Level 3 measurement:

• What was the specific behavior we want participants to apply on the job?
• How many participants performed the behavior correctly while on the job?
• How long did the change last?

Very few organizations measure Level 3. If you do, congratulations! You're on the cutting edge (please tell us your secrets in the comments below). Indeed, there is a great need for research, experimentation, further application, and success stories in this area.

Level 4: Results

The degree to which targeted outcomes occur as a result of the training and the support and accountability package.

How many more customers were served per hour? How many more widgets were produced per day? How did client satisfaction scores improve? These are high level metrics that indicate the overall success of the organization.

Occasionally these are measured. I believe the reason they are not measured more often (and the reason why these measurements may be less helpful) in relation to training and development experiences is because so many things affect these outcomes. There are many lurking variables. This is not so say that the measurement of Level 4 is worthless. I believe it does require more deliberate measurements to be helpful. These measurements often take the form of treatment and control groups, and merit their own article. Moving on (for now).

Deliberate Practice: The Mechanism of On the Job Application

In light of Kirkpatrick’s four levels, I believe our greatest opportunity in the adult learning industry is in the second half of Level 2 and Level 3.

The first half of Level 2 is knowledge. If we aren’t measuring knowledge, we can be aware: this is low hanging fruit! A simple quiz can be aligned with the learning objectives of any module. Any legitimate LMS can be leveraged to automate the administration and grading of multiple choice style tests. Through a bit of user testing and iterative revisions, solid Level 2 learning assessments can be created if they are not already in place. That said, measuring knowledge may not even be necessary if you're going to measure skill, and on-the-job application.

Rarely is it the case that the learner merely needs to know something and not do anything with it. Here is where the second half of Level 2 (skill) comes in. Although it's often more involved, I believe this half offers far more return on investment. Luckily, the groundwork has been laid (for example) in competitive sports coaching. Competitive sports utilizes an easy-to-visualize model for becoming an expert in specific skills. This model is called Deliberate Practice (DP). DP is the secret weapon for skills development, and it can be more easily scaled using a formal rubric.

Note: Please see a full article on Deliberate Practice Rubrics by clicking here. Deliberate Practice Rubrics can also be leveraged in Level 3 (on-the-job application) with even greater ROI.

In short, a Deliberate Practice Rubric lists several components of a skill and enables a rater to give a score for each one. The participant sets a very specific goal to improve one aspect of performance and practices in a simulation (Level 2) or on the job (Level 3) and is rated. The participant then has the opportunity to reflect on the ratings and make a specific plan to improve performance. Finally, the cycle is repeated and performance is measured again. Each cycle builds on the last. Improvements in performance can even be tracked and graphed over time (if you're that nerdy – like me).

Professional athletes, musicians, physicians, pilots, and others use these general steps to hone their skills and demonstrate excellent performance (and you can too!). The more cycles, the more opportunities to take a tiny step upward in mastering the skill.

Learning can be great. And without deliberate practice cycles to actually develop a skill (receiving feedback, reflecting, and trying it again with better plan), learning can be a worthless, dead end process. I find that simulations (at the least) and on-the-job application (preferred) is more engaging, fulfilling, memorable, motivational, real, and valuable than learning by itself. If you find this sort of thing interesting, watch for my future articles for more ideas and examples of Levels 2 and 3 evaluations.

I’m curious how you have leveraged deliberate practice to measure Kirkpatrick Levels 2 and 3 – on-the-job application (please tell us in the comments below). To be honest, this is the primary struggle of my occupation and my own career development. I can use all the help I can get!