How do you design evaluations of learner reaction and learning outcomes?

Designing evaluations of learner reaction and learning outcomes involves a systematic approach to gather feedback and assess the effectiveness of learning initiatives. Begin by clearly defining the objectives of the evaluation, including what aspects of learner reaction and learning outcomes will be measured. Utilize a combination of surveys, interviews, and observation techniques to gather data on learner satisfaction, engagement, and perceived effectiveness of the learning experience. For learning outcomes, employ assessment tools such as quizzes, tests, or performance evaluations to measure the acquisition of knowledge and skills. Ensure that evaluation methods are aligned with learning objectives and provide actionable insight.

The Kirkpatrick Model is the gold standard for evaluating the impact of training & development programmes, since the 1950s. The four levels are:-- 1. Reaction: How do participants react to the training-- immediate perceptions, engagement, relevance. 2. Learning: To what extent did participants acquire the intended knowledge, skills and attitudes from training. This uses assessments, tests, simulations, and observations. 3. Behaviour: Are participants applying new skills and knowledge in their daily roles? What are the changes in behaviours and practises? 4. Results: The impact of the training on organisational objectives. ROI includes improvements in productivity, increases in sales, higher quality levels, reduced costs.

To design effective evaluations of learner reaction and learning outcomes, start by defining clear objectives and aligning them with the course content. Use a mix of quantitative (e.g., surveys, quizzes) and qualitative (e.g., interviews, focus groups) methods to gather feedback on learner satisfaction, engagement, and knowledge retention. Consider using pre- and post-tests to measure learning progress. Analyze data using statistical tools and feedback analysis techniques to identify trends, areas for improvement, and the overall effectiveness of the learning experience. Iterate based on insights gained to continuously enhance learning outcomes.

While Kirkpatrick's model is the gold standard, evaluation often occurs post-program creation, limiting realistic attainment to levels 1-2. For levels 3-4, as learning professionals, we must start with desired results. We must identify necessary behaviors, and address attitudes, knowledge, and skills in participants. Beyond participant analysis, prep work for managers, tools, and the environment is crucial. Training's success hinges on building a partnership with stakeholders (team leads, managers, leaders etc.), emphasizing a collaborative partnership for optimal outcomes, not just delivering programs.

Understanding Kirkpatrick's Model: A Four-Level Framework Developed by Donald Kirkpatrick, this model outlines four levels of evaluation, each building upon the previous one: Level 1: Reaction - Gauges learner satisfaction with the training program. Did they find it engaging? Level 2: Learning - Measures the acquisition of knowledge and skills. Did they learn what they needed to know? Level 3: Behavior - Assesses whether learners apply their knowledge and skills on the job. Are they using what they learned? Level 4: Results - Evaluates the impact of the training on organizational goals. Did the training improve performance?

If you want to create truly impactful learning activities, you need to start with the right goal in mind. Get to the bottom of the problem at hand, what employees need to do, and why they're not doing it before you consider what solution will be a fit.

Level 1 & 2 Questionnaires: Crafting Compelling Inquiries Developing effective questionnaires is key to gathering valuable feedback. Level 1: Reaction Action Verbs: Identify, rate, choose Sample Questions: Identify your preferred learning format (e.g., classroom, simulation). Rate the clarity and organization of the training materials on a scale of 1 to 5. Choose the aspects of the training you found most helpful. Level 2: Learning Action Verbs: Define, explain, demonstrate Sample Questions: Define the key concepts covered in the training on aircraft systems. Explain the steps involved in performing a pre-flight inspection. Demonstrate your understanding of emergency procedures through a simulation exercise.

I love to use activities that aren't obvious questionnaires or assessments. I find that when I blur the lines between training activity and assessment, the results are more impactful. When it comes time to test the learner's knowledge, the questionnaire will feel very similar to the practice activities they have done throughout the training module, but without feedback and multiple chances to get it right.

Delivering Level 1 & 2 Questionnaires: Optimal Timing Matters The timing of your questionnaires can significantly impact the data you collect. Here's when to deploy them: Level 1: Distribute a short survey immediately after the training to capture fresh reactions. Level 2: Implement a knowledge-based assessment shortly after the training to gauge immediate learning and a follow-up assessment after a period of time to assess knowledge retention.

Mixed methodologies work here. Analyse quantitative data to identify trends, averages, and areas of strength and weakness. Use qualitative feedback to provide context and insights into the numbers. Present findings in a clear and actionable manner, highlighting areas for improvement, success stories, and recommendations for future training. Use the insights gained from the evaluation process to refine training objectives, content, delivery methods, and follow-up practices. This should be an ongoing cycle of improvement, each training iteration informed by the previous one's evaluations.

Oftentimes unbiased feedback is very hard to obtain, since many people are either too busy with other topics or do not dare to tell the truth for other reasons (in particular if there is a perceived risk of the feedback not being anonymous). So take good care in designing your surveys. One recent example that I liked involved asking half of the group slightly different questions than the other. People were asked to state with how many of the listed statements they would agree (most of them very straightforward answers) - one of the groups received an additional more daring feedback option (e.g. "the training was entirely useless"). By comparing the average of both groups, one could derive what the group really thought of the training.

Collecting & Analyzing Feedback: Gleaning Insights from Data Once the feedback is collected, it's time to analyze the information. Here are some tips: Utilize online survey tools that automatically generate reports and charts, making data analysis more efficient. Look for trends in responses. Are there specific training aspects requiring improvement? Review open-ended questions for deeper insights into learner experience.

You can get a baseline of what an employee has learned from assessments, but the real measure of impact comes much letter when they're back on the job.

Using Feedback for Improvement: Taking Action for Enhanced Learning Utilizing feedback to elevate training programs. Address learner dissatisfaction. If a specific training format doesn't resonate, consider incorporating alternative options. Refine content based on knowledge gaps. Identify areas where learners struggle and revise training materials or add targeted practice exercises. Develop a continuous improvement cycle. Regularly evaluate your training programs and incorporate feedback into future iterations.