The True Cost of Not Conducting a Learner Analysis: A Case Study | Part 1 of 2

Can conducting a learner analysis save time and money??

That’s the question we are trying to answer in this 2-part series. In part 1, I’ll present a case study from one of my clients, who we will call “Company A.” Company A has approximately 2,000 employees. Each year, 1,000 or more employees are required to complete an internal policy and process training.?

Due to a tight timeline, Company A made the decision not to conduct a learner analysis when developing this new online policy and process training. Instead, their strategy was to take existing instructor-led training materials and use this as the foundation for the training. Since the ILT version of this training had been active for a year, stakeholders felt this was a simple way to “fast-track” the project.?

Was the company able to save time and money by using the existing ILT training materials and skipping the learner analysis??

Let’s find out.?

Disclaimer:?

While I am not disclosing the name, the information presented is from an actual client. The information has been gathered as part of an overall analysis of the training program and has been published with their permission.?

Training Profile

The purpose of this project was to provide employees with an understanding of internal policies and procedures. The purpose of these policies and procedures is to keep employees from,?

1. Using products that do not align with the company standards and requirements; and?
2. Distributing inaccurate information or content.?

The overall objective of the training is to increase the Customer Satisfaction Score by 3%, therefore reducing the customer service budget by $66,000 annually.?

When these objectives are not met, the company sees an increase in customer complaints and this leads to a reduction in sales. Sometimes, customer complaints can evolve into lawsuits. Therefore, the key performance indicators (KPIs) used to measure this training program included:?

1. The number of customer complaints recorded by the sales team that are related to the distribution of inaccurate information or content.?
2. The impact to the Customer Satisfaction Score due to the distribution of inaccurate information or content.?
3. Information from the legal department on the number of complaints or legal actions that derive from the distribution of inaccurate information or content.?

It was determined that the new policy and procedure training would be offered as eLearning, and would consist of:?

• Content delivered as Storyline 360/SCORM packages?
• Voiceover narration
• Closed captions
• Minimum interaction?
• One assessment (in the form of a quiz)?

The design team had 3 months to complete the design process and included stakeholder reviews. To save time, the design team did not complete a learner analysis. Stakeholders and the design team decided to use the ILT PowerPoint slides that had been used the previous year.?

Project Requirements

• Take the existing ILT content and turn it into eLearning.?
• Reduce the amount of training time from 18 hours to 3-4 hours total.?
• Make resources more readily available (on-demand).?
• Educate all employees on the internal policy and processes of the organization.?

Project Constraints?

• The team was given 3 months to complete the design and development of the eLearning.?
• The decision was made to remove the activities and interactions to reduce training time.?
• Due to scheduling conflicts and a leadership change, the design team had limited access to Subject Matter Experts (SMEs).?
• The PowerPoint slides from the ILT did not contain complete talking points.?
• The review process and expectations were not made clear in the beginning.?

Training Outcomes

Training Version 1

• Final training time: 3.5 hours.?
• Training development for the first version took 3 months.?
• Resources:?1 Designer, 1 eLearning Developer, and 1 Project Manager
• 6 Subject Matter Experts (SMEs)

Outcome:

• Training did not pass the pilot/testing phase and did not launch.?
• Stakeholders felt that the training information contained errors and outdated information. Training did not address the same problems related to the customer complaints.?

Training Version 2?

• Final training time: 3.5 hours.?
• Second version of training included an update to the design and the content.?
• Closed captions were added.?
• Resources:?1 Designer, 1 eLearning Developer, and 1 Project Manager
• 6 SMEs??

Outcome:?

• Training version 2 launched and then retired after 3 months.?
• Learner evaluation surveys showed low engagement and low completion rates. Many comments included references to incorrect information.?

Training Version 3?

• Final training time: 3.5 hours.?
• New SMEs were brought into the training development.
• Only 2 weeks were given for updates.?
• Changes had to be done within a short time frame and so additional developers were brought in to meet the timeline.?
• Resources:?1 Designer, 3 eLearning Developers, and 1 Project Manager
• 6 SMEs (Due to a leadership change, 2 left and 2 new SMEs were added)

Outcome:?

• Training version 3 was launched and retired after 6 months.?
• Learner evaluation surveys showed low engagement and low completion rates. In the feedback, learners shared that the course was too long and did not relate to day-to-day activities.?

Training Version 4?

• Note: Stakeholders decided to target a specific learner, but did not change the content.??
• Training was shortened from 3.5 hours to 2.5 hours.?
• Resources:?1 Designer, 1 eLearning Developer, and 1 Project Manager
• 7 SMEs??

Outcome:?

• Training launched.?
• Learner evaluation surveys continue to show low engagement and low completion rates. The change in audience only led to more frustration and multiple comments asking, “Why do I have to take this training?”?
• An analysis of the training and process is currently being conducted. Version 4 of the training will be retired after only a year.?

Calculating ROI of the Training

Here’s how I calculated the return on investment (ROI) of this training.?

First, we had to find the Total Monetary Benefits of Training. For Company A, this was the amount of money that we were trying to reduce from the customer service budget. Next, we had to find the Total Cost of Training, as well as the total amount of time spent on creating the training. Once we had these numbers, the ROI could be calculated.?

Total Monetary Benefits of Training

To recap, the KPIs for this program included:?

1. The number of customer complaints recorded by the sales team that are related to the distribution of inaccurate information or content.?
2. The impact to the Customer Satisfaction Score due to the distribution of inaccurate information or content.?
3. Information from the legal department on the number of complaints or legal actions that derive from the distribution of inaccurate information or content.?

After working with the client to collect this data, we found that over one year of the eLearning course being implemented that Company A received slightly more customer complaints and dissatisfied survey results. However, the legal team did not report any new complaints or legal action over the course of the year. We found that customer complaints and negative survey feedback increased by 1%. The overall objective of the training was to increase the Customer Satisfaction Survey by 3% and reduce the customer service budget by $66,000. Since the Customer Satisfaction Score increased by 1% the customer service cost increased, therefore adding an additional $22,000 in costs to the customer service budget. The goal of the training was to reach a Customer Satisfaction Score of 80% and reduce costs. To achieve this goal, the total monetary benefit of the training would now have to yield a savings of $88,000 annually.?

Total Cost of Training

The total resource costs includes the designer, eLearning developer, and project manager rates.?

• The project manager spent 5 hours a week on the project.?
• The design team each spent 40 hours a week for every week they worked on the project.?
• SMEs were employees and therefore their rates are not included in the resource costs.?
• Total time spent by the resources and SMEs were calculated separately.?

Total Cost of Training (4 Versions): $138,650

Total Design/Development Team Time: 1,770 Hours?

Total SME Time: 196 Hours?

Return on Investment (ROI)

To determine the ROI of the training, the following formula was used:?

ROI = (Total monetary benefits of training - Total cost of training) / Total cost of training x 100%

Total Monetary Benefits of Training: $88,000

Total Cost of Training: $138,650?

Total Amount of Time Estimated (Resources + SMEs): 1,966

Return on Investment: -37%

Was this training effective??

With a ROI of -37% this training would not be considered effective. The learner feedback surveys consistently provided data that showed that the training was not helping learners. However, being able to show the time and money spent on the training reinforces that this training is not helping the company achieve its goals.??

Would a learner analysis have saved time and money??

Yes! While there are many parts of the training process I would have done differently, my primary change would have been to conduct a Learner Analysis. Taking this step would help us identity:?

1. The audience for this training.?
2. The problem that they needed solved around understanding company policies and procedures.?

This information would then be used to build learner personas to determine the content for the training. Also, the stakeholder and review team could have included potential learners, in addition to the SMEs. This would have helped with the decision: “should we use the existing ILT training materials for this training?”

Next month we will visit part 2 of this case study in which we will review a similar training to compare the cost and amount of time spent on training after a learner analysis was conducted.?