Quantifying Learning Impact: Long-term Lessons Learned Organizational Influence

In my past two articles I wrote about learning impact and how it might be addressed. As previously mentioned, impact is a tricky thing to speak about because it happens in stages. The first article discussed initial impact, the second one on midterm impact. They can be found here: Initial Impact and Midterm Impact.

The purpose of this article will be about long-term impact of a training class or program. In this case, we are looking somewhere beyond 18 months after the training. When looking at long-term impact, a few items we should be concerning ourselves include:

• Measuring the audience’s competency on the subject matter
• Getting feedback from them regarding how the learning session impacted their capabilities
• How the learning experience affects their career development
• How they are working with their teams and beyond.

Measuring Competency

I am starting here because I believe having skillset competency information is vital to understanding how a learning program is directly influencing a business. Competency measurements should be completed on a periodic basis so we can capture baseline readings and appropriately measure skill development progress over a period. Best practice is to complete a competency assessment biannually if possible but at least annually.

Competency measurements show how different individuals and groups are progressing in their skills. We will be able to gather important information that will help to determine gaps in the employees understanding of differing subjects. These gaps can then be used to plan future learning events. Competency measurements gives us the benefit of knowing whether current content is providing adequate training for learners. When using a well-developed Competency Management System (CMS), we can use organizational heat matrix models to see the impact of our programs. It provides the opportunity to find holes in the organization’s skillsets.

If we are not seeing appropriate skills advancement after learning programs, the competency assessment will also identify a starting point for an investigation. Before deciding on a path of action, the investigation followed with proper analysis will help to determine why the gaps are present. Action could include but are not limited to running an entry level event again if we see low levels of competency. If we see some competency, but not at the levels we might expect, a Q&A session with an expert, or a workshop might be appropriate.

Competency Management Systems can also help to see how employees are progressing under each organizations team leader. We compare how everyone is progressing and can see if team members need to be rotated as projects wrap up. Skill progression is conditional and while it might show one team developing faster than others, we need to appropriately investigate the reasons before making any conclusions.

If one team is developing the best new leaders, we should investigate as to why. Is this person doing something different than others? Are their projects more challenging or using newer technology because contractually they have been allowed to do so? Then it might not be something they are doing specifically. However, if they are more invested in making sure everyone on the team is properly supported and work to keep everyone in the loop with the technologies they are deploying, that is something to celebrate. That person can then be considered as a potential person to fast track in their career advancement.

If someone’s team is not developing essential skills, then it will allow us a chance to take a deeper look to determine why this is occurring? Are they being assigned the most difficult projects? Are they on time with their work? Is there a compelling reason why skills development is not occurring for these team members? If the answer is, yes? Then it might be appropriate to rotate team members on and off this team as the project manager is needing to complete work quickly. This type of project completion rate requires employees with talents best suited to completing work quickly to singular tasks. While using this strategy on a short-term basis is acceptable, long-term employee satisfaction will wane if required to complete projects without a possibility to develop themselves. Team rotation is one way to combat this and still get certain projects with quick turnaround completed.

However, if the individual team leader is completing work at the same rate as a team leader that is also developing their people, we really need to better understand why? Does this team leader need leadership training or might have been misidentified for a leadership role. While being a strong performer is typical of career advancement, not everyone is suited for leadership and these strong performers might be better suited for roles in R&D or as project advisors. Before making any decisions, a proper investigation is necessary because we want our people to develop so they can take on additional tasks so the organization can complete projects at a rate and quality that will keep our clients returning.

Feedback

It is at this point that we as course designers and managers, should be speaking with the learners again to understand how our course affected their abilities. In these feedback sessions, we need to get a better understanding of how effective our courses were in their skills development.

In a perfect world, the lessons learned from their course work should be a part of the learner’s inherent vocabulary. They should be able to not only recite what was learned, but share that knowledge with others as a mentor, because they should have used that knowledge in their work. If the answer is no, then we also must determine why not? Failure or lack of opportunity?

Gaining usable feedback from our learners should be able to disseminate quantifiable information on what we got right and what we missed during their courses. Questions we should include in the feedback session are.

Did the learning course give you enough information to complete your tasks? Did the learning course give you enough theoretical understanding of the work processes. Did the learning course cover resource management? What did you have to learn outside of the class? Could these steps be covered in the learning class or were they project specific? Are there tips and tricks you have discovered on your own to improve the workflow? Are there any lessons you learned that can help others?

Using the previously recorded competency skills assessments will help us to understand the feedback we receive. The feedback will vary from individual to individual. Each person learns at a different level and each person has been tasked with using their knowledge in differing ways. Some might be leading projects, some might be supporting them, some might never be asked to use their knowledge. We are not responsible for what they have been asked to do in their jobs. As an interviewer, asking about what tasks they do routinely in their jobs as a part of the feedback session will help to calibrate the information received.

If an individual has not been assigned work that directly utilizes a training course, but has been given other tasks, their feedback is as important as those individuals that are using those skillsets. Maybe that skill has not be required of them yet, but they will need to use it soon. It might help us understand if we need to follow up with them and make sure they are ready at a future date.

They might be upset because they want to use the skill they learned and were not given an opportunity to use it. This could cause them to give a previous training course lower feedback mark. It would be understandable because they might blame the learning instead of understanding that the time is not right yet to use that skillset. It would be a great opportunity to give feedback to both the team leader and their managers and let them know they want to use their new skills.

New skills are vital to an individual, the team and the organizations capabilities looking towards the future as employee happiness helps to create a positive environment. It also shows the organization is committed to employee career development.

Career Development

We have evaluated employees’ competencies; we have received useful feedback. Now it is time to analyze these data points. Analyzing this data will help us to understand the gaps and feedback will reinforce the importance of specific skillsets needed to complete work. We also need feedback from management to determine what skillsets will be required based on future work. We should look at both present project work being completed as well as keeping an interest in what R&D is developing. By understanding both present and future technology, it helps us understand what potential skills might be needed to develop so our team is ready for the new challenges.

From here we create our strategy to deploy future learning aimed at career development. As a part of this initiative, it is important to remember to receive approval from the executive team that our plans are aligned with the goals of the organization. By doing so, we put everyone in the best path towards success. Strategy aimed at utilizing our known resources, creating a plan to develop new programs and courses as well as identifying the necessary SME resources to complete our strategic objectives.

Once we have these items identified, we can look to see if our present staff can complete these goals, or if acquiring new staff might be needed. When stretching our capabilities, it makes sense that hiring new personnel might be required as internal staff might not have the skills necessary to complete newer processes. Hiring outside talent that can synergize with staff makes sense as it immediately fills gaps. If there are long term concerns for hiring new employees, but the organization requires a specific skillset immediately, use of third-party vendors and contractors can help to minimize long term exposure to cost increases beyond the necessary period while working to develop internal personnel.

As these courses are rolled out and are completed, we should also watch for those people that have become peak performers as they might be able to deliver courses in the future as new SME’s. The new SME’s do not need to be teachers to share their experience. They can help by giving short reviews of their work in a workshop/symposium environment. They can participate also panels discussions to give their opinions or answer audience questions. Over time, these individuals might feel comfortable enough to help teach others. We should impress upon our new SME’s that sharing their experience helps their Career Development by improving their communication skills. The ability to speak in front of an audience of peers is one of the most important skills to develop and it requires multiple engagements to hone.

Continuous Improvement by Working with Others

As we have been discussing, learning program efforts should be positively affecting the entire organization. Individuals should be helping others and sharing their experiences. With this, it is important to make sure these people are actively engaging with management and R&D groups.

Management should be vested in understanding how new knowledge is being shared and effecting the bottom line. We should see work being completed more quickly and accurately by our teams. We should see efforts toward innovating our work. As learning and development people, we need to stay engaged with our learners beyond the class by checking in with everyone. We need to continue to ask questions. We need to not be afraid to ask what didn’t work.

If these individuals are giving us pertinent information, we also need to ensure it is communicated not only with management staff, but also with our R&D department. They need to better understand how their efforts are being used. Feedback for them could be as simple as; software responsive and working as intended? Are there pain points as a user? Are there features you wish existed? Are there features you need to be better explained?

This information is vital for R&D teams. It can help to improve the system. Best example of this is how Microsoft rolls out windows. It is constantly needing feedback from the user community. Why? It is such an extensive operating system that even with quality assurance in place, there are so many possible failures that can still occur. Why? There are millions of daily users that are working on different devices and have different levels of understanding. There are going to be unintended consequences when you have that large of an audience. So, feedback is critical to continue to develop the system.

This works the same way at any organization. We need individuals to continue to contribute to the overall improvement of their respective organization. It isn’t always obvious to everyone that they are a part of the solution, and we should encourage our learners during their time in courses. We should encourage them by letting them know that they will grow themselves by continuing to give the best feedback they can.

Long term strategy for our learning programs is one of the most difficult to nail down initially. But it should be the end goal you set out from the start. It is our job to make sure to stay involved with everyone and continuously engage from start to finish. As we review class impact over time, the one common theme is staying engaged with our learners. Keep in touch with them and see how they are progressing. Continue to ask questions, ask what help they need and if they might be able to give back in the future.

If you have read this far and have anything else you would like to contribute, please do so or contact me directly. I am always happy to discuss and look forward to hearing your comments.

Thanks for reading!

Craig Findley