WHERE IS MY ROI? - measuring training returns

Author : Manik Senanayake

WHERE IS MY ROI? ?MEASURING LEADERSHIP TRAINING EFFECTIVENESS

Disrupting the routine commitments and taking decision maker level leaders from their daily role responsibilities to attend an executive leadership training program is a costly proposition. Before anything they need to be convinced that time away from the desk is worth a while for the betterment of self as well for the organization. Hence it is fair to assume the only convincing remains tangible evidence of bottom-line business impact. At a granular level several conscious decisions have to be made by the stakeholders on ROI. Will there be quantifiable results to show in the form of either “revenue-generation or cost-saving benefits? will the entire training experience outweigh the training investment? In other words, is the leadership training really worth it?

?What’s in a measure?????????????????????????????????????????????????????????????????????????

Measurement challenges are two-fold. Firstly, the inferences made on program effectiveness vary drastically according to who is being asked (e.g., peers, manager-grade, subordinates, C-suite cadre); hence, efficacy may be in the eye of the beholder. This is a universally apparent across markets and culture. Secondly, even within same hierarchical pay grades, individual differences for (e.g., IQ, EQ, personality, character, perspectives, values, life experiences); influence program evaluations. There is a need to consider how individual characteristics of leaders’ impact measurement indicators.

?It has been over 60-plus years since Donald Kirkpatrick published his landmark dissertation on the four levels of learning evaluation which provide some context to measuring the effectiveness of training. Yet the debate is still very much alive about our ability to measure the impact of leadership development training. It’s far less straight forward to assess things with a direct result like reduction in error rate in manufacturing plant, campaign product picks up rate, call resolution efficiency in a call center and most commonly sales performances. It’s a matter of selecting the right metric and review pre and post training to determine the incremental impact. Kirkpatrick/Phillips Model is made up of four levels:

1.???????Reaction?- This includes thoughts and feelings on the program, the moderator, the venue, the content, etc. Whether they find the training favorable, engaging and relevant to their jobs.

2.??????Knowledge?– this is the intended knowledge, skills, attitude, confidence and commitment gained by participating in training. This is often measured via knowledge tests and via hands-on performance simulations.

3.??????Behavior 360-degree?– through behavior assessment and observation it measures the application of learnt leadership skills on subordinates, stakeholders and that of direct reports, supervisors have on their teams. This involves any visible transformations in the perception of style of management, engagement etc.

4.??????Result and ROI?– here assessments measure changes in the productivity of the participant by the supervisors, trainees, and direct reports, and how they benefited the entire company. This can be any KPIs which the learning tries to influence, from error rates, sales revenue, time saved to cost saved

?There were few other noteworthy models emerged on evaluation measure of LDPs. Such as Phillips’s ROI model introduced two additional levels to Kirkpatrick model namely Data which constitute input such as training hours, investment etc., while second level, ROI measured monetary outputs. Together with this further empiric evidenced based proposals are assessed to convert to monetary values. The High-Performance Measurement impact model developed by Brandon Hall differentiates between another four levels of tactical, functional, operational and strategic which integrates diverse units of business modules for review. Josh Bersin, another well-known analys, compliments these models with a similar one.

?Research suggests only 24 percent of organizations conduct some form of impact measurement. This often is a post program survey where feedback of the participants treated as an absolute conclusion. Hence subsequent most decisions are made on this which is wrong in so many levels. Firstly, participants irrespective of their level they are not qualified to conduct a comprehensive review on program’s all deliverables. This is because their feedback is just a mere expression of how they feel which often involves their likeability of the resource person, his style, demeanor or even use of what type of examples etc. These influences could be so diverse and subjective it may not have any direct relevance to program content or objectives set for the program. Here one of the biggest errors is when entertainment value meaning if the audience have had a fun time and enjoyed sessions well, often these experiences get high ratings and concluded as successful. At times despite lack of substance nor any fresh knowledge or skills acquired, inexperienced trainers use this tactic to deceive the audience by manufacturing false perceptions. Evidence indicates no strong correlation between program effectiveness and participation enjoyment. Beside when it comes to especially leadership training the impact needs to be sustainable. If not transformational, then incremental changes expected from participants need to sustain beyond the program day. Assuming the organizational environment set-up to facilitate such changes, the real impact needs to be monitored over a period of time.

Purpose definition & outcomes

As an executive leadership coach what I have experienced is that the purpose set for leadership development programs often not clearly defined. As with many companies the intend emerges not by need but as a routine process in human resource policy to conduct programs. This allows for subjectivity and confusion to set in when it comes to program evaluation and measurement. Often when program purpose is not established the expectations from the client becomes a wish list, an assortment of deliverables which not only doesn’t make sense, but almost impossible to achieve. Establishing program purpose is therefore imperative if we are to device a good assessment matrix to determine investment’s ROI. So, let’s revisit what constitutes a leadership development program and more significantly what you should hope to achieve? fundamentally all leadership development learning falls in two strategics outcomes - driving performance, or preparing for the future. Here stakeholders need to understand the strategy the solution supports, then only they’ll understand what type of metrics to choose to assess. Performance-based leadership development will support performance-based results, and pipeline-based leader development should support talent metrics.

Few key considerations need to be mindful of. Firstly, you need to agree on impact metrics before build the program dynamics. Most often we follow a reactive approach in that we first design and build the learning experience, roll it out, then reactively trying attempt to show value through measurement. Measurement needs to be part of the proposition at the inception and linked to the program objective(s). Secondly, synergy needs to prevail between company’s strategic intent and program philosophy and structure, as such it is reflected in the program content. Especially involving C-suite senior leaderships level or sector heads where constructive discussions need to be held on what organizational issues need solving. This could be either at department level or at more macro level for example strategic direction etc. This way participants are asked to apply their learning to solving a real-world critical business challenge. The results demonstrate the tangible impact the learning has on the organization. ?Thirdly, Shoot for the ‘impact’ instead of proof. With so many variables it’s difficult to directly prove leadership learning is the root cause for any measurable improvement. However, impact study supports strong correlations exist between the learning and the result much more than stakeholders expect. In practice though, we’re usually much more concerned about “proof”?than our stakeholders.

Some sanity to madness

So, how do we measure training ROI? Kirkpatrick/Phillips’ model considered to date one of the useful tools developed to provide some level of logic to measure effectiveness. As such it makes sense to understand its’ frame-work logic. The approach is based on six levels where it measures success at every level and milestone of the trainee’s exposure. At level.1 it involves at a very macro level overall did the participants like it? Were participants engaged with content, facilitators and overall delivery of the training experience? Level.2 here evaluation becomes more specific as its question is there any new knowledge learnt? If so, what are those knowledge and skills? Level.3 explores whether subjects are doing anything differently, more effective and efficient? If it is behavioral change what transformations apparent? Level.4 attempts to assess the correlation between the learning and impact on the business. Here the influence performance metrics as increase revenue or create cost saving efficiencies are measured against improved behaviors on the job. At Level.5, it will compare investment for program against the net benefits achieved to determine the worth of the leadership training. And at final Level.6 it was added to determine what aspect of the working environment or climate either conducive or a hindrance to the application of acquired leadership principles thereby directly effecting ROI.?

?There are no “silver bullets” offered on measurement

Even accepted methodologies as the Kirkpatrick evaluation model is not without its limitations as significant traps exist. Leadership development today is not direct as it transcends a linear focus on leadership “skills” (horizontal development) and involves a certain leadership mindset (vertical development). This combination can create a substantial uplift in return on learning, however itself is more complex to measure.?How do you measure impacts over time and at multiple levels of the organization? Here the assessment itself need to be carefully structured and timing too needs to get it right. These assessments should include specifics of skill and knowledge application on the job and those projects which expose such competencies need to be allocated for the participant. As far as timing of assessments ideally it should be during periods of real business challenges and should last over a period of several months. Thus, it’s important that hold departments accountable to sharing whatever results positive or negative so that it could be feedback loop for everyone’s learning

?Patience is a virtue

Organization climate needs to be leadership ready. Meaning learning mentalities need to be part of the DNA of what drives the company. The environment needs to facilitate new leadership behaviors where freedom to fail is allowed as a part of cultivating experimental leadership mindset. Contrastingly effective leadership development interventions can significantly influence culture to instill a progressive behavior which could lead to sustainable growth. Given multi-faceted nature of leadership cover, the measure of ROI therefore represents a learning journey involving times of resistance and disorientation before level of leadership proficiencies acquired are exposed. Hence measurement needs to be over a considerable period of time, as such ROI is stretched over time. If right leadership philosophies are embraced through training, then leaders are encouraged to become more adaptive and to change the ways they look at leadership and the world. ?As scale of organizational challenges vary such as global financial crisis or covid 19 related disruptions, participants will be put test, their intrinsic leadership competencies and mentalities years after the training has concluded. With only 18 percent [of businesses] are gathering relevant business impact metrics, which may be why some experts have questioned the worth of these programs’ ROI.

Short term validation?

That being said, there are some?metrics?and?questions that can be utilized to see what the impact is in the short-term for ROI, as well: These include examining duration for promotions of participants and for what performance related reasons. This demonstrates application of skills acquired during training. Retention rates is another indicator, how many of the participants stay longer vis-a vis non participants. Assessment of participants’ creativity, proactive value contributions to company profitability. The efficiency levels of those under subordinates under their management, their mentor capabilities, strategic view for their SBU beyond just routine operational commitments. To do this pre and post assessments checks need to be maintained to keep a pulse on how well someone is developing around their leadership skills is a must and can elevate the productivity, velocity and overall success of a team and/or organization.?

Despite the subjectivity of leadership training measure, no one can doubt the positive impact it generates; as evidenced by numerous researches conducted on the correlation between the two variables L&D and ROI. Besides training promotes employee satisfaction as it leads to high loyalty and low labor turnover rates. Further it also makes them less rely on compensation and other benefits. Empirical evidence shows training helps employees develop a positive attitude which help shape a learning and performance driven culture. In the long run any organization with strategic intend to upskill and develop its human capital couldn’t do any harm as absolute value created by L&D clearly outweigh any investment. It’s imperative that before one worry about program measure that you contract competent trainers which not only well verse with leadership philosophies but also possess hands-on multi-industry experience (as oppose to story tellers reading of a script) so that right balance of theory and real-life challenges are combined. With right trainer competencies, purpose based pre-designed metrices, the possibilities for success are infinite for organizations in terms of revenue growth, bottom line improvements with right leadership development program..