Trend Watch Vol 2: Measure What Matters

“What gets measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know.” Jerry Muller, Tyranny of Metrics

When we’re measuring success, when we are seeking to tell the story of program performance or dreaming into our next innovation, we turn to data.

But…

Data, metrics, and learning analytics are a hot topic in the education trendscape, particularly now that technology is getting better at collecting and reporting on things from larger data sets making even more pie charts possible. But while leaders and stakeholders may be romanced by more numbers and more dashboards, in this Trendwatch I’m going to challenge you to ask why and give you a few tools as well so you can craft a metrics plan that makes a difference for your learning portfolio.

More data, more metrics, more dashboards and even more collecting, interpreting and reporting on this data can be a real problem. Adding AI to this mix (once it’s accurate and reliable to do so) will kick open the double doors to more data than you can imagine. But YOU define meaning.

Here’s my take: Collect only the data you need to make better decisions.

This assumes:

1. We are collecting data to answer key questions (not just what’s available to collect). This means we have articulated our key questions and that they can be measured.
2. We are accessing data insights in time to make better decisions. This could be course correcting, challenging or affirming assumptions, telling a true story of our journey toward a success indicator, milestone, or end goal.

So when a client asks me what questions should be on a program evaluation, I ask:

1. What do you need to know from your program evaluation that will help you make better decisions? (Which means we have to articulate those key questions)
2. How do the questions you currently ask contribute to #1?
3. What other questions might we need to ask to obtain the information you need?

An example.

If our key question is: Is this program effective (aka, has learning happened)? Then a Likert scale on speaker satisfaction does not answer this question. What’s being measured here is an approximate “liking” snapshot, not whether the program was effective. Likerts are troublesome to begin with, especially when the scale is presented with no defining terms for what each number means leaving it entirely open to interpretation and the mood of the moment, so you cannot in good faith with any confidence extrapolate a data story from an average pen score. It doesn’t measure learning and it doesn’t measure speaker performance either. Think about it. A participant can dislike the presenter and still walk away from the session with a valuable learning that changes the way they do things. If you are intending to measure presenter performance, let’s have a conversation about what that looks like so we can select a measure that actually tells that story.

The real conversation around data that’s happening in L&D is ditching vanity metrics in favor of measures that matter. In the corporate training sector, it’s a shift from vanity metrics to business metrics. Reinterpret for your sector.

Example: Instead of surveying satisfaction, measure progress toward closing a skill gap. Instead of measuring courses completed, measure the number of new skills learned (or better yet, applied).

Not all of these are appropriate metrics for continuing education when you’re not able to measure in the context of application. I get it. But do you see how the shift from what’s easy to collect toward what is meaningful to track, while challenging, is closer to the truth of what we need to actually know?

Trendwatch Challenge: How can we better align what we measure with decisions we are making?

Side note: This convo is part of my Accelerator consulting intensive, and we get into the nitty gritty of what you need to know to make better decisions for your learning portfolio.

Allow me to leave you with some inspiration from Accelerator. The education strategies I co-create with clients have success measures on three levels: Strategy, Portfolio Performance, and Learning. Here’s some of what we talk about that you can take to your whiteboarding sesh with your team and refine your metrics plan. But again – take only what applies to the key questions you are asking.

Strategy

QUESTION: What are our strategic goals for our learning portfolio?

Sample Measures:

• Revenue: sales, units, refunds, profit/loss, market share, buying cycles
• Registration: first-time registration, return registration, unique registrants
• Controlled Costs: staff time, technology expenses, honoraria, etc.

ACTION: Define your strategic objectives and then measure your progress meeting them.

Portfolio Performance

QUESTION: What are our expectations of optimally executed learning programs?

Sample Measures:

• Customer satisfaction: reaction ratings, customer service call spikes, flagged issues, net promotor score
• Development: delivering on schedule, noting where courses may be getting hung up in the development process, quality standards met, amount of time anticipated over amount of time actuals to develop programs, volunteer hours contributed
• Utilization: desired targets are participating, course completion rates
• Customer Service: Number of issues logged, number of issues resolved, amount of time resolving issues, etc.

ACTION: Define what a high-functioning learning portfolio looks like and measure your progress toward that target.

Learning

QUESTION: Did learning happen?

Sample Measures:

• Learning objectives: measure the course delivered upon its promises
• Confidence: ask how confident participants are that they can apply what they learned in practice
• Commitment: ask whether learners intend to apply what they learned in practice
• Behavior: measure whether change happened – was the learning actually applied? Is deepened knowledge, skill, and competence evident? Were outcomes achieved?

ACTION: Define the expected learning outcomes and then design measures that will evaluate success – learning, behavior change, results.

"Training is not our product. Results are our product. None of us joined L&D to develop training modules that waste people’s time. We are here to make a difference – to develop effective employees and high-functioning teams.” Brandon Burtner

AI is changing what we can measure, how and when. Take this moment in time to get clear on what you need to know to make better decisions. Then survey the tools available (and they will rapidly change over the coming year) to help you assess the intelligence supporting your query.

Let me know how this Trendwatch shifts how you think about your learning portfolio metrics.

Bonus Tip: Leave room in your dashboard for innovation. Don’t crowd out creativity. When we are innovating, we are iterating and refining. It’s not a straight line to the finish ribbon. There’s trial and error. There’s redirecting efforts because data is correcting our assumptions or adding new vital info we need to move on. Rigid metric expectations lead to rigid performance toward metrics. New ideas can’t breathe there because the risk of failure and the resulting admonishment doesn’t feel worth it. (If this is an interesting topic to you, we can dip in a little deeper in another post – let me know.)

Tracy King, MA, CAE

As Chief Learning Strategist & CEO of InspirEd, Tracy King leverages more than 20 years in workforce development consulting with organizations on education strategy and learning design. Tracy is the author of the award-winning book?Competitive Advantage, and they advise on how to grow reliably profitable and sustainable continuing education programs that transform learners. Tracy specializes in the intersection of learning science and technology. They are a thought leader, master learning designer, trainer-facilitator, coach and DELP Scholar. Their work has been featured on NBC, ABC, FOX, USA Today, Forbes, The Star Tribute and hundreds of nationally syndicated television, newspaper, and magazine outlets. For more information, please visit them online at www.inspired-ed.com?