DEVELOPMENTAL EVALUATION

THE IMPORTANCE OF DEVELOPMENTAL EVALUATION IN A CHANGING ENERGY LANDSCAPE

By Katie Cary,

In this post, I discuss how evaluators and consultants can help utilities navigate this new landscape by using developmental evaluation to stay agile, innovate, and adapt programs as they are being implemented.

But first, what is “developmental evaluation”?

Developmental evaluation is an evaluation approach coined by Michael Quinn Patton that supports the development of innovations occurring in dynamic, complex environments, where knowing “what to do to solve problems is uncertain and key stakeholders are in conflict about how to proceed”. Within the utility industry, evaluators often perform prescriptive, summative evaluations after programs have been running for a few years, with the goal of assessing whether it worked after the fact. Development evaluation, by contrast is about adapting the program as it is being run.

Comparison of Summative versus Developmental Evaluations

Developed and presented by Katie Cary at the 2018 AESP Summer Conference.

Hot Tip:

When faced with uncertain futures, incorporate these key elements of developmental evaluation into your evaluation design:

#1. Incorporate evaluation early. In an uncertain environment, you cannot afford to wait to see if your intervention is working. Build staged evaluations into the program design – it will pay dividends in avoided costs of ineffective programs.

Tip in Action: A program we evaluated was giving out free equipment to customers, and our evaluation found one that customers did not use. The program was able to remove this measure before it was sent to thousands of customers with no impact on customer satisfaction. 

#2. Be flexible in your approaches. In some evaluations, approaches are designed far in advance of the work being completed. For developmental evaluations, evaluators must be able to adapt their approaches to changing program designs and stakeholder needs.

#3. Iterate and adapt. In evaluating an ever-changing program, evaluators need to evaluate frequently as changes are made, building on past findings and recommendations.

#4. Don’t be a stranger. To be effective at developmental evaluation, evaluators cannot be kept at an arms-length from program implementation staff; instead, evaluators should be trusted members of the team, abreast of any program changes or shifts in the program’s goals.

Katie Scary is a senior consultant with EMI Consulting, an evaluation and consulting firm based in Seattle, Washington that focuses on energy efficiency and renewable energy programs and policies. Her firm predominantly works as evaluators and research partners for utility clients.