What is your definition of Program Evaluation?

The more I talk with people about program evaluation and what an evaluator is, the more I realize we may not be talking about the same thing. On the surface, the generalities make it appear as though we are, but when you get into the details, I'm finding some significant differences of opinion. I’ll attempt to define the words, and let’s see you if you agree or have a different perspective.?

First, let’s define Program Evaluator

A Program Evaluator is someone who has foundational knowledge of evaluation theories, methodological approaches and research designs to allow them to conduct valid and reliable evaluations.?Dr. Sebastian Lemire, PhD Social Research Methodology (Evaluation), created an Evaluation Metro Map to help visualize the different schools of thought and the unique approaches within those schools. The Metro Map provides some context to the in-depth knowledge and practice an evaluator must have to conduct an evaluation. ?

(Image 1 - Dr. Lemire's Metro Map, 2020)

In addition, an evaluator must act with the highest ethical standards. It's important for them to know how to collect targeted and valid information, perform reliable analyses and make decisions based on evidence all while managing conflicts of interests and confidentiality.?

The Credentialed Evaluator

Once an evaluator has been practicing for two years, they're eligible to apply to the Canadian Evaluation Society (CES) and earn the Credentialed Evaluator (CE) designation. The application is a rigorous process where candidates are required to show practical experience demonstrating the CES competencies, standards and code of ethics. A common theme across all CES requirements is an evaluator must act with a strong sense of honesty, integrity and live the CE values and moral principles.

It's not necessary to have the CE designation to be a program evaluator however, this designation is a sure sign of excellence in evaluative practice.

To learn more about the Credentialed Evaluator, follow this link to the CES webpage: https://evaluationcanada.ca/ce

Next, let’s define Program Evaluation

The CDC Program Performance and Evaluation Office defines the word program as “any set of organized activities supported by a set of resources to achieve a specific and intended result”. Therefor, a program can refer to a strategy, initiative, policy, project or educational program.

My definition of evaluation is it's a systematic process that determines the value, merit and worth of a program using one or more of the evaluation approaches identified in the Metro Map in Image 1. As I conducted research to write this article and validate my thoughts, I've come to realize my description of evaluation may be too narrow.

When I attempted to find the definition online, it became clear that there's no common explanation of the word evaluation, even within the evaluation community. The challenge of confirming the definition is explored in the article by Poth et al, in the June 2014 Canadian Journal of Program Evaluation, called ‘Toward a Definition of Evaluation Within the Canadian Context: Who knew This Would Be So Difficult’.?The authors describe how they conducted several studies in an attempt to define the word. The results of one of their studies done through social media was summed up in a Wordle showing the diversity of responses, see below - the larger the word, the higher their frequency in the research.?

(Image 2 - Poth et al, 2014)

In conclusion, the article explains that evaluation is one and all of these words and more. Therefor, it may be more beneficial to not have a shared formal definition as this could restrict and limit the evaluation profession. As program evaluation and its processes are full of complexities, it would be difficult to encompass the multifaceted nature and inclusivity of the profession in one simple description because it wouldn’t do it justice.??

Reviewing the article by Poth et al has helped me understand why I'm discovering various opinions on what evaluation is, and why there are significant differences when the details are discussed.

Evaluator and Evaluation Definitions

It appears as though there’s a shared and common definition of a program evaluator, but not of an evaluation.?Now that I've attempted to define the two words, do my descriptions align with yours or do you have a different perspective? Do you believe evaluation can be defined? I’d love to hear your thoughts.

Wishing you great success in your endeavors!

Lisa Langton is an evaluation and audit consultant who focuses on higher education and professional accreditations. She’s a professor in veterinary medicine who’s been designing, delivering and evaluating educational initiatives and programs for 20 years, helping organizations achieve their vision of success.?

References

Canadian Evaluation Society. Website accessed August 2021. https://evaluationcanada.ca/

Centers for Disease Control and Prevention, Program Performance and Evaluation Office (2011) ‘Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide’. Website accessed August 2021. https://www.cdc.gov/eval/guide/introduction/index.htm#anchor_1585059617

Kistler, S (2011) 'Susan Kistler on Leaving Wordle for Tagxedo'. Website accessed August 2021. https://aea365.org/blog/susan-kistler-on-leaving-wordle-for-tagxedo/

Poth et al, (2014) Canadian Journal of Program Evaluation, ‘Toward a Definition of Evaluation Within the Canadian Context: Who knew This Would Be So Difficult.’ Website accessed August 2021. https://journalhosting.ucalgary.ca/index.php/cjpe/article/download/30921/pdf