Dr. Jeff Sheldon’s annotated bibliography: accounting for exogenous variables in the relationship between evaluation outcomes and utilization

I haven’t published an academic article for the longest time, but I happened to come across this gem a few days ago, gave it a read, and after a bit of editing thought it worthy of publication, and that you might enjoy it. Articles chosen for analysis, at the time, reflected the cutting edge of thinking on evaluation utilization.  However, I think they’re as relevant today as when this was written a few years ago, especially if you’re not well-versed in the relationship between evaluation findings, how those findings are ultimately utilized by program or organization stakeholders, and the far-reaching influence of the evaluation enterprise. Consider this a primer; you can always read the most recent literature to engage in the current thinking on evaluation utilization and influence if this subject piques your interest. I think it’s pretty interesting and I hope you do too. Please feel free to contact me directly with your comments and questions, and as always, enjoy.    

Introduction

The purpose of this annotated bibliography, which offers both a summary and critique of each of six articles (in order of presentation: Kirkhart; Henry and Mark; Preskill and Torres; Russ-Eft, Atwood, and Egherman; Brett, Hill-Mead, and Wu; Henry; and Torres and Preskill) is to better define evaluation utilization (also referred to in the literature as use; use and utilization are interchangeable throughout this piece) especially in light of discussions in the field about evaluation influence (i.e., the way we think about the broader effects of an evaluation at a macro level, for example, from the learning that comes from engaging in evaluative inquiry to using findings in policy making). The subtext of this piece is to show that even though definitions of evaluation use have evolved, based on new theoretical perspectives, there a lack of research on evaluation taking into account the attitudes people hold about evaluation that either moderate or mediate evaluation use (i.e., individual factors) thereby limiting the ways in which evaluation utilization is defined. The strength of this piece is that it offers the specific perspectives on evaluation utilization and influence of noted evaluation theorists. A summary, including testable evaluation research hypotheses ends the discussion. As a disclaimer, this is neither a comprehensive picture of evaluation utilization (i.e., process-based or results-based) and influence as it discusses only a handful of proponents, nor does it bring in literature from other areas of social science research through, for example, the respective works of Weick, Rogers, and Ilgin on the influence of the collective mindset within organizations or teams which influences decision-making. As noted, the following articles were written by some of the luminaries (i.e., theorists) in the field of evaluation and reflect much of the thinking about utilization dating back to the earliest days of the profession.

Kirkhart, K. (2000). Re-conceptualizing evaluation use: An integrated theory of influence. In Caracelli, V. J., & Preskill, H. (Eds.). The expanding scope of evaluation use. New Directions for Evaluation, 88, 25-38.

In this article, Kirkhart offers an integrated theory that re-conceptualizes the inherent influence and purposes of evaluation into three dimensions – source, intention, and time—in the process raising influence up to a loftier position than in past discussions of use. Part of her argument for reframing evaluation into these dimensions is a familiar refrain; the language and semantics of use of the not so distant past have impinged on our understanding of the influence of evaluation. Kirkhart’s main argument, however, is that we misperceive the influence of evaluation itself because we tend to only consider the influences that are results-based, the traditional heart of evaluation use. Theorists have, however, entertained other viewpoints and these are what Kirkhart attempts to bridge in her integrated theory. Apparently, Kirkhart provides this framework so that previous thinking and understanding of results-based use can be put into, and examined within the broader context of influence. What Kirkhart does so well in this piece is to sample from past discussions and theories on use and unite them in her new model. Accordingly, Kirkhart claims an integrated theory will move the field of evaluation ahead by pulling in all the disparate theories, of use. In proposing her theory, Kirkhart intentionally shifts from the terminology of use to the terminology of influence, (i.e., “the capacity or power of persons or things to produce effects on others by intangible or indirect means”). Influence allows us to examine outcomes of evaluations that are “multi-directional, incremental, unintentional, and non-instrumental in combination with those that are unidirectional, episodic, intended, and instrumental.” 

What I think is missing from her integrated theory is an accounting of individual factors that play into evaluation influence. Kirkhart’s explanation of influence makes it seem that source, intention, and time as the agents of change are completely external to the individuals who the evaluation attempts to influence. By failing to take into account internal factors associated with the change process Kirkhart offers a very limited perspective on how influence actually works. To Kirkhart, process happens and findings result, and thus influence takes place, but there is no vital link between the process – findings dyad and influence in the pathway model.      

Henry, G. T., & Mark, M. M. (2003). Beyond use: understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24(3), 293 – 314.

Given the historical imprecision of the terms use and utilization, Henry and Mark offer the field of evaluation a framework to distinguish between different types of use, allowing us to see the “influence pathway” to each. Their framework combines the processes of change that are engendered by an evaluation into “causal pathways” so that use or utilization can be better defined and understood. Knowing how a particular use was arrived at helps make the definition of use much more precise. The framework operates in such a way as to be able to distinguish between mechanisms and interim outcomes of evaluation taking place on three levels: individual, interpersonal (small group), and collective (organizational). These three levels and the types of outcomes associated with each have not been separated and looked at individually in any previous research on use. Henry and Mark have organized change processes and outcomes into “outcome chains that correspond to the ways that evaluation’s consequences are linked together.” In this framework, at the individual level, “evaluation processes or findings directly cause some change in the thoughts or actions of one or more individuals, changing beliefs and opinion.” At the interpersonal level, change is persuasive and occurs in the interactions between individuals. On the third level, collective, evaluations either directly or indirectly influence the decisions and practices of organizations. 

The major limitation in this article is the depth to which Henry and Mark go in explaining how and to what extent evaluation processes or findings “directly cause some change in the thoughts or actions of one or more individuals, changing beliefs and opinion.” Similar to my criticism of Kirkhart (2000), Henry and Mark discuss the potential influences that a particular evaluation might have on an individual, but only from a perspective external to the individual. While they do acknowledge that processes and findings are part of the causal chain in the change process, they do not account for individual variation, merely describing the behavior that might result from such influence. Henry and Mark fail to take into account, in the influence – individual behavior link, the process that individual goes through so that influence can manifest itself in the form of evaluation utilization.  

Preskill, H., & Torres, R. T. (2000). The learning dimension of evaluation use. In Caracelli, V. J., & Preskill, H. (Eds.) The Expanding Scope of Evaluation Use. New Directions for Evaluation, 88, 25-38.

In this article, yet another in a series on the ever-expanding definition of evaluation use, Preskill and Torres look at the ways in which evaluation and evaluators can intentionally facilitate a learning process in organizations by involving stakeholders in the evaluation process. Unquestionably, engaging in evaluation “collaboratively, reflexively, and dialogically” impacts individuals and organizations involved in the process, but also brings into focus the changing nature of the evaluator’s role. Evaluators who see themselves as facilitators of learning will function as much as “educators, consultants, interpreters, mediators, or even emancipators as they do technical, expert-science researchers.”  Use of findings, as one of the many goals of evaluation has received a great deal of space in the literature on evaluation use, yet the learning aspect of use is now coming to the fore in both practice and theory.  According to the authors, there is an emerging interest in expanding evaluation use beyond the intended use of findings and some of the recent focuses on the learning that stems from participating in evaluation activities, positioning evaluation as one of the pathways to learning in organizations. When evaluation is collaborative, reflective, and dialogic the evaluator is creating a culture and community of evaluation practice within the organization that intentionally promotes “constructivist and transformative” learning. In the final analysis, Preskill and Torres believe evaluation has a unique opportunity to help individuals, teams, and organizations learn from past and current practices. 

While making the argument that another aim of evaluation use should be enhancing both individual and organizational learning, Preskill and Torres do not take it one step further by explaining how learning facilitates change leading to an increase in evaluation utilization. They discuss the importance of involving individual in the evaluation process, and that by this involvement it leads to learning, but this is as far as they extend their reasoning. If in fact learning takes place and is personally transformative, then that transformation should include the likelihood that the evaluation findings will be used. If learning leads to a higher valuation for evaluation, then another aspect of learning as use would naturally extend to the use of findings that result there from.  

Russ-Eft, D., Atwood, R., & Egherman, T. (2002). Use and non-use of evaluation results: case study of environmental influences in the private sector. American Journal of Evaluation, 23(1), 19-31.

This article is a case study in how recent debates on evaluation use, including the concept of process use (Patton, 1997), as cited and discussed by the authors, can, and should be extended to the private sector. Traditionally this is not a sector included in many debates on evaluation, but should be a focus of empirical research and theory generation. The evaluation on which this case study is based specifically contributes to the evaluation literature in two fundamentally important ways. First, it offers, almost as a novelty, an example of an evaluation undertaken within the for-profit sector. Russ-Eft & Preskill (2001), as cited in this article, acknowledge that “some of the political and environmental aspects of such organizations have received little attention in the evaluation literature.” Second, it gives an example of how process use can initiate reflection on practice by stakeholders and can lead to “engagement, self-determination, and ownership” among the stakeholders, in this case internal organizational consultants. In turn, this can lead to action, and ultimately to organizational learning. Basing this evaluation on a logic-model, monitoring the organizational climate for unanticipated changes, and responding to client issues as appropriate all contributed to evaluation process use. The relevance of Russ-Eft, et al’s work is not so much in how they conducted the evaluation, but in how they took the concept of use and expanded it to include and illustrate Patton’s theory of process use. The ultimate end of process use is that it enables evaluators, stakeholders, and audiences to make use of the logic and process incorporated into the evaluation itself rather than just to an evaluation’s outcomes, which may be of limited value depending on a number of organizational factors including politics and a de facto decision making process.

There are two major limitations in this work. First, it is a single case study and the findings only have significance within the context of the organization in which the evaluation took place. As the authors point out, “the processes, results, and level of use and non-use may be unique to this particular evaluation and this particular organization.” The second limitation is that while they claim process use can “initiate reflection on practice by stakeholders and can lead to engagement, self-determination, and ownership among the stakeholders,” they offer no explanation about how this change process actually works. Russ-Eft, et al make a relatively bold leap of faith that there is a direct causal pathway between process use and self-determination, engagement, etc…, but they fail to account for individual variation in their model.

Brett, B., Hill-Mead, L., & Wu, S. (2000). Perspectives on evaluation use and demand by users: The case of city year. In Caracelli, V., & Preskill, H. (Eds.). The Expanding Scope of Evaluation Use. New Directions for Evaluation, 88, 71-84.

This is a case study of a national non-profit organization with multiple local sites that developed an internal evaluation department and then fostered evaluation capacity and organizational learning at both those local sites and in the national office, with varying degrees of success. Evaluation process and use, along with challenges to organizational support for use, are examined within this context. Kirkhart’s (2000) integrated theory is also discussed as a way of reframing the concept of use. The authors present a compelling case for their specific examination of evaluation use by City Year. First, there were various levels of stakeholders at both the national and local levels which present challenges to, and opportunities for, use. Second, City Year introduced a centralized evaluation department as it moved from start-up to institutionalization so the subtleties of use and ongoing efforts to encourage use could be tracked over time. Third, because City Year’s main focus is on action for social change as a learning organization, systematized evaluation activities fostered the development of capacity to be a true learning organization. Once they became involved in evaluation through “gentle persistence” and technical assistance from the internal evaluators, staff (including local corps members) and leaders of City Year realized both its limitations and potential, from the agenda-setting stage to the interpretations of findings. As a ground-swell of individuals found use for the data collected during evaluations and embraced the processes that led to the data, a culture of evaluation and evaluation use gradually took root such that both internal and external stakeholders came to view and accept evaluation as a part of the program and not something forced upon them. Lastly, City Year realized evaluation is a crucial capacity of learning organizations that want to self-perpetuate, of which it is one. 

While there were many successes vis-à-vis evaluation use highlighted in this study, the authors did not go into much detail about the challenges the City Year internal evaluators faced, those they chose to illuminate had to do with unrealistic stakeholder expectations of and knowledge about evaluation, e.g., how long it really takes to obtain findings on long-term program impacts, how much data can actually be collected, and the ease with which findings can be misinterpreted. Other challenges cited were organizational, not individual and had to do with the culture of the organization. These challenges are useful in that they illustrate factors that lead to non-use by organizations, but not by the individuals who embodied that organization’s culture and competencies. The authors provided no explanation of the link between the process of learning how to collect data and individual changes that lead to a valuation of data use for various purposes. No accounting for individual variation in the transformation process was offered to explain how the individual goes from a veritable non-participant in evaluation to a full and active member in evaluation, collecting and using data on a regular basis. 

Henry, G. T. (2000). Why not use? In Caracelli, V. J., & Preskill, H. (Eds.). The Expanding Scope of Evaluation Use. New Directions for Evaluation, 88, 85-98.

Henry’s choice of verbiage in the title of this article makes it appear that the construct of use has been given little attention or has been somehow down-played in the literature. We know the opposite, however, is true.  Without doubt evaluation use is tantamount given the attention it continues to receive in the literature (especially Patton’s utilization-focused evaluation). Henry simply questions whether use should be the defining goal. “Should evaluators set out with the goal of use as the criterion by which they judge the success of their work?” What else is there if not use? In this article, the author argues that evaluation use should not be what defines the ultimate purpose of evaluation, but rather what he terms, “social betterment.” His argument turns on the idea that the goal of evaluation is not use, the goal is social betterment and that use is only a means to social betterment. His caveat is that if social betterment is going to be achieved, evaluation findings have to be purposefully applied towards that end. Thus, evaluation use has much broader implications than what we’ve been led to believe in the literature. Henry’s premise is that if use and only use is the ultimate achievement or outcome of an evaluation, it can, as he says, “distort the allocation of evaluation resources and reduce the contribution of evaluation to broader social goals.” The pursuit of use, in the ways in which we have come to define it, tends to emphasize, overly so, program and organizational improvement. Henry believes that knowledge development, oversight and compliance, and assessing merit and worth as applied to the end of social betterment should “receive their fair share of evaluation resources.”

Yet, where is the individual within this new context of use? By omission, Henry negates the role of the individual in the process of using evaluation for social betterment. A linkage that could have been made might have shown how individual involvement in the process of evaluation might have been motivated by the idea that findings would be used for social betterment which in turn would have lead to the individual change process and transformation necessary to make the connection between process and ultimate use. In the pathway model between use and social betterment, the individual and his or her proclivities around evaluation use has been removed altogether.  

Torres, R. T., & Preskill, H. (1999). Ethical dimensions of stakeholder participation and evaluation use. In, Fitzpatrick, J. L., & Morris, M. (Eds.). Current and Emerging Ethical Challenges in Evaluation. New Directions for Evaluation, 82, 57-66.

This article reviews important ethical considerations and challenges in our efforts to facilitate and promote evaluation use in the “context of stakeholder participation.” Much of the past and present literature on evaluation use indicates that to increase the likelihood of evaluation use by stakeholders, stakeholders need to be involved in the evaluation process. According to the authors, stakeholder involvement is expected in “good evaluation practice and stakeholder involvement leads to greater use of evaluation findings and evaluation processes.” Torres and Preskill add that involvement in the evaluation by stakeholders: 1) increases its responsiveness to their unique perspectives and need for certain types of information; 2) increases stakeholders’ belief in the validity of the findings especially if they are part of the dialogue where those findings are interpreted and meaning is constructed; and 3) increases stakeholders’ skills of inquiry and helps them become more savvy about evaluation. While stakeholder involvement may facilitate use, one ethical dilemma arises when competent evaluation practice is compromised by the lack of formal scientific inquiry skills on the part of stakeholders who may be collecting data or making analytical decisions. Another potential ethical risk is that stakeholders may misuse findings based on their own self-interest as personal biases and feelings were not kept in check by the evaluator, resulting in a distortion of the reported findings.    

 Focusing in on the second point the authors make about involvement, the authors omitted (or failed to acknowledge) how participation in evaluation by stakeholders impacts the internal mechanisms within the individual that influences a belief in the validity of the findings. Again, more weight is given to factors external to the individual involved in evaluation than to those that are internal and upon which an increase in belief or a decrease in disbelief actually turns. As with most of the literature reviewed in this annotated bibliography, the authors fail to place individual variation in the pathway between evaluation process and evaluation outcome.    

Summary

In sum, these articles serve as a point of departure for future research into the exogenous variables that may or may not mediate or moderate evaluation utilization, and also reveal gaps in the literature that might merit investigation. What this brief depiction of a very narrow segment of evaluation utilization literature has attempted to bring to light is that there are multiple ways of defining utilization, that the debate continues, but that there are still no references to individual variables that impact on utilization, specifically on how they impact on the utilization of findings. The literature is, however, fairly clear on when it is more likely that an evaluation will be used, and for what purposes, e.g., enlightenment, program improvement, social betterment, etc…but that only accounts for a limited understanding of utilization. As was mentioned at the beginning of this work, the significant limitation that will be corrected for as the research process moves forward is the addition of literature from other areas of social science research. Based in part on these articles, and in part on personal insight, the following hypotheses have been formulated to address this issue:    

Null Hypothesis: There is a direct relationship between evaluation findings → evaluation utilization such that no matter the evaluation findings, stakeholder involvement, stakeholder understanding of evaluation, or attitudes towards evaluation some measurable degree of utilization will take place.

First Alternative Hypothesis: If the findings → utilization relationship is mediated by utilization factors, the “correct” number and combination of utilization factors attended to during the evaluation process should yield a high degree of utilization. If not, then perhaps there is some other, identifiable, variable that impinges on the relationship. 

 Second Alternative Hypothesis: If the findings → utilization relationship is moderated by utilization factors, then the link between findings and utilization should still be strong, but there will also be a relationship between findings and utilization factors, and between utilization factors and utilization. If all three relationships can’t fully be accounted for, perhaps there is some other variable that has not been taken into account.

References

Brett, B., Hill-Mead, L., & Wu, S. (2000). Perspectives on evaluation use and demand by users: The case of City Year. New Directions for Evaluation, 88, 71-84.

Henry, G. T. (2000). Why Not Use? In Caracelli, V. J., & Preskill, H. (Eds.). The Expanding Scope of Evaluation Use. New Directions for Evaluation, 88, 85-98.

Henry, G. T., & Mark, M. M. (2003). Beyond use: understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24(3), 293 – 314.

Kirkhart, K. (2000). Re-conceptualizing evaluation use: An integrated theory of influence. In Caracelli, V. J., & Preskill, H. (Eds.) The Expanding Scope of Evaluation Use. New Directions for Evaluation, 88, 25-38.

Preskill, H., & Torres, R. T. (2000). The learning dimension of evaluation use. In Caracelli, V. J., & Preskill, H. (Eds.) The Expanding Scope of Evaluation Use. New Directions for Evaluation, 88, 25-38.

Russ-Eft, D., Atwood, R., & Egherman, T. (2002). Use and non-use of evaluation results: case study of environmental influences in the private sector. American Journal of Evaluation, 23(1), 19-31.

Torres, R. T., & Preskill, H. (1999). Ethical dimensions of stakeholder participation and evaluation use. New Directions for Evaluation, 82, 57-66.