Evaluation Budgets Redux: How Evaluation Budgets Need to Consider the Costs of Data Plans in Linking Monitoring and Evaluation

Scott Chaplowe and Susan Legro have authored a useful resource on evaluation budgeting that supports climate change evaluations for the Adaptation Fund .

I highly recommend this resource. In addition to including a checklist, it is especially useful to see how one organization – the Adaptation Fund – is leading on serious evaluation at the organizational level. Kudos to the Adaptation Fund which is a major donor addressing #climate change and has developed a strategic approach supported by an #evaluationpolicy! And Scott Chaplowe is correct in noting that the literature is sparse on evaluation budgeting overall which has led to many misconceptions and -- thanks to Scott Chaplowe and the Adaptation Fund for sharing approaches and a checklist!

There are a couple of cautions I would add for those doing a deep dive into evaluation budgets. Different donors ranging from governments to global donors have adopted different approaches in the “evidence era” whose mantra is to treat “data” as an asset and promises that management decisions are both evidence-based and data-driven. In the U.S., we can trace the evidence era to the Evidence Act of 2018 and ongoing Office of Management and Budget (OMB) guidance as well as other initiatives originating in the 1990s at the state and local levels around performance measurement and management. In the evidence era, data plans are a necessary component that comprise an unfunded mandate when they are omitted from budgeting.

First, it is important to recognize that evaluation has become highly specialized within content areas. Climate and environmental intervention logic models and theories of change – the content-specific foundation for most evaluations – differ from those in other content areas. There is no one source detailing these content differences but the USAID listing of frameworks and indicators in specific sectors is helpful. In brief, each content area will have different evidence and data gaps which means that evaluation budgets and evidence-building will differ as well as increasingly, evaluators are specialized within the health, agriculture, energy, investment, economic development, human rights, governance, rule of law, youth, gender, creative economy, and anti-corruption fields (to name a few). And specialized subfields are emerging as well within each of these broader areas. For example, in the DRG field, parliamentary strengthening is distinct from political party or civil society capacity building or election administration, or in the economic development field, “doing business” is distinct from “foreign direct investment.”

Second, the Adaptation Fund formal separation of monitoring budgets from evaluation budgets – based on its internal results-based management framework - is distinctive. Unlike the Adaptation Fund, many organizations lack an Evaluation Policy and also one that is grounded in strategies related to the desired outcomes. This separation, which assumes that program staff do the monitoring and evaluation staff do the evaluation, may work for climate change projects or may make sense for one donor organization’s evaluation policy, but it may not work for all.

From an evaluation standpoint, this separation is unusual because it:

1. Predates the evidence era which provides an alternative path such that monitoring and evaluation are used in increasingly integrated ways for learning and evidence-building both for the U.S. government and for a growing number of global development organizations such as CARE Care Systems Approach and Oxfam Oxfam Systemic Evaluation Stories which both use the model of systems change; and
2. Ignores the responsible data responsible data revolution where the challenges, benefits and risks of data misuse require a balanced data management plan.

Let me explain a bit more: evaluation budgets in the evidence era are topics I address in two LinkedIn posts I have previously shared that discuss both budget tiers and how to prepare a direct cost budget:

Part 1: What Should the Evaluation (M&E/MERL/MERLIN) Budget Look Like?

Part 2: Understanding the Components of a Direct Cost Evaluation Budget

Two related topics include evaluation independence and responsible data:

1. A blog I am working on related to evaluation independence - a key component of rigorous evidence-building (that will be posted February 16, 2025 on the American Evaluation Association AEA365 blog series) - emphasizes the importance of embedded evaluation. Embedded evaluators link monitoring and evaluation but remain independent (a good resource is The Big Book on Evaluation Good Practice Standards); and

2. The requirements of responsible data in the era of digital data, big data, and the mosaic effect in data which I briefly outline below that preclude an artificial separation of monitoring and evaluation. This is especially true for projects where the desired final outcomes are people-focused (e.g., democracy, rights and governance projects; gender mainstreaming and equity; youth programs; education and others) compared to climate change projects where the final outcomes reflect environmental indicators based on behavioral change(s).

Data should be collected once and used many times to reduce burdens on respondents and increase data value (data asset). Separating monitoring and evaluation budgets ignores the fact that evaluators – distinct from program staff – have specialized skills that allow them to link and re-link personal identifying information to clean and update data and meet the requirements of specific types of evaluation designs and baseline-endline comparisons. Any project which collects data from persons needs to consider data ownership and the burdens of data collection, risks to the individuals who share information as well as risks to the organization (and project) which collects and maintains the data should data be inappropriately released or hacked.

Program staff report on “one-off” data collections whether program or other data (cross-sectional surveys, key informant interviews, focus groups, etc.) that cannot be further analyzed and compared at the “case” (individual) or “dosage” (number or strength of program contacts) levels to account for tracking multiple and varied contacts with diverse target beneficiaries over time. One-off data collections are usually anonymous or de-identified after data collection and then easily shared with program staff and donors – but they then become unusable for before and after comparisons. They are repeated measures that become more expensive to collect at repeat intervals. Responsible data managing evaluators develop and keep a data file with limited access separate from program staff and others that permits periodic or regular re-identification for specific purposes in data preparation, cleaning, and use. This allows them to use the data many times and do extensive data disaggregations for evaluative purposes as questions arise.

Evaluators have the unique ability to re-integrate monitoring and evaluation data by managing trade-offs because they distinctively:

? Hold Human Subjects Research Training Certification and applies these standards;

? Comply with Professional Ethics Standards that separates values from data;

? Use “Informed Consent” that ensures that participant know and understand the risks;

? Work Responsibly with Data “Confidentially” NOT “Anonymously” to allow for secure data re-identification;

? Possess Data Management (avoid double-counting) & Data Analysis Skills to provide for responsible de-identified data sharing that respects privacy and ensure a lack of bias; and

? Utilize Best Practices Standards for meaningful, transparent, accountable and credible data collection, use and interpretation.

Evaluators who are engaged in both monitoring and evaluation designs are well equipped to manage ethically and responsibly the competing tensions of data value, rights and beneficience. The U.S. federal data strategy, based on existing data and human subjects research laws, includes the principles of data as an asset, the needs and benefits of stakeholders, transparency, accountability and ethical governance (among others). Yet, these values also need to be based on organizational and individual risks, security and privacy and the burdens of collecting data. Three of these values principles reflect competing values that must be balanced:

? DATA VALUE: Data as an Asset vs. Data as a Risk

? DATA RIGHTS: Transparency & Open Data vs. Data Security & Privacy

? BENEFIENCE: Burdens vs. Benefits of Collecting Data

Finally, the artificial separation of monitoring and evaluation budgets in most cases fails to be cost-efficient. Even if the data collections for monitoring and impact evaluation have different final purposes (interim donor reporting vs. drawing ex post impact conclusions), the @Donor Committee for Enterprise Development (DCED) – a thought leader on results-based evaluation which includes both home office evaluation leadership and embedded evaluators trained with their method. The Results-based measurement approach developed by the DCED which emphasizes both results and the ability of the evaluation to support attribution of the project results to the intervention, includes a standard for budgeting for results measurement between 5-10% of the overall project budget (DCED Attribution in Results Measurement (2017). If a separate evaluation budget is prepared in the same range (or more), the total MEL cost for a single project risks ballooning with unnecessary and multiple data duplications that increases respondent burdens to a total of 10-30% or more of the total project budget (and thus likely will undermine funding the project intervention activities).

Evaluators need to join evaluator skills and content knowledge with management skills in the evidence era. I look forward to seeing more great examples like the Technical Evaluation Reference Group of the Adaptation Fund has shared in their work with Scott Chaplowe and Susan Legro .