EvalIndigenous at AES 2024 Melbourne, Naarm, Australia

Participation of Africa Co-Convenor of EvalIndigenous, Dr. Awuor Ponge, at the Australian Evaluation Society (AES) 2024 Conference in Melbourne /Naarm, Australia, 17 – 20 September 2024.

My participation at the Australian Evaluation Society (AES) 2024 Conference in Melbourne /Naarm, Australia, 17 – 20 September 2024, was made possible through the generous support from the EvalIndigenous Global Network, through the support from EvalPartners and the International Organization for Collaboration in Evaluation (IOCE). Additional support for conference registration was obtained from AES and from Katoa Limited.

?The Conference theme was on ‘Wayfinding’. Wayfinding is an ancient process for navigating unfamiliar territory towards an intended destination. It encompasses physical and mental processes, tools and strategies we can use to identify cues, interpret our surroundings and navigate through complex spaces.

The first Keynote address at the AES 2024 International Evaluation Conference in Melbourne, was delivered by June Oscar AO, a proud Bunuba woman from Fitzroy Crossing in Western Australia's Kimberley region. She is the Inaugural Chair of The Wiyi Yani U Thangani Institute for First Nations Gender Justice. June Oscar is known for her work on First Nations gender justice and women's voices. Her keynote address was titled: “Re-imagining evaluation with a gender justice lens." Her address touched on themes related to Indigenous rights, gender equality, and the importance of incorporating First Nations perspectives in evaluation practices. She shared the voices of the women from First Nations women whom she had spoken to during her term as Aboriginal and Torres Strait Islander Social Justice Commissioner. Some of the notable quotes include:

·????? "Women are the social fabric of communities and the glue that holds everything together."

·????? "We are the ones that are the backbone that everyone depends on to get things done."

·????? "What keeps me strong is my grandmothers. They didn't have the power to have a voice and I am bloody determined to make sure my girls have a voice.”

?The second keynote address was delivered by Prof. James Copestake. He is a Professor of International Development at the University of Bath in the UK. He has background in international development and evaluation. Prof. Copestake addressed two key points: promoting useful forward-looking evaluative practice, and exploring examples of forward-looking evaluative practice. He then made suggestions on specific approaches for forward-looking evaluation that included increased use of causal mapping and more backcasting. He argues for a shift towards more forward-looking evaluative practices that integrate anticipation and backcasting into evaluation processes. He emphasised the need for evaluators to adopt a transformational role, moving beyond conventional methods that focus primarily on past outcomes. Ultimately, Prof. Copestake advocates for a more inclusive and anticipatory evaluative practice that can inspire radical change and enhance the impact of evaluations on public action. His keynote address also touched on themes related to impact evaluation, mixed methods approaches; and the Qualitative Impact Protocol (QuIP) which he has worked on extensively.

The Chairperson of our Session was Alice Muller, a Senior Monitoring & Evaluation Advisor: FMNR Scale Up, World Vision Australia. I made a joint presentation on a study that we conducted with Grace Wanjiru Murigo among the Mijikenda and the Ogiek communities in Kenya. Our presentation was titled: “Evaluation that adds value for People and Planet: Perspectives, Challenges, and Opportunities for Indigenous Knowledge Systems in Africa.” It focused on the importance of integrating Indigenous Knowledge Systems in evaluation practices. We emphasised on a respectful and collaborative approach that values the cultural traditions, practices, and rights of Indigenous peoples. We argued that evaluations should not only assess social and environmental impacts but also contribute to cultural preservation, sustainable resource management, and social justice. We also highlighted the challenges faced by these communities, including loss of traditional knowledge and land rights issues, while also identifying opportunities for leveraging Indigenous knowledge in areas such as agriculture, biodiversity conservation, and health.

The other paper in the session where I presented was by Steven Legg, Associate Director, NSW Treasury and Eugenia Marembo, a NSW Treasury, Senior Analyst, First Nations Economic Wellbeing. Their presentation was titled: “Culturally Responsive Initiatives: Introducing the First Nations Investment Framework.” In their presentation, they argued that there is a need for practical guidance to improve capability across the sector in evaluating and implementing initiatives that impact First Nations people and communities. They mentioned that the First Nations Investment Framework had been developed to address this need. The framework provides structured guidance across several key areas, including foundations for genuine relationships, partnership for shared decision-making, community-led design, data collection and analysis, and transparency and accountability. They concluded their presentation by asserting that implementing culturally responsive approaches can lead to tangible positive impacts for First Nations people and communities.

?

I attended a presentation by Dr. Sandra Ayoo and Stacy Huff titled: “Evaluating Meta-Evaluations: Addressing Social Justice in Program Evaluation.” The presentation focused on the importance of integrating social justice into program evaluation, particularly through the lens of meta-evaluation. It highlighted the current lack of consensus on standards for conducting meta-evaluations and emphasised that addressing social justice is context-dependent rather than based on fixed guidelines. The findings shared advocate for evaluators to prioritise social justice in their designs and methodologies, while also selecting quality assurance tools that align with their specific evaluation contexts. The presentation underscored the shared commitment of organisations like the United Nations and the American Evaluation Association (AEA) to enhance evaluation quality by embedding social justice considerations throughout the evaluation process, aiming to ensure that evaluations benefit both individuals and society at large.

On Day 2 of the Conference, Thursday, 19 September 2024, the first Keynote address was delivered by Elizabeth Hoffecker. The presentation was titled: “Wayfinding Tools for Learning and Evaluation in Complex Systems.” The presenter talked of the context for a systems-aware evaluation, mentioned the key systems components as: elements – whether living or non-living, relationships and purpose and function. She also talked of the need for capacity development – which is a learning-focused activity and research. The presenter then addressed the navigational strategies and tools for systems-aware evaluation. ?The strategies included:

·?????? Getting the lay of the land – a systems mapping and stakeholder analysis;

·?????? Orienting inquiry with frameworks;

·?????? Navigating change with middle-range theory – what are the middle-level causal mechanisms?

·?????? Probing to learn with pilots and learning questions;

·?????? Course-correcting with reflective practice – action-learning; collaboratoive sense-making and simple After-Action-Reviews (AAR);

·?????? Identifying Change and results at the systems level, using systems-aware methodologies such as Outcome Mapping, Outcome Harvesting and Contribution Analysis, to mention just but a few.

Dr. Ellie Taylor and Ellen Hays made a presentation titled: "Nothing about us, without us": Developing evaluation framework alongside victim-survivors of modern slavery using representative participatory approaches.” The presentation outlined the development and implementation of an evaluation framework for the Lived Experience Engagement Program (LEEP), which aims to involve survivors of modern slavery in shaping policies and responses to this issue. Using a representative participatory action research approach, the evaluation sought to understand the program's impact on stakeholders, particularly survivors, while adhering to principles of trauma-informed, strengths-based, and culturally safe practices. Key findings highlighted the program's success in facilitating survivors' transition from lived experience to lived expertise, increasing confidence, fostering peer connections, and developing knowledge and skills. The evaluation process itself emphasised sustainability, fairness, and respect for human rights principles, while allowing for iterative improvements and the integration of diverse perspectives from survivors, staff, and government stakeholders.

?

On Day 3 of the Conference, on Friday, 20 September 2024, the keynote address was delivered by Indy Johar on the topic: “Addressing transitions through risk and uncertainties.” Indy is a celebrated personality, who is a RIBA register architect, serial social entrepreneur, and Good Growth Advisor to the Mayor of London, UK. He is a co-founder of Dark Matter Labs and of the RIBA award winning architecture and urban practice Architecture00. Indy emphasised the need for large-scale, systemic transitions to address planetary-scale challenges. He argues that we're moving from small experiments to "micro-massive" swarms of interventions happening simultaneously. He therefore concluded his speech by advocating for the development of new capabilities to build and coordinate these distributed interventions, rather than relying on top-down, centralised approaches.

Mitchell Rice-Brading, a Consultant with ARTD Consulting Australia made a presentation titled: “Involving children and young people in evaluations: Equity through active participation” The presentation addressed three key issues regarding involving children and young people in evaluations. It explored the ethical considerations of balancing risks and rights, emphasising the tension between protecting children from unnecessary risks and respecting their right to participate in matters affecting them. The importance of careful ethical protocols and informed consent was equally highlighted. The presentation also discussed recruitment challenges, including obtaining approvals, limited participant pools, and low confidence among potential participants, while suggesting strategies for strong engagement and obtaining consent. Lastly, it outlined the approaches to data collection and engagement, emphasising person-centered and strengths-based principles. This presentation provided practical suggestions for age-appropriate data collection methods, such as focus groups, adapted surveys, and interactive activities, while stressing the importance of making participants feel comfortable and addressing power imbalances.

?

Julian King and Adrian Field made a presentation titled: “Evaluation and Value for Money – Value propositions: clearing the path from theory of change to rubrics.” The presentation discussed key issues related to value propositions in program evaluation. It touched on the distinction between impact and value, emphasising that a theory of change should explicitly address value creation. Impact refers to changes caused by a program, while value relates to what matters to different stakeholders. It also discussed the usefulness of value propositions in bridging the gap between theories of change and evaluation rubrics. They mentioned that by articulating to whom a program is valuable and in what ways, value propositions add depth to theories of change and facilitate the development of clearer evaluation criteria. Lastly, they highlighted the process of developing a value proposition, which involves answering questions about stakeholders, resource investments, ways of working, equity considerations, and factors affecting value creation. This approach helps evaluators define context-specific aspects of good resource use and value creation, leading to more meaningful evaluation criteria.

?

The closing keynote address was delivered by John Gargani. He is the President of Gargani + Company and a former President of the American Evaluation Association (AEA). His areas of expertise include impact evaluation, scaling impact, value for money, and the use of AI in evaluation. His address was titled: “Finding our way to the future profession of evaluation”. His address focused on impact evaluation and scaling impact. He discussed how evaluators will need to focus more on assessing and helping scale the impact of programs and innovations; the need for integration of AI and technology in evaluation practice and developing new methods for evaluating "value for money" from multiple perspectives and more nuanced ways of assessing value beyond just monetary measures. He also argued that quantitative methods can deepen qualitative understanding of impacts, thereby stressing on the importance of continued use of mixed method approaches in evaluation.

?

At the close of the event, by AES President Kiri Parata, the 2024 Conference Organising Committee officially handed over to the next Conference Organising Committee for aes25 to be held in the Australian Capital City of Canberra / Ngambri. All in all, this was a wonderful event full of learning in the ever-evolving field of evaluation practice! Canberra 2025, here we come!