What are the benefits and challenges of using AI in evaluation?

AI's great for streamlining tasks and spotting patterns. But when it starts offering diagnostic or therapeutic information, it's stepping into a role that can directly impact patient health and well-being and we've got to treat it like any other medical tool. That means rigorous testing, validation, and regulatory oversight to ensure they're safe and effective. After all, patient safety comes first, no matter how advanced the tech is!

AI's tendency to hallucinate is a major caveat to its "efficiency and accuracy." I have tried to use it to help with literature reviews, and the general concepts it comes up with are valid, but it is terrible at references--it makes them up or says that they say things that they really don't, when you look at them. I have found you have to be selective in terms of the tasks AI is helpful for and those it is not.

There is an inherent bias culturally in AI programmed by Westerners often northerners that doesn’t begin to cover the diversity of hundreds of thousands of different cultures around the world

I agree with Donna Hillman. AI can reduce the workload of data collection and synthesis, but the general consensus of what I have researched says that analysis and conclusions should be checked by humans to ensure accuracy and logic. I have also read that bias is likely to be present in AI-generated results unless you have full control over the information used to draw the conclusions. The same holds true for accuracy. If the AI is drawing information from the internet, for example, it doesn't have the ability to discern what is accurate or correct. If the information is biased or wrong, the same bias and errors will likely be present in the AI-generated results.

AI systems are designed to process and analyze vast amounts of data, identify patterns, and make predictions or recommendations based on that data. They excel at tasks such as data analysis, pattern recognition, and repetitive tasks. However, they lack the ability to fully understand context, exercise judgment, and possess real-world experience or intuition that subject matter experts bring to the table. Just as a calculator is a tool that performs mathematical operations quickly and accurately, but it still requires a mathematician to input the correct equations, interpret the results, and apply mathematical principles to real-world problems. Similarly, AI should be viewed as a tool that complements and enhances subject matter experts.

Artificial Intelligence tools are not creative in themselves. They are fast. That speed provides the 'illusion of creativity.' How? You can quickly iterate and build off of low-level ideas and solutions. I always say: Don't cut and paste generative AI tool results. The creativity arises from tweaking, editing, and re-prompting to achieve a result. The original output gets you close. What you do with it makes you creative.

You can also use AI as your critical friend, though keep in mind that your friend's knowledge is based on a limited and potentially skewed dataset. I have asked AI to scrutinise assumptions of a theory of change, or limitations of an evaluation design. The results need human interpretation, but can be seen as an extra data source to consider.

Given the swift expansion of Generative AI, numerous tools are now accessible to users through open-source platforms. Witnessing the remarkable capabilities of a well-trained AI is truly fascinating. With extensive training on vast datasets, AI exhibits the ability to rapidly generate text, images, audio, and video. However, it is crucial to emphasize the need for data verification, particularly in light of the recent surge in the development of deepfakes.

Recognizing the overuse of terms like "AI," having more advanced technology help gain awareness of diverse factors, influences, and contextual drivers, as well as outcomes, can give us a far more accurate, complete, and unique picture of community needs and the impact of services/programs. So often, we can easily ignore or miss critical components that are influencers and/or impacts of our work. Having AI help us identify things we may otherwise have missed opens the door to better questions we can answer, more innovative methodologies and insights, and greater creativity fueled by curiosity.

Another limitation of AI is that the outputs generated may be accepted as accurate without being cross referenced, checked and verified. ChatGPT has given me erroneous information on more than occasion and it was good that I validated the data before using it.

This strikes to the heart of some concerns. Are we complicating the work of human developers and participants by creating risks and harms created by limitations of AI?

Another consideration is who developed the AI systems? What biases do they hold, and what does that mean for which datasets and decision points are included? Technology also reflects biases, but that can be very easy to overlook.

Bias and inaccuracy are two issues I faced when using AI for evaluation. ChatPDF once directed me to a reference that was not mentioned in the document, but interestingly, it corrected its mistake a few seconds ago. There is also a potential bias in AI applications. ChatGPT once displayed bias while discussing an evaluation theory, and I have heard this observation from many other fellow evaluators.

As another risk of using AI: In a time when community owned MERL and the decolonization of MERL both make important strides, AI could actually move us back on these two developments. Another ethical concern for me, evaluating on the topics of climate change and the environment, is that AI is incredibly energy hungry; Training one AI model can burn more electricity than is consumed in a year by 100 US homes.

Here, again, are areas that need to addressed - “appropriate data, infrastructure, and tools” to use AI. Also, are there enough human researchers who have the background and skills to implement and monitor this process?

The AI field is experiencing a rapid boom, leading to a rise in opportunities. As interest in AI grows among the public, the associated challenges can be addressed through the passage of time and increased awareness. Although the number of experts is currently limited, it is anticipated that this shortage will be addressed in the near future.

One think i found helpful is that AI simplifies the process and reduces time one could spend when doing work manually. Always one must validate information processes using AI, the chance for AI to manipulate information or data is very high! The other shortcoming is that AI consistently uses the same word as pivotal, crucial, nuances etc., these are common words, as a result it is boring to use same words across the entire documents. Lastly, Ai uses hard language, EVERYONE NOW IS A EXPERT OF EVERY LANGUAGE, which is not true. Most people now they are expert of english let us say, but they dont even know to speak

I'm not convinced that AI can engage effectively with qualitative data. I am concerned about a shift in evaluation practice that prioritises data and coding rather than the work done and its effects on people. I see this as a general trend that is strong in North America in particular, but I have no doubt others will follow in time. I work on feminist and participatory evaluation and I don't see how AI can positively contribute to this, as well as efforts to decolonise international development in general and evaluation in particular.

Explore online courses from reputable platforms like Coursera or Udacity to learn the fundamentals of AI evaluation. Dive into hands-on projects and competitions on platforms like Kaggle to experiment with real-world datasets. Collaborate with peers, join AI communities, and stay updated on the latest research to continually refine evaluation techniques. The key is to combine learning with practical experimentation for a comprehensive understanding of AI evaluation.

In response to Donna Hillman: skills development is always on the card and whilst reducing time available to continue other activities, it means that we can focus on the key processes that can only be done by humans... applying judgement and values we care about.

My only hesitation here is that we are asking proficient professionals to “take online courses, read blogs and books, watch webinars and podcasts, and join communities and networks related to AI and evaluation.” Becoming proficient in AI would, in my opinion, reduce the ability and time available to continue other activities and academic research that they may be involved in.

Collaboration and Reflection are always valuable tools. However, many of us have been willing to work with others but have occasionally found that those already involved are sometimes reluctant to share their knowledge and experiences.

A co-creation or co-production approach may become more important as we are all struggling to accommodate the principles of transparency and fairness with skills development and shared learning. This drive to be open about what is AI produced and what is not, may give us a positive push towards more openness about the evaluation and analysis processes with funders and program participants.

AI-based evaluation models may need to add a module with the middle level of synthesis to address some of the limitations in capturing qualitative diversity between respondents. AI models could be designed to analyze and synthesize data at different levels of granularity. By incorporating micro, meso, and macro levels of analysis, they would be better equipped to handle the complexities of qualitative data. For example, using unsupervised learning techniques, such as clustering or topic modeling, it might be possible to identify sub-themes at the micro level, and then aggregate them together into themes at the meso and macro level.