I spent this weekend experimenting with ChatGPT. I found 3 key things that I wanted to share with my fellow evaluators. Keep these in mind as you use this generative AI platform to support evaluation work.
- You can use ChatGPT to write code. If you’ve been dreaming of using natural language processing, supervised or unsupervised machine learning - ChatGPT can help you write the code to prepare your text. It will provide you with the code you request in the format you request it (Stata, R, Python or other). Awesome - right? Well don’t stop your Udacity or Coursera course quite yet. ChatGPT isn’t always right. You will still need a basic understanding of coding to troubleshoot its mistakes. On the bright side - ChatGPT will take you much further, much faster than you would likely have gone on your own.
- ChatGPT does not do citations! You can ask ChatGPT to provide you with an answer to a prompt and even ask it to provide citations. But evaluator beware - ChatGPT fabricates citations! I recently asked ChatGPT “What are the benefits of using chatbots for data collection? Please provide a citation too.” It responded with my answer and provided this citation “Sim?es, L. M., de Lima, E. F. F., & Neves, M. C. (2020). Chatbots in data collection and monitoring: An overview. Journal of Business Research, 116, 13-25. https://doi.org/10.1016/j.jbusres.2020.04.022
” Looks perfect right? Except that citation DOES NOT EXIST. You still have to do your own research - ChatGPT will not do that part for you.
- ChatGPT cannot analyze your evaluation documents (yet). The holy grail for evaluators is a program that will take the relevant evaluation documents (program reports, interview transcripts, policy documents etc…), analyze them and provide the most salient findings, conclusions and recommendations. This is possible (see for example the World Bank’s excellent report)
, but ChatGPT cannot do it. Even if your evaluation documents were in the right format for ChatGPT to ingest, ChatGPT would need to be retrained to look at “just” your content. This day is coming but we are not there yet.
ChatGPT is an excellent tool to quickly understand a new topic and to help summarize some key ideas. It is not a replacement for rigorous evaluative thinking and practice. We evaluators need to stay abreast of emerging technologies, but our jobs are safe for the moment.
Founder @ TAU Consulting | Product Owner | Monitoring, Evaluation, Accountability and Learning | AI | Workflow Automation | Digital Transformation
1 年Interestingly enough, I have developed and AI solution which can be easily adapted but it already used for Rvaluation Report reviews, easily. Thus, chat GpT understands your evaluation and your documents: https://coda.grsm.io/AI-ReportReview
Climate change and environment-focused lead evaluator / reviewer / validation expert
1 年Kerry Bruce have a look here: https://beebom.com/how-train-ai-chatbot-custom-knowledge-base-chatgpt-api/ It seems quite simple, training an AI Chatbot with a limited dataset. I will give this a try with a dataset from previous evaluative work and see what it brings.
MEL Director
1 年Susana Guevara
Program Manager at Bixal
1 年Thanks Kerry! Regarding citations, I've found one very helpful use for ChatGPT—checking citation formatting across a range of styles. It's great for picking up an error that tired human eyes miss, but that is as far as I'm willing to trust its accuracy.
Director of Evaluation at TCC Group
1 年Love these insights--thanks for sharing!