AI in Education: New Research 6th May

I've been a lot more picky about the research I've included this time around, and focused on research that's got an immediate impact for a teacher or educational organisation

Featured Research

The Artificial Intelligence Assessment Scale (AIAS): A Framework for Ethical Integration of Generative AI in Educational Assessment

https://open-publishing.org/journals/index.php/jutlp/article/view/810

A bit of background to this - the paper is from the Journal of University Teaching and Learning Practice and comes from authors Dr Mike Perkins Leon Furze Jasper Roe and Dr. Jason MacVaugh , from British University Vietnam , 澳大利亚迪肯大学 and James Cook University .

I love this paper in particular because it's not a theoretical paper - it's something that every educationalist can grab with both hands and implement in their classroom/school/university tomorrow.

The paper discusses the introduction of the Artificial Intelligence Assessment Scale (AIAS), a tool designed to ethically integrate Generative AI into educational assessments. The AIAS helps educators determine the appropriate level of AI involvement in assessments, aiming to balance innovation with ethical considerations. The scale provides clarity and transparency for both educators and students, ensuring fair use.

The scale is really clear for students - from Level 1 (No AI)? to Level 5 (Full AI), but those are the easy stages - it's the middle stages that are really helpful. Like Level 3 (AI Assisted Editing), where it clarifies for students that they can use AI to help with clarity or quality, but no new content can be created using AI, and they need to attach their original work as an appendix to the assignment.

Although? this? scale? was? originally designed? for? use? in HE ,? it? is? sufficiently? flexible for application in schools too.

We had Leon on the AI in Education Podcast late last year, and he talked about this, so I'd recommend following Leon Furze on here, as he's constantly sharing good practice on AI

Best of the Rest

Automated Social Science: Language Models as Scientist and Subjects

https://arxiv.org/abs/2404.11794

This paper, from Benjamin Manning and John Horton at 美国麻省理工学院 - 斯隆管理学院 and Kehang Zhu at 美国哈佛大学 , suggests a potential massive revolution in social science. It develops a system where Large Language Models (LLMs) automatically generate scientific hypotheses, and then test those hypotheses with simulated AI human agents. Even at this early stage, it works surprisingly well. As Ethan Mollick said "agent-based models have been a staple of social science for a long time. The idea of using simulated agents is not that big of a deal". So what this gets to is the possibility to use LLMs to create digital twins of a group - your students, parents, prospective students - to test hypotheses against. LLMs have demonstrated the ability to simulate human-like responses in various scenarios with a surprising degree of realism. For your panel, this means the LLM can realistically mimic the responses of your target group when asked about something - eg how to design an activity, or how they would respond to a message, considering different personas or profiles you input into the model.

If you want to give this idea a try (for example, to work out how you might improve a parents' evening for everybody involved), then here's a step by step guide to The Infinite Focus Group https://www.ai-mindset.ai/ai-mindset-newsletter/the-infinite-focus-group ?

It's important to include this next paper too - because this isn't a panacea for everything, as a second piece of recent research highlights the flaws that might exist!?

Large language models cannot replace human participants because they cannot portray identity groups

https://arxiv.org/abs/2402.01908

From Angelina Wang at 美国普林斯顿大学 , Jamie Morgenstern at UWA, and John Dickerson at 美国马里兰大学帕克分校 , the research discusses the limitations of LLMs in accurately representing diverse human identities, such as gender, race, and disability. It argues that LLMs like ChatGPT struggle to accurately portray the varied perspectives of different demographic groups because of the way they are trained. These models often rely on stereotypes and can't capture the subtleties of individual experiences, leading to misrepresentations. The study highlights the risks of using these models as substitutes for human participants in research settings where identity and personal experience are crucial. It risks silencing real voices and experiences, particularly those from marginalised or underrepresented communities. The paper argues that while LLMs can assist in research by providing preliminary insights or helping in pilot studies, they shouldn't replace human participants when authentic identity perspectives are crucial. The authors propose more nuanced training methods and the use of identity-specific prompts to reduce errors, but they caution against relying on these fixes entirely.

The impact of large language models on university students’ literacy development

https://www.tandfonline.com/doi/epdf/10.1080/07294360.2024.2332259?needAccess=true

Daniel W. J. Anson from one of my local universities, 澳大利亚麦考瑞大学 has produced a great article on how poor use of LLMs could harm academic literacies of students. The primary concern raised is that while tools like ChatGPT can enhance engagement and assist with certain tasks, they might also lead students to bypass the rigorous mental processes involved in reading, understanding, and critiquing academic texts. For instance, if a student relies solely on summaries generated by an LLM, they miss out on developing a deep understanding and the ability to engage critically with texts—skills that are crucial for academic success. The paper suggests that educators need to design curricula and assessments that encourage students to engage deeply with learning materials and to use LLMs as supplementary tools rather than replacements for their own cognitive efforts. This might include structured activities that integrate LLM use with critical reflections, discussions, and revisions based on feedback.

Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays

https://www.sciencedirect.com/science/article/pii/S2666920X24000109

This paper, from 8 researchers in Germany and Switzerland, basically says "No", they cannot spot AI. In a longer answer, both novice and experienced teachers were mostly unable to identify AI-generated texts, with some slight improvement in accuracy among experienced teachers for high-quality texts. Additionally, teachers were generally overconfident in their ability to spot AI-generated content, even when their actual performance did not support this confidence. They had the participation of 89 pre-service teachers and 200 experienced teachers, so this isn't a small sample.

Pre-service teachers only identified 45% of AI generated texts correctly, and even worse, they incorrectly said that 46% of texts that had been written by students were written by AI!

For experience teachers, it was just as bad - they only identified 38% of AI generated texts correctly, and incorrectly said that 27% of texts that were written by students were written by AI.

So, in a class of 30 students, if they all wrote their own work, experienced teachers would falsely accuse 8 kids of using AI. And if they'd all written their work with AI, they'd miss 10 assignments that had been robot written.

And, according to this research, they'd make these bad judgements with confidence!?

Feedback sources in essay writing: peer-generated or AI-generated feedback?

https://link.springer.com/article/10.1186/s41239-024-00455-4

Omid Noroozi at Wageningen University & Research , along with Seyyed Kazem Banihashem ,?Hendrik Drachsler ,?Nafiseh Taghizadeh and Jewoong Moon published the research, in the International Journal of Education Technology in Higher Education. It investigates whether AI-generated feedback, specifically from ChatGPT, can be as effective as peer-generated feedback for students writing argumentative essays. The study was conducted with 74 graduate students from a Dutch university who were tasked with writing essays in English, which were then reviewed both by their peers and by ChatGPT. The quality of the feedback was analysed and compared to determine which type was more beneficial for improving essay writing. ChatGPT excels in providing more descriptive feedback on essay composition and structure, peers are better equipped to identify significant issues within their peers' essays. The findings suggest that ChatGPT and peer feedback could serve complementary roles in the educational process, combining detailed descriptions from AI with the critical insights of human peers.

?