The Responsible AI Bulletin #29: Cultural alignment in LLMs, novice programmers + automated repair, and the importance of audits.

Welcome to this edition of The?Responsible AI Bulletin, a weekly agglomeration of?research developments?in the field from around the Internet that caught my attention - a few morsels to dazzle in your next discussion on AI, its ethical implications, and what it means for?our future.

For those looking for more detailed investigations into research and reporting in the field of Responsible AI, I recommend subscribing to the AI Ethics Brief, published by my team at the Montreal AI Ethics Institute, an international non-profit research institute with a mission to democratize AI ethics literacy.

How Culturally Aligned are Large Language Models?

Imagine a conversation with an LLM like ChatGPT or Bard, but through the lens of different cultures – from the busy streets of New York to the serene deserts of Saudi Arabia. Our research engages in this fascinating journey, exploring how LLMs resonate with the cultural values of various countries. Amid the growing concern that AI systems often reflect the perspectives of Western perspectives, our work attempts to evaluate the cultural alignment of LLMs using Hofstede’s CAT. This innovative tool, grounded in Hofstede’s well-known cultural dimensions theory, examines the cultural values embedded within LLMs like ChatGPT and Bard.??

Through our research, we prompt LLMs to echo the cultural values of four countries with diverse cultural norms: the US, Saudi Arabia, China, and Slovakia. Our findings are eye-opening: while models like GPT-3.5 and GPT-4 show closer alignment with the US, they significantly misalign with other countries, particularly Saudi Arabia. Surprisingly, Google’s Bard exhibited the highest misalignment with US cultural dimensions. These results highlight a pressing need for culturally diverse AI, paving the way for more inclusive and globally relevant technology.

Continue reading here.

How Helpful do Novice Programmers Find the Feedback of an Automated Repair Tool?

The availability of ChatGPT created a buzz around artificial intelligence and how it can impact education. The latest version of ChatGPT allows students to interact with an AI agent for programming-related matters. Some papers report using Large Language Models (LLMs) to give feedback to students learning programming. At the same time, many papers also reported that LLMs may give incorrect or out-of-context feedback to learners. Such inaccurate feedback can be harmful to learning.?

One way to give accurate feedback is through the use of automated repair tools. CLARA, which stands for CLuster And RepAir, is one such example of an automated repair tool. The basic idea of CLARA is to use a group of correct solutions from other students to automatically generate feedback for an incorrect solution. The authors enhanced CLARA to support more Python language features and made it available as a service through a RESTful API. They also created a Jupyter Notebook extension to allow students to access CLARA directly from their Jupyter Notebook assignments. They called this enhanced system CLARA-S for CLARA Service.

The authors studied how students use CLARA-S in solving programming problems. They found that one of the difficulties faced by novice programmers in using such a system is that they could not understand the feedback generated by CLARA. However, those who have programming experience can utilize it better. The most useful feature when using CLARA-S is to locate their logical or semantic errors. They found that such a system is still useful but mainly for students with some prior good programming skills or for instructors. For CLARA to be useful for novice programmers, the feedback message has to be made easier to understand by these users.

Continue reading here.

The importance of audit in AI governance

AI, through its applications, is steadily enveloping humans and their environment. These applications are often envisaged by organizations aiming to create value in varied ways, among others, through innovative products and services. That technology leads to undesirable effects is well-researched. Yet, this time, it is maximi momenti: the rudimentary classification arrangements in data-driven AI systems fringe some solemn and historically hard-won fundamental rights, most prominently discrimination. It resulted in a vigilant European Commission proposing the Artificial Intelligence Act (AIA) to safeguard its citizens’ fundamental rights. Legislations (regulations) without proper enforcement mechanisms are heterogeneous in their impact, which is noticeable with the roll-out of GDPR. Attempting to comprehend how organizations (intend to) comply with the AIA, we initiated primary research in the Netherlands through a survey and validation interviews.

Continue reading here.

Comment and let me know?what you liked and if you have any recommendations on what I should read and cover next week. You can learn more about my work here. See you soon!