Is GenAI Capable of Outperforming Australian Law Students?

We all are aware of GenAI's popularity and the rise of generative artificial intelligence has sparked the debate across various different industries, including law. As these powerful AI tools continue to advance, the question arises about their potential to replace or outperform human professionals.?

Recently, a question came to light: can generative artificial intelligence (GenAI) outsmart 90 percent of law students??

Back in 2023, OpenAI made a claim and announced that ChatGPT-4 scored higher than 90 percent of human test takers on a simulated version of the US bar exam .

It’s an interesting concept and something that Dr Armin Alimardani, a lecturer in Law and Emerging Technologies at the University of the Wollongong School of Law, and a consultant at OpenAI, wanted to investigate further.

His findings form the basis of a new paper, Generative Artificial Intelligence vs. Law Students: An Empirical Study on Criminal Law Exam Performance , which was published on Tuesday, 24 September in the Journal of Law, Innovation, and Technology.

He said: “The OpenAI claim was impressive and could have significant implications in higher education; for instance, does this mean [that] students can just copy their assignments into generative AI and ace their tests?”

“Many of us have played around with generative AI models, and they don’t always seem that smart, so I thought why not test it out myself with some experiments?”

The experiment

During the second semester of 2023, Dr Alimardani was the subject coordinator of Criminal Law . He prepared the end-of-semester exam question and generated five answers for the question using five different versions of ChatGPT.?

He used five different prompt engineering techniques to get five more AI answers to enhance the response.?

“My research assistant and I hand-wrote the AI-generated answers in different exam booklets and used fake student names and numbers. These booklets were indistinguishable from the real ones,” Dr Armin Alimardani said.

After the Criminal Law exam was held at the end of the semester, Dr Alimardani mixed the AI generated papers with the real student papers and handed them to tutors for grading.

“Each tutor unknowingly received and marked two AI papers and my mission impossible was accomplished.”

Dr Alimardani said the exam was marked out of 60 and 225 students took the test. He said the average mark was about 40 (around 66 percent).

“For the first lot of AI papers which didn’t use any special prompt techniques, only two barely passed and the other three failed,” Armin Alimardani said.

“The best-performing paper was only better than 14.7 percent of the students. So this small sample suggests that if the students simply copied the exam question into one of the OpenAI models, they would have a 50 percent chance of passing.”

The other five papers that used prompt engineering tricks performed better.

Dr Alimardani said none of the tutors suspected any papers were AI-generated and were genuinely surprised when they found out.

“Three of the tutors admitted that even if the submissions were online, they wouldn’t have caught it. So if academics think they can spot an AI-generated paper, they should think again.”

This experiment raises a lot of questions. How professors can deduct AI if students are using it. GenAI has the potential to do certain things just like lawyers, but it can not completely replace them. Few exam papers are a clear example of it.

What are your thoughts on this experiment and how do you think AI will be advancing in the legal industry??

To read more about the study click. ?

#GenAI #AI #Law #Australian #LawStudents #LegalTech #GenAIinLaw #AustralianLaw #LawAndAI