Is it me or ChatGPT? Detecting AI-generated content
Dr. Nikolina Ljepava, PhD, FCIM, CDMP
Acting Dean - College of Business Administration, Chartered Marketer
As the semester ends, we all have been occupied with grading various projects and assignments. We know that students are always quick to adapt to new developments. With Chat GPTs and Generative AI's availability, the major concern immediately emerged in academia - how can we distinguish between assignments written by students and those written by AI? We have dealt with plagiarism for a long time and have very effective tools to identify plagiarized submissions. Additionally, a significant effort is placed into informing and educating our students about academic integrity. The availability of ChatGPT and similar tools, however, opened another Pandora's box of dealing with assignments generated by AI.?
As a response, Turnitin recently introduced an AI tool to detect the utilization of Chat GPT and similar tools, providing the AI detection score for every assignment submission. I have to admit that I did not face many challenges with my marketing students, probably because we spent much time discussing many different ways of using (and not using!) chat GPT and similar tools. However, in some cases, some not-so-high percentages of AI were showing while students were telling me they did not use any AI tools. And indeed, in some cases, using AI in particular sentences or paragraphs in the student assignments did not make sense, so I did a small experiment. I entered some of my writing (newsletter posts from here and segments of some articles) in the ZeroGPT tool. At the same time, I generated some content in ChatGPT to check if this will be fully recognized.
Text generated by ChatGPT was immediately recognized with no mistake. It seems that the tool was able to identify AI-generated content efficiently. Great! So what about human-written content?
At first, everything looked fine. I tested several paragraphs from this newsletter – which showed 0% AI. Then one paragraph from my recently published interview was highlighted as most likely AI generated! A paragraph where I wrote my perception of success, my opinion? Does the paragraph below look AI-generated? That was very strange, so I continued with testing.?
As I continued, things became stranger. AI tool recognized some specific sentences from my writing as AI written, even from the content written a few years ago. I thought about the possible reasons for this, so I initially suspected that it might be Grammarly's fault since I use it for proofreading, which might cause false positive results. Then I decided to test some even older writings of my own. I took the paragraph from my publication from 2013 and ran it through ZeroGPT. And wow! According to results, this too includes parts written by AI! An article that was written ten years ago!
So, what to say? I researched a bit on the topics and found out that although the tools can very well identify text which was indeed generated by AI, at the same time, other people were increasingly reporting similar things and false positive results in AI-generated content detection. One of the AI detection tools even flagged segments of the US Constitution as AI-generated (time travel, maybe?)! ?Furthermore, two weeks ago, Turnitin admitted that their AI detection tool, ?used so far on 38 million student papers, might be unreliable! Imagine those poor students blamed for cheating they never conducted!
领英推荐
As professors, we must remember that we usually know our students and are the best at assessing their writing skills. This common-sense approach is something we have used so far to fight what I believe to be the largest nuisance of academia – ordering assignments from essay and assignment writing companies.?
There is no reason to behave differently with AI-generated content. We need to be very cautious when assessing whether something is AI-generated. Many instructors recently faced similar challenges, with high AI recognition scores for student assessments, but we cannot blindly trust these tools. Otherwise, we risk being unfair and penalizing our students for no reason.
ChatGPT and similar tools are here to stay. And instead of fighting them, we should find ways to adopt them in our curriculum and teach students how to use them properly and ethically. They will use those or similar tools in their work one day, and we should guide them on how to do it properly and benefit from this. However, to do that, ?academic institutions must stop being afraid of those tools and embrace progress. ?
How can we get there? Tell me what you think in your comments.?
Artificially yours,
NikolinaGPT
Generating text since 2013
AI Strategist for Financial Services Companies | Keynote Speaker | Digital Transformation Leader | Salesforce Optimization Expert
11 个月Some researchers put in the US Declaration of Independence into a similar product to ZeroGPT and the system clearly declared this was written by an AI. This makes for great science fiction movies around time travel, but it also shows us how helpless we are. Instead of keeping students from using ChatGPT by threatening to find them out with a tool that does not work, we need to encourage its use and test the students differently. Yes, these may be oral tests and therefore cost more work, but they will truly test whether a student is in command of the material.
YOU - You in the past and You in the present.
Senior Specialist: Ministry of Education, UAE | Chartered Physiotherapist, UK | Frontiers Journal Editor | Health Education Consultant and International Speaker | Interdisciplinary researcher |
1 年Very insightful article Dr. Nikolina and highlights that the AI datasets and models m that are in current development are still somewhat too primitive in their current state to be used as a stand-alone detective/supportive tool. You raise points that can be collectively categorised under the overarching label of of ‘greater standardisation and regulation is required’. Before adopting AI methods in education assessment practices and procedures for ‘high stakes’ decision making purposes, concerted focus must be placed on assuring the quality of the AI materials is effectively benchmarked and regulated for specified uses in sectors and industries related to education. It would be interesting to know from yourself and others reading this - are you in agreement or disagreement with my critical thoughts and statements?
Dean - College of Business Administration, PMU, AACSB Accredited
1 年I like the article especially the view …to adopt AI in education and learn the student how to use it ethically …. Perhaps the teaching profession is.also changing .maybe soon there will be no teachers as they known today
L&D and Talent Development Manager | Certified EQ Coach ?? Corporate Trainer
1 年Your experiment is quite interesting and detailed. Thank you for sharing this article Dr. Nikolina Ljepava, PhD, FCIM, CDMP