Detecting AI-Generated Content: Generative AI Text Editing Excerpt
The following is an excerpt from AI Content for Editors: A Pocket Guide to Generative AI Text Editing - Kindle Edition, available now on Amazon.
Detecting AI-Generated Content
The first wave of ChatGPT brought this sense of awe and wonder as people realized, I can get this to write FOR me!
Worse, some jumped straight to, We don’t need writers anymore!?
That’s no exaggeration. I recently received a distressed message from a content strategist/editor I know and respect. The CEO had instructed her to let go of all their writers and use only AI-generated content. Just give it a quick check before publishing, the CEO had said. (They quit, and I applaud that person for leaving that short-sighted mess behind.)
Yes, you can fire all the writers, or pledge never to write again and have generative AI do the work for you. However, asking ChatGPT, Frase, Copymatic, and other AI writers to pen an article typically results in a predictable, formulaic, and frankly basic read.?
That’s the first issue — whether or not you choose AI content creation yourself. But as an editor, you may receive submissions from others where you have no direct line of sight into their creative process.?
What if your freelancers or in-house writers are using AI writers, and you aren’t even aware??
This is coming up in all kinds of publications and marketing teams right now. Content quality is increasingly important for ranking in Google’s search results, and your company or publication’s reputation is invaluable. Not to mention, no one loves a lawsuit. You cannot afford problematic content slipping through any cracks in your editorial process.?
So is it worth adding an AI detection tool to your workflow? Let’s start with whether they can even do the job they claim to do.
Do AI detection tools even work?
OpenAI launched what it calls its “AI Classifier” in January 2023, in response to educators’ complaints about ChatGPT’s AI-written essays and papers. However, by its own admission, OpenAI’s classifier correctly identified 26% of AI-written text (true positives) as “likely AI-written,” while incorrectly labeling human-written text as AI-written 9% of the time (false positives) in early testing.?
Another option, Writer.com’s AI Content Detector, lets you check only 1500 characters (~300 words) at a time and returns a percentage score. It’s important to note this isn’t the percentage of the content deemed AI-generated but the tool’s level of certainty that the passage was created by either a human or machine.?
Another caveat is that this tool is designed to help generative AI users beat detection and encourages users to “decide if you want to make adjustments before you publish.” I don’t love this suggestion that you only need to tweak text enough here and there to receive some arbitrary score, and you could then call it good content. That’s a race to the bottom.
Are these tools worth your time, and do they belong in your editing toolkit???
I’m not overly concerned with determining whether the text in front of me was AI-assisted, and maybe you don’t need to be, either. As this technology evolves and adoption increases, I expect more writers will use it in their creative process. Just as quickly as AI detection tools hit the market, generative AI advances and those detection tools become outdated. It seems a fool’s errand to spend time on AI detection as an editor for those reasons.?
Don’t take my word for it; people much more intelligent than me in this area have shared their thoughts. Melissa Heikkil?’s Why detecting AI-generated text is so difficult (and what to do about it) for MIT’s Technology Review is a good read if you’d like to learn more.
领英推荐
The results of Australian researchers Armin Alimardani and Emma A. Jane’s AI detection tool tests are also enlightening; you can check those out here.
Whether content is machine-generated is more contentious in some sectors than others, though. In academia, for example, the use of AI may be prohibited by school or board policy. Using one of the tools currently being marketed as a solution for detecting AI content may be tempting.?
However, these tools introduce risks for both parties – the editor/teacher and the writer/student – in that they may miss machine-generated content or erroneously flag human-created content as AI text. Any use of these tools must come with a healthy dose of caution.?
Sidenote: If you want to learn more about using generative AI in academia, read my article ‘How to Use AI Essay Writers Ethically & Without Cheating [With Prompts]’ on CreativeAIs.com.
In short, remember that many are using generative AI to write fluent text that reads as a human wrote it (although that’s not its best use case). Even OpenAI warns of its AI classifier/detection tool: “It should not be used as a primary decision-making tool, but instead as a complement to other methods of determining the source of a piece of text.”
[Editor's Note: OpenAI's AI Classifier tool has since been decommissioned.]
I think that’s excellent advice for editors, and it applies to all tools that claim to detect AI content – especially any that ask you to pay for the service.?
Even so, it can be useful to recognize AI content so you can ask your writers which tools they’re using and get that context around where the AI got its information and how it’s been treated.
Tips for sniffing out AI content
You have many tools aside from unproven AI detectors at your disposal, and they’re already part of your editing arsenal. Generative AI text is plagued by many of the same issues as human writing:
These are the issues I’m looking for in my edits, and technological advances haven’t changed that. I don’t care if your essay was written by a sixth-grader, ChatGPT, the neighbor’s cat, or a Pulitzer winner. Poor writing is poor writing.?
ChatGPT, in particular, tends to regurgitate words from the prompt in unnatural ways, and present text in simplistic formats such as numbered lists. Here are a few examples for comparison. Pay attention to how ChatGPT structures an article...
Keep reading in AI Content for Editors: A Pocket Guide to Generative AI Text Editing - Kindle Edition, available now on Amazon.
About the Author
Miranda L. Miller is a Canadian business and technical writer/editor, content strategist, and SEO. For nearly 20 years, she's been the ghost pen and keyboard behind thousands of articles, ebooks, and whitepapers in B2B, SaaS, and tech. A part-time digital nomad, Miranda spends the warm part of the year at her home base on Georgian Bay with her husband, sons, and English Shepherd. The rest of her year is spent working remote while chasing sunshine and happiness around the planet. You can connect with her at miranda-miller.com.