Editor or Auditor?

"AI will change everything."

This statement is certainly no surprise; no deep, valuable, or unseen insight will be found here. Social media feeds are, in fact, right now full of this very sentiment, discussing the whys, the hows, the whos, and the what fors of the AI revolution. Despite, however, the seeming relentless feed of both news and workplace practice changes, a number of fundamental shifts have fallen through the cracks. It is my hope that with this short article, we can notice, plan for, and ultimately rectify that situation, such that "AI will change everything" ceases to become a platitude and instead becomes a meaningful guide as to how such change might take place and how we might deal with it.

We've spent countless words and hours discussing the impact of AI on education - specifically, the composition, English, humanities, and other essay-based classrooms - usually in the context of mitigating or eliminating student cheating. Over and over again, novel technologies (GPTZero, CopyLeaks, et al) proclaim to finally have the "solution" to determining AI-based writing versus human writing. Over and over again, these technologies are proven both highly fallible and, as you'll see in a moment, ultimately moot.

A Short Primer on Academic Writing

For those that have been out of academia for a while, allow me to begin by giving you a short recap of the writing process as it is taught to undergraduate students of Composition I and II, very often a required course at every institute of higher education, regardless of major. The first step, and the most laborious and time-consuming, is Research. If we were to draw a pie chart of "% time spent per task during the writing process," Research alone would consume over 50-60% of that time. This process includes, of course, reading, synthesizing, developing, and inventing ideas. It is approached with a hypothesis (a literal "hypothetical thesis") and the research process either confirms that hypothesis or rejects it, and summary writing may be performed as such. The writing process, as I'll be elucidating shortly, is, in fact, just the scientific process, a fact which will become important in a moment.

Once a hypothesis has been proven or disproven, it transforms into a thesis, effectively a short summary of the writing's contents. You can see the thesis of this article on display above (it's the sequence that begins with "Despite" and ends with "...might deal with it"). From here, the writing process itself commences, whilst you're already more than halfway through the time you would budget for such an endeavor. This phase begins with composition, either prewriting or launching directly into the text itself, and doesn't last long. In fact, if you're doing it properly, since you've already completed the research, the assembled ideas should come together quite rapidly. You should compose using a measure of Kahneman's System One (incidentally, this is where most students get tripped up; they try to compose each sentence perfectly using System Two, and the time investment composing in that manner is immense). Overall, composition should consume roughly 5% of your total writing time. This may seem like a small investment - and it is - but that's because the rest of your time should be spent on the next three phases of writing: Revision, Editing, and Proofreading (all of which are different). Revision comes first, and takes the longest of the retroactive writing skills. In Revision you edit major ideas, topics, and add or remove entire paragraphs or discussions. You may even need to go back to the Research well to add more information. Once Revision has been completed to your satisfaction, you move on to Editing. Revision adjusts the content of writing, while Editing adjusts its presentation. Here you examine features like sentence structure, clarity, tone, verbosity, use of jargon, and other metadata-specific writing qualities. Once edited, finish with Proofreading - a grammatical, style guide, and formatting review (which is also the shortest of the three). Together, these processes should require 25-30% of total writing time, with the majority given to Revision. In sum, the true writing process looks like this:

Phase I: Research (the scientific method), 50-60% of total time

Phase II: Composition, 5% of total time

Phase III: Post-Writing (Revision, Editing, Proofing), 25-30% of total time

With a little time left over for additional Revision or Editing. With that recap out of the way, let's move on to why I bothered discussing it at all.

Where AI Writing "Fits"

As you can probably guess from the above primer, "writing" isn't as simple as "writing." Students, interns, business associates use AI to write, but that doesn't absolve the entire writing process. In fact, if one were to place AI writing at its most efficient point within the writing process, it truly fits in only three places: Composition (5%), Editing (5-8%), and Proofing (~1%). This is because the bulk of research is still handled by Google and other database search engines like ProjectMUSE, JSTOR, and so on, while the AI-associated writing tasks comprise a mere, at most, 15% of the time spent writing an essay or journal article. Even the most effective prompting and most advanced LLMs find it difficult to bypass these hurdles. Of course, this isn't to say that they never will - in fact, I'd wager on it happening sooner over later - but it is to say that as currently constituted, despite the chaos that currently reigns in the academic world, the writing process is not particularly upended by students using LLMs.

A much deeper conversation may be had here, which I will only touch upon, as it is not the primary goal of the article: What do we grade, process or product? Do we care only about quality writing? If so, then we should furnish students with AI, train them on its use, and tell them to have at it. That will, of course, produce both the highest average quality and most efficient writing, freeing assessment time for other learning tasks. On the other hand, if we care about process, then we should outlaw AI at every level of education, as its use circumvents the purpose of the educational application itself.

To only partially offer an answer to this question, we must first ask ourselves why students write essays to begin with, and two primary reasons emerge: first, to learn the mechanics of good writing (a process subsumed entirely by AI), and second, to gauge a student's understanding of the material. In short, it is an imperfect "window into the learning mind of the student," wherein we make justified and meaningful guesses about the student's thought processes and conclusions. The thinking is what we're grading, but since we can't, of course, read minds, the writing is the only thing that gets us there. This argument, of course, says nothing about the tangential aspects of composition education (such as close reading ability, extracting, developing, and honing conclusions, developing arguments from prior knowledge, and so forth), but, at the very least, it should demonstrate that the process vs. product argument is a bridge that, at some point, we will inevitably be forced to cross when examining the role of AI in higher ed. This is also, not coincidentally, why AI Writing Detection Tools essentially have a moot application: "writing" is far more than just "writing."

Where AI Writing "Doesn't Fit"

A year-ish ago, I published an article in the SJU Humanities Review titled "Who Writes the Future? Prophets, Ghosts, and the New Romantic." In it, I made the argument that "A new parallel economy and artistic movement is coming," one I dubbed the "New Romantic" (in part because the classic Romantic literary and artistic movement was centered around the sublimity of nature against the backdrop of the industrial revolution, a partial rejection of that ethos), an economy which would see a schism between "real" or "human" writing versus "augmented" or "AI" writing. Of course, each economy would have its adherents, and for different reasons, but an excellent parallel here is in food markets, which have plenty of market room for both traditional and organic versions. Human-only writing, I argued, would be the "organic" writing.

If this supposition is true, then, as you may have guessed, it is necessary to tell the two apart, something that, as I eluded to earlier, as of now completely evades our grasp.

If you're doubting the efficacy of automated writing detectors, you're not alone. Research I conducted in December of 2022 showed that none of the supposed AI detectors currently available was any better than a coin flip in detecting writing generated by AI. Further, it also opened up another can of worms: how much AI is too much? Think back to the writing process I outlined above for a moment. If a student uses AI to generate thesis statements, then writes the essay on their own, is that outlawed or allowed? What about students that use AI to generate outlines, or revise, but still research and develop synthesis on their own? What about students with documented accommodations; is the use of AI permissible in those particular circumstances? Writing isn't as simple as "writing," like I posited earlier. It's a process, and AI fits into a relatively small portion of it. (Please note that prompt engineering and agent design allow LLMs to accommodate a much broader portion of the writing process; I'm speaking here in the general of how most students perceive or currently employ AI in their coursework).

Even Amazon has begun to develop a framework based on this question, as has the legal ramifications of Thaler v. Perlmutter. How much AI is too much? At what point does work cross over from being, in the words of both the Amazon policies and the Congressional Research Service, "AI-Created" (with required disclosure) or "AI-Assisted" (without such disclosure necessary). Is there a number? An amount? A particular step? The question that all of these governance, legal, and academic measures seem to be missing is the one I posed at the outset of the article: How can we tell? ..."Here thar be Juvenal."

The New Editor

There is a scientific basis for the unexplainable. Let's call it intuition, though it goes by many names, including a gut feeling or a notion. In the absence of automated technology to determine AI-composition versus not - and in the absence of the metaphysical argument about what even is AI-composition - then it seems fitting that human intuition is the virtue upon which we must rely. This means that the editor of the future, provided the parallel economy I envision will become a reality (and it already is, given the current state of disclosure policies) will necessarily become not only someone that edits for literary metadata like audience, situation, purpose, genre, or rhetorical method, but will also take on the role of auditor, the divisive judge whom may determine algorithmically-generated text versus that created by human hands. My personal view of the situation is that this will eventually become a valuable and required skill for anyone looking to enter a writing-based industry, and further, I argue that such a skill - let's call it Turing Auditing, based on the famous test - will become a part of the editing toolkit in such a ubiquitous manner as to be indiscernible. The editor of the future will necessarily become the auditor, and to that end, the writing course of now should strive to educate that future editor on how to accomplish this task.

Of course, precisely how to do that, short of exposing the student to huge amounts of writing, both AI and non-AI and then pointing out the difference and allowing intuition to bloom - seems at once both highly unreliable and highly unscientific. How does one educate for intuition? I'd love to hear thoughts, if you have any, because the editor of the future will need a well-sharpened, experienced mind, in the absence of automated detection systems, in order to divide the two types of writing.

Of course, if we want to integrate AI into the composition process in the first place, then such a skillset becomes moot.

I suppose only time will tell, but I trust my own intuition, based on previous historical trends, that such a skill will be necessary given the emergence of the parallel economy, only to eventually disperse as the generational shift makes AI less of a novelty and more of a customary application.