Can ChatGPT Write? ? Assessing AI Creative Outputs Like A Teacher

In a past life, for quite a while actually, I worked as an English, Maths, & Science tutor for primary and secondary school kids in London. It was fun because I got to give constructive feedback like “Learn how to spell, Andy” and “What???” and occasionally just “!” ????

Today, I’ve got my red pen out because we’ll be looking at just how good ChatGPT’s writing ability is. We writers can apparently call it quits now because ChadGPT is getting swole AND lyrical so I just want to make sure the succession planning is all in order.

—

So let’s start with a poem test. Say, a haiku about vinyl:

Oh man, screw this. I quit! See ya! ?? When did ChatGPT get so damn emotional?

...

Ok so this actually kinda works (other than that the middle line is 8 syllables rather than 7) because a haiku is intrinsically evocative due to the lack of words. All it’s done here is string some semantically related things together and we do the poetic filling-in ourselves. (For further reading on evocative-and-brief writing, see the Imagist Poets)

Let’s instead get it to do something a bit more demanding of talent: A Short Story.

ChatGPT writes a short story

At this point, the apprentices in the class will purse their mouths, raise their eyebrows, sniff out of their noses a bit and go “yeah, to be fair Mister George that’s actually done actually a pretty good job to be fair I got really emotional”. But this is also the part of the lesson where I start banging a book on the desk while shouting “no no NO. This is not good”. ??

It starts off fine, right? Your good ol’ “once-upon-a-time” setup, but Stephen King would definitely not be happy with the amount of adjectives used in the first sentence alone: a cozy corner, a bustling city, a dusty guitar – fine, but you’d better watch it, kid.

“Whispering forgotten tales of melodies”? Bit weird but ok, sure.

Ok cool so Sarah decides to finally be decisive and actually just learn the darn thing.

“Sarah’s fingers fumbled at first, but she persisted.” Ok... ?? I've not really been made to care at all so far.

“Chords became familiar friends” No. Chords are not your friends. Where are your real friends, Sarah? You fricken loser. ??

So basically Sarah learns literally all the chords until her fingers bleed before bothering to play her first song. This is a terrible way to learn an instrument. I suppose playing your first song under a starlit sky because, like, you’re a child and don’t have work in the morning is really nice and everything, and I’m glad that Sarah was able to make such a connection with music (it’s powerful stuff, brother) but there’s a lot to unpack here from a creative writing perspective.

ChatGPT’s been trained on the entirety of the internet so, considering the volume of the internet that is categorically bad writing, it’s no wonder that ChatGPT’s output is absolutely average.

There’s a massive reliance on clichés here: “bustling city”, “Day by day”, “Chords became familiar friends”, “Late Nights turned to early mornings”, “strummed her heart out” ??, “a vessel for her emotions”.

And who ever said that a short story has to have some big moralising ending? Don’t tell me what it’s about, make me understand what it’s “about”.

ChatGPT writes 'another' story

In another chat window, let’s try generating a new story.

Ok cool once upon a time ok, nice young woman ok, captivated, got it… wait.

Oh… er.. didn’t we just read this one? Oh ok this one is lessons, not strumming off on a balcony. Oh, that comes later...

–

Look, I mean, this ticks all the boxes of being a “different” story, but it is in essence the same story. It’s exactly the same arc, the same settings, the same person but with a different name. I’ve tried generating a third in yet another chat window and it’s fundamentally the same story. I could prove it but then this would look ugly so you’ll have to just trust me here –?or try it out yourself.

This isn’t generating anything worthwhile or new, it’s regenerating a mediocre core – like Trigger’s eternal broom for the folks here old enough to remember Only Fools and Horses.

Stories aren’t lists of events –?those are basically police reports. Stories are markers of thought. What “happens” in a story is important, sure, but it’s almost never as interesting as how it happens, why it happens, and where it happens.

Hell, what happens in a story is also usually not as interesting as what happened before the story.

GeorgeGPT writes a short story

In order to connect with anyone, you have to speak to them –?not rattle off a list of events. For fun, here's a 150-word story about learning to play the guitar that I wrote sleeplessly at 2:30am:

—

When the city heaved in late summer heat, past the rains, past the droughts, in the middle of the mosquitos, Sarah liked to take up learning the guitar again. She figured she never wanted to be one of those people with regrets – those who looked prunefaced and haggard in the mirrors of old age wondering What If.

She’d got the hang of tuning a while back, it was the tuning correctly she was still working on.

“Every Adder Does Go Back Elsewhere” she’d mutter to herself while turning the pegs freeform. This is, of course, not the commonly used mnemonic and she did, of course, decide the balcony would be the best place to take up learning properly. Where better than under a starlit sky to really start growing as a person, to wave those regrets away, to have the whole cosmos as your audience?

—

AND, in the light of morning I’m going to mark myself 7.5 out of 10 for this assignment and I’m going to mark ChatGPT 3 out of 10 overall.

Take note, class, EXCELLENT use of VOICE in MY stories ?? I'll award myself a Nobel prize, thank you very much!

But we’re here on LinkedIn to talk Business, and who’s paying for stories about Sarah the Loser? Nobody. So let’s assess ChatGPT’s writing ability in a real business use case.

How well does ChatGPT write in a business setting?

Let’s say I run a company that sells artisanal cheese.

It’s really fancy, really ripe stuff and most of our regular customers are folks who love cooking with it or just eating it – as gourmet as it gets. Sarah, who’s the brightest curd in our cheesery, has had the brilliant idea of writing a bunch of recipes which use our cheeses in order to “capture traffic on SEO”. I’m really proud of her; she’s really come into her own since she developed her new creative outlet and it seems to have become a real vessel for her emotions – so I’ve earmarked her for a promotion if this recipe stuff goes well.

We already have the common recipes that people make, but we need to make stories for them so that they can provide some emotional connection with our customers AND help the pages get found on Google. The problem is that none of us at the cheesery, unfortunately, are literate –?so Sarah’s also suggested we use ChatGPT to “write” the recipe stories and then we can maybe get someone to “make them nice”.

I love Sarah.

So let’s head over to ChatGPT dot com.

If I could read, I’d tell you that this is a 10/10 introduction and that I have no notes. It’s so great that there’s a symphony of flavours and textures that invites me to savour the joys of homemade food. I love that it encourages me to imagine crust. Oh, boy, I can’t wait for my senses to be awakened ??

?

You know, I’m touched by this. To be fair it’s actually done actually a pretty good job to be fair I got really emotional. Let’s do another!

Oh wow, not at all like the previous one! ?? I love that it welcomes me to the culinary journey! Oh my god I can’t wait for this delectable culinary journey and this symphony of colours, textures, and tastes. This is cooking, baby.

I’m astounded –?we basically never need to read or write again!

Ok let’s try yet another foooor… Lentil Soup?

Ok, right, welcome, imagine, symphony, journey… ok. ???

We’re getting the same Sarah/Emily problem here. This is essentially the same FORM of the Thing but with the details changed. We’re still going on a culinary voyage every time, still discovering the same platitudes.

Perhaps the problem is my prompting. By using the same type of prompt, ChadGPT is giving me a similar form of response.

Let’s try something different to jog it:

Cool, this is actually a lot better than the previous ones but I’m starting to think this story is about Sarah again –?though with the guitar replaced with Food. I really love this symphony of flavours motif… conduits of emotions… undeniable power of family bonds… cozy blankets… pinches of nostalgia…?

Is it worth me marking this one up? Ah, screw it, here you are:

So what would we grade this? 10/10? ??

?

Let’s try changing the prompt again because, ya’know, we have 100 cheese recipes to get through.

Ok now we’re back to 3/10. At least we’ve held on to the symphony of flavours inviting us to savour life’s simple pleasures etc. etc. etc. that’s the best bit!

Let’s try the original prompt but for Caprese Salad.

Right, ok I’m picturing myself sun drenched ok cherished memories inviting me to savour flavours and NO symphony this time wow! ??

...

Look. I think I’ve flogged the horse enough here.

Assuming you’re going to be able to generate genuinely engaging, genuinely useful creative content with ChatGPT (or any language model, for that matter) – is always going to result in horrendously mediocre things at best.

And no amount of prompt massaging is going to help out here.

It’s simply not feasible in a business context to change the prompt for every variation you need. The whole idea of using ChatGPT here is to enable you to generate these things at scale but, as we’ve seen, all it’s doing is generating the same thing with different details. You can scale this to infinity but you’ll still have a universe filled with shit.

—

Now, if you (yes you ??) generate some GPT outputs and think “I probably couldn’t have done that better myself” – you likely need the help of someone like me.

Let’s make work worth making ?? ??

In my next article, I’ll be expanding on why language models can’t write very well and why I don’t think they ever will.