Experiences of Using Generative AI

I got an email from Devesh Khanal at Grow & Convert last week, in which he said:

AI writing is controversial. But besides anecdotes, we haven't seen a study on how much people are actually using AI writing tools and how they're using it. So we decided to do that study ourselves.

Following the link to the survey, I discovered a carefully thought-out set of questions around “Using AI-assistance in Writing Content”. And I do mean seriously good questions; ones that deserve long and detailed answers.

Most of the questions are open-end with a free-format space for text because Grow & Convert believes in actually listening to people. Ain’t that refreshing in this day and age? As I started filling out the form, I came to the view that this content might be interesting to a wider audience - so here it is. Help yourself.

The Headings are the questions from Devesh, followed by my experiences of using generative AI for writing Marketing Content.

Have you published AI-assisted content?at all, ever??

Yes lots. I started experimenting with generative AI in December 2022.

How much AI-assisted content have you published in total?

Well over 15 pages. Multiple articles and professional projects for clients. I use it several times a week.

Why did you start using AI to help with content production?

The original idea was to find out what it could do, what it was capable of. To run a reality check on the hype. And – to be totally honest – to figure out if it was likely to be an asset or a threat to my own role as a professional writer.

The first impression was "wow - this actually sounds like a person writing: the sentences make sense; the ideas are structured as paragraphs".

After a while you realise that although this AI system has wide scope and considerable depth of knowledge, the personality is very immature. They can't be trusted to follow instructions without frequent reminders about the objective and working parameters. They do not always research diligently or report accurately.

Significantly, an AI has no built-in moral compass and no sense of responsibility for the truth. As a user and communications professional you have to carefully assess what you will actually use generative AI for.

How exactly did you use AI to help write? What was your process like?

I’ve explored a wide range of uses for generative AI to learn what it can do. Some of the things I’ve tried include:

Understanding the parameters of the AI

• Finding sources for information - e.g. “find internet webpage sources for [subject x]” … the results were useless: ChatGPT invented links and told me they were ‘hypothetical examples’ before finally admitting "I’m unable to directly browse the internet to find real-time sources". Duh.
• Memory & continuity of chats - e.g. “Hallo again! Remember our chat yesterday?” … answer: no.
• Scope of data access - e.g. “Can you access my LinkedIn Posts or Articles?” … answer: no.
• Creative thinking and thought experiments - e.g. this article titled The Duck Test .
• Generating new ideas - e.g. “Suggest an alternative to the Dewey Decimal system for library classification and explain why you chose it” … answer: "a human-curated version of Google’s Knowledge Graph" is a variation on an existing concept rather than something new.

Research

• To identify key issues in a subject - e.g. “list the essential and optional fields in a Master Supplier Record typically required for the ‘order to cash’ process”. Answer quality: good.
• To summarise the key facts - e.g. “summarise Cooper’s stage-gate process”. Or: "explain 'packaging component characterization' in the pharma industry".
• To explain a subject I know nothing about - e.g. “how does a hydrogen engine work?” … answer: hydrogen is the fuel for an electric motor, not a type of engine. Or: “explain why dew point is an issue for compressed air applications” … answer: five concise reasons given.

Text Creation

• Creative writing - e.g. ”write an article on Dwarf Courtship Rituals in the style of Terry Pratchett”
• Reviewing material that I provided to the system - e.g. “review, synthesise and summarise the attached 61 customer feedback comments gathered from a website” … data input is very limited – and it’s really annoying to enter data into ChatGPT in small batches.
• Interpretation and discussion of historical fact - e.g. "what facts support the proposition that 'the content of the Rosetta Stone is a press release'?" … answer quality was 5 good out of 6 facts.
• Information preparation and presentation - e.g. “list the uses cases for [specialised technology] and give detailed examples” … “present the examples as rows in a table with two columns: use case and data to be analysed” … “now add a column to show analytical techniques".

Editing

• Editing text - e.g. “Replace the passive with active voice in the attached text” … some decent suggestions here.
• Improving readability - e.g. “calculate the Fleisch Reading Score for the attached text, then re-write as necessary to achieve a score of 60” … Chat GPT hit the target, but it didn’t sound like my writing.

Mathematics

• Creating dummy data (for generating graphics via excel) - e.g. “the Deming red bead experiment: 1000 beads; 90% white, 10% red. create a set of 100 random dummy values with average of 10% to represent a stable process: also show the upper and lower control limits, what is the std deviation?" … dummy data and answers delivered as required.
• Calculations for graphics - e.g. “a pentagram has 5 equal sides, each of 6 cm long: it fits into a square. What size is the square? Give answer only.” … this saved time in building a powerpoint.
• Typography for design - e.g. “34 rows of text set at 1.2 line spacing in a design height of 14 cm is what point size?" … answer: 9.73pt, so I used 10pt and tweaked the design.
• Speculative maths - e.g. “assume Archimedes weighs 100kg and uses the moon as fulcrum, how long does the lever have to be for Archimedes to move the world 1 metre with his own weight?" … answer: six hundred and nine quintillion metres.

Which AI tools did you use?

I started using ChatGPT in February 2023 and signed up for a paid subscription from March 2023, mainly to avoid annoying ‘system unavailable’ issues. It’s currently my primary go-to generative AI tool. I tried three to four others in 4Q2022, but quickly settled on ChatGPT in 1Q2023.

In addition, I have tested the Bing-Copilot search-thing and found it flaky at best.

What kind of content did you produce with AI help?

I’ve tried quite a lot of use cases ??

• Article plans, outlines and structures - AI is a fast way to start the creative process, with content then written by myself.
• Subject / Article Research - AI can provide a balanced synthesis because it derives information from multiple sources. However: using AI requires an editor to apply a healthy shot of scepticism and critical frame of mind.
• Editing and text improvement - as described above.
• SEO optimisation of edited writing - using ChatGPT to edit headlines, tweak text for keyword density, suggest the Title Tag and Meta Description.
• Graphics - Only use AI if you’re willing to accept a very loose fit to your description. The more I try to get an accurate fit with my expectations, the more frustrated and annoyed I get.
• Programming - I’ve also experimented with ChatGPT’s ability to generate both HTML and CSS with very mixed quality of results. The lower your expectations are, the better. A lot of the time, the real issue is my inability to communicate in a way that the AI understands me.

I know exactly what I want, but the AI is too stupid to understand me.

How much editing did you have to do to AI-written content??

Loads. Every time. IMO, generative AI is only capable of churning out raw text that needs heavy copy-editing to get a truly individual voice.

As a matter of ‘best practice’ I now refuse to accept any output from an AI without further qualification. For example: the answer to a question may be detailed and therefore – on the surface – highly credible. But to be sure, my next instruction is usually “re-check all statements in the previous answer for accuracy, validity and safety, then confirm; or re-write using accurate language”. ChatGPT typically gives an updated, verified explanation.

What results did AI-assisted content get you? Did it produce more/better rankings, traffic, or conversions? How did it compare to human written content?

In a February 2024 LinkedIn article I wrote:

“the AI reminds me of a spotty teenage office trainee: it is immature, temperamental and unreliable. Even the simplest tasks need careful explanation and supervision. It is most definitely not yet ready to be integrated into production systems.

But what an ego! It has the cheek to think it is already mature enough to contribute to the real world! No way, junior; there is a lot still to be learned. Artificial? Yes. Intelligent? Hmmm. I’m still not convinced."

I haven't seen anything to change my view since then.

Do you think you will use more or less AI to help write content in the next year??

I think of and use generative AI the way I think of and use a pencil / ball pen / fountain pen ... i.e. interchangeably; or switch between paper / powerpoint / word doc. / spreadsheet. In other words it’s just another implement in the creative toolbox.

Which implement do I reach for first? It all depends on the job at hand. Most jobs require multiple tools in a series of operations from start to finish. I'm happy to use an AI to assist with design and/or shaping and/or creating and/or polishing and finishing, but ...

Would I delegate an entire job to an AI from start to finish and expect to get a result that will meet my quality criteria without further intervention? No way.

IMO: all AI systems need human intervention. (A lot more about that here. )

As a reader, do you think you can spot AI written content versus human written?

As of today, yes. There are some ghastly give-aways. Redundant verbiage like:

• “In the white-hot heat of today's business world, it's no secret that …”
• “In the second part of our journey, I want to delve deeper …”
• “In the vast landscape of …”

I’m all in favour of AIs learning to become concise and precise, but human writing goes far beyond what AIs can do.

AIs can only synthesise or re-phrase what already exists. Genuinely new insights only come from human beings.

As a searcher on Google, do you prefer to read AI written responses to your queries or human written?

If I want pointers to information, then machine interaction (like a list of pre-selected links) is acceptable.

But for actual answers that deliver information I prefer human responses, please. Why? Because the context of the question usually needs to be clarified to ensure the answer is relevant.

Speaking as a customer and consumer, my message to Product Managers and Customer Service Managers is to think very carefully before entrusting the reputation of your business to automated message generators. The Air Canada story should be a clear enough warning.

Safety setting: don't use unsupervised AI in public-facing roles.

What most closely matches the job role you had when you used AI to help produce content?

My job roles when using generative AI cover a very wide scope. I’m generally working as a consultant to an in-house marketing team, but the actual 'roles' cover the entire process from Commissioning Editor, through Researcher, Writer, Copy Editor, Reviewer & Approver, SEO expert, Publisher and Reporter of Results.

How do you use AI?

I'm curious to hear how you have used Generative AI and on which points you agree/disagree.

By the way ...

If you'd like to add your own contributions to the Grow & Convert survey, the link is here: https://docs.google.com/forms/d/e/1FAIpQLScHCJB2O5pGXU51WObylx-jMZUAUYZ_1TB3h3y3pjy9i8XkGQ/viewform