Experiments with ChatGPT: Short vs. Long Prompts (Part 1)

Prompt engineering is an experimental and iterative process - not unlike marketing.

You try different things, you document what works best, and you build off of that. Test, learn, rerun. It’s the same thinking used in digital marketing.

The thought of having my job automated and replaced by AI used to make me toss and turn at night. Now, ChatGPT has become such an asset to me that I couldn’t imagine running my business without it.

I now rely on ChatGPT to help me run my business. It is the only way I can reliably produce quality content at scale on my own, without hiring an assistant.

I use ChatGPT to automate the parts of my content creation process that take up the most time but produce the least results, so I can just focus on writing good content. Specifically, what I use ChatGPT for most often, by far and away, is to make my blog post outlines.

Creating the outlines for my blog posts is by far and away the most time-consuming part of my writing process. It's a major part of my research process for content creation.

Researching my topic and compiling my research notes into an outline used to take up as much time as writing the post itself. It’s hard to run a content marketing business this way when you have startup clients in niches from finance to web development to HR and recruiting - it’s hard to gain true subject matter expertise in each.

With ChatGPT’s help - or Bradbury, as I like to call him, I can now learn everything I need to know about my topic and have a usable outline in about 30 minutes.

Writing a publishable and optimized blog post used to take me a full day’s work. Now, I can have something I’m proud to show my client by lunchtime.

Over time, I documented the ChatGPT prompts I use for different parts of my content marketing business - creating outlines, personalizing cold emails, brainstorming content ideas - into documented templates that I can use over and over again.

There’s only one problem. I’m a really wordy writer, and I tend to over-complicate everything. That means that my prompt templates are pretty bloated, and could use some fat trimming.

So, I decided to test my ChatGPT prompts and find ways to optimize them. Bradbury, of course, will help me. I’ll share my results with you here on my LinkedIn.

Bradbury has helped me to structure the tests, while I execute them. Then, after they’re done, Bradbury will help me to analyze and interpret the results. Then, I’ll use my findings to find ways to improve my prompts.

I’ll start with a basic one. Do I make my prompts shorter, or longer? Read on to find out, or jump to part 2 to see how it ended.

Do Short Prompts or Long Prompts Yield Better Results?

My current prompt sequence for creating outlines takes place in 5 stages:

1. Understand the Task: Give Bradbury a role with subject matter expertise on X topic, tell him I’m outlining a blog post on the topic, lay out the stages, and confirm for understanding
2. Stage 2 - Understand the Client: Give Bradbury samples of the client’s web pages and blog posts so he can understand their brand voice and writing style.
3. Stage 3 - Competitor/SERP Analysis: Analyze the competing pages ranking for my target keyword, and find ways to approve them
4. Stage 4 - Research Questions: Ask research questions about my topic. This is both for my benefit to understand the topic, but also to train ChatGPT to include the kinds of information I want for the outline
5. Stage 5 - Make a Detailed Outline: I have Bradbury make the outline.

This is a pretty lengthy process, and my first stage prompt in particular is pretty bulky. I wonder if there’s a way to trim it down?

My first ChatGPT experiment will test short prompts against long prompts.

My hypothesis: A shorter prompt can be just as effective in making a comprehensive, detailed, useable outline.

Control Group and Variables

For this experiment, I will use my existing prompt for having ChatGPT do my research and outline process as my control group, or the long prompt.

Here are the variable groups, the short prompts, that I tested with:

Short Prompt 1: Create an outline for a blog post about conducting a performance review.

Explanation: This prompt is extremely concise and general. It measures how well ChatGPT can generate an outline without specific guidance or details.

Short Prompt 2: Outline a blog post on performance review methods, including key steps and best practices.

Explanation: The prompt adds a bit more detail by mentioning the focus on methods, key steps, and best practices. It measures how a slight increase in specificity affects the generated outline.

Short Prompt 3 (2 stages):

1. What are the main topics to cover in a blog post about performance reviews?
2. Based on the topics, create an outline including subheadings and key points for each section.

Explanation: With this experiment, we will test stages. This two-stage prompt breaks down the task into identifying the main topics first and then creating an outline. It measures how staging the prompt affects the coherence and detail of the final outline.

Short Prompt 4: Create an outline for a blog post about conducting a performance review, focusing on challenges and solutions.

Explanation: This prompt introduces a thematic focus on challenges and solutions. It measures how introducing a theme affects the direction and content of the outline.

Short Prompt 5: Outline a blog post about performance reviews, targeting HR professionals. Include sections on preparation, execution, and follow-up.

Explanation: This prompt specifies the target audience (HR professionals) and key sections. It measures how defining the audience and main sections influences the outline’s structure and relevance.

These short prompts all vary slightly in focus, theme, and structure. By adding more context to each experiment, I hope to arrive at an outline I can use with minimal additional research.

Methodology

Here’s how it worked. Bradbury, my ChatGPT assistant, would structure the experiments. He came up with the prompts, and together we came up with a testing process.

I use each prompt to make an outline, starting with the control group (my standard prompt template - long prompt) and going down the list of variations. I used web-enabled ChatGPT with the VoxScript and Wolfram plugins installed for each one.

The outline would be for a blog post about “How to Conduct a Performance Review” for an HR SaaS platform. I record how long it takes to produce each outline. Then, I give Bradbury each outline and have him evaluate the outlines in comparison to one another.

Some of the criteria we looked at were:

• Completeness: Does the outline cover the topic comprehensively? I should be able to spend no more than 5 minutes skimming the outline before moving straight into the writing portion
• Target Audience Alignment: How closely does it align with the target audience of HR and recruiting professionals?
• Usability: How much do I need to modify the outline before writing the draft? Am I able to write the blog post based on the outline without doing any further Googling on my own?
• Time Recorded: How long does it take to produce a usable outline using the prompt?

To ensure consistency, I used the same brief, the same keyword, the same company, and the same everything else.

After I run the final prompt, I run the results through Bradbury, he offers his observations about what each outline does well. Then together we form our conclusions and determine whether my original hypothesis was proven or disproven.

Finally, I make decisions about how to alter my original ChatGPT prompt sequence for creating blog post outlines.

I want to make my process faster, better, and easier. Test, learn, rerun. That is the creed of the marketer.

Follow me to see how the experiment turns out in part 2, coming soon!