6 Questions to Prevent ChatGPT Shiny Object Syndrome

New technology like ChatGPT is exciting because it provides new opportunities.?

Every day, professionals in different industries, including Learning and Development (L&D), are sharing creative ways to use the tool. But if we aren’t careful, our excitement and enthusiasm can quickly turn to shiny object syndrome. Since there are so many possibilities, we often want to try them all which can lead us on a wild goose chase and can cause us to overlook potential risks.?

That doesn’t mean we shouldn’t use ChatGPT for L&D. We just need a strategy built on evidence-based practices. Taking a strategic, systematic approach to integrating ChatGPT in our L&D processes can increase effectiveness and manage risks ChatGPT presents.?

To prevent shiny object syndrome with ChatGPT, ask yourself these 6 questions before adding ChatGPT to your L&D processes.

1. What L&D Process Problem Will ChatGPT Solve?

From creating e-learning scripts to developing learning plans, ChatGPT is very versatile. Narrowing it down to a single process helps keep you focused and better able to adapt to challenges that arise. It also allows you to evaluate other non-ChatGPT related barriers that may need to be addressed as you adapt your process.

A common mistake with any workflow optimization or performance improvement initiative is jumping to solutions before understanding the problem we are trying to solve. With ChatGPT (a solution), it's important to recognize that, like any solution, it’s just the means to an end. Before focusing on the means, we need to understand the need.

What problem are you trying to solve with ChatGPT? Is it too much development time? Delay or wait time while working with Subject Matter Experts? Content quality concerns? Identifying the specific problem you want to solve can help you find the most effective ways and places to integrate ChatGPT into your process.

2. What ChatGPT Use Case Will You Use?

Once you’ve picked a problem to solve, you can identify your ChatGPT use case. There are many ways to improve L&D processes with ChatGPT. For example, if your problem is that you don’t meet targeted time frames for summarizing and posting questions and answers from synchronous training sessions, your use case could be using ChatGPT to summarize Q&A content from transcripts.

Consider different ChatGPT use cases and pick one that aligns with the L&D problem and process you previously identified. Evaluate your potential use case to ensure it meets your requirements and is a good fit for your organization and team. Some factors you may want to consider include feasibility, cost, resources, and time requirements.

3. What Does ChatGPT Success Look Like for Your L&D Process?

Like anything, ChatGPT results can, and should, be measured. Focusing on measurable outcomes can help you prioritize potential ChatGPT uses, ensure alignment with other L&D team members or leaders, and help you evaluate results. Unlike your learning outcomes which focus on your learner, these goals should be specific to your L&D processes.?

The measurements you select should tie directly to your process and the problem you are trying to solve with ChatGPT. The table below provides examples of this.

Some additional examples of L&D ChatGPT measurements to consider include:

• Consistency in L&D team outputs
• Learner enrollment rates
• New activity ideas generated

4. What Are The Most Effective L&D ChatGPT Prompts?

Your inputs (prompts) determine the quality of your ChatGPT outputs (responses). All prompts you use should be clear, concise, and well-structured. They should also include good background information about your learners, cultural considerations, training objectives, topic, and any other requirements. You can also include your expectations about the instruction design framework or methodology to be used, the writing tone or style, and more.

Short vs. Long ChatGPT Prompts

While there are some debates about whether short prompts are better than long and vice versa, it really depends on your situation, needs, and preferences.

Short ChatGPT Prompts for L&D

Shorter ChatGPT prompts (3-5 sentences max) are generally good for limited scope questions. For example, questions about a specific topic with a single objective. These prompts usually take less processing time and allow you to pinpoint where prompt improvements should be made if the response is vague or not aligned with your intentions. Once you receive an initial response, you can build upon it with another prompt to deep dive and build the output you need.

Example Short Instructional Design ChatGPT Prompts

• "What are the common challenges faced in [topic] and how can they be overcome?"
• "How can [topic] be adapted for [specific audience/context]?"
• "Can you summarize the key takeaways from [research paper/book] on [topic]?"
• "Please provide some interactive and engaging small group activities for virtual training on [topic]?"
• "Use the following content to create a checklist with key steps on [topics]."

Long ChatGPT Prompts for L&D

The alternative to short prompts is longer, detailed prompts. In these, you can instruct ChatGPT in depth about your topic, audience, requirements, and expectations. While it may take longer to get a response and require multiple attempts to refine your prompt, ChatGPT can produce some very impressive outputs.

Example Long Instructional Design ChatGPT Prompt

"Imagine you are an instructional designer tasked with creating a training lesson plan for a new software program that is being introduced to a group of employees at a large corporation. The employees have varying levels of technical expertise and will be using the software in their daily work tasks.

The lesson plan should be 1 hour in length and should cover the following topics:

1. Introduction to the software and its main features
2. Navigation of the user interface
3. Creating and organizing files
4. Performing basic tasks such as formatting text and inserting images
5. Tips and tricks for increased productivity

The lesson plan should be designed for adult learners and should include a mix of lecture, demonstration, and hands-on practice. Assessment should be built into the lesson to ensure that employees have a good understanding of the software and its features.

In your response, please provide a detailed lesson plan that follows Gagne's 9 events of instruction and includes the following:

1. Learning objectives
2. Instructional strategies and activities
3. Assessment methods
4. Materials and resources needed
5. Evaluation plan to assess the effectiveness of the training

Please note that the lesson plan should be visually appealing and easy to follow for both the trainer and the learners."

Systematically Prompting ChatGPT for L&D

As you test different prompts for your process, try to document and evaluate which result in the best outcomes. This ensures you will be able to replicate outcomes in the future and make continuous improvements. Although ChatGPT produces different responses each time you enter a prompt, the response will always be based on your inputs.

Consider creating your own organizational prompt library specific to your L&D processes. Keeping these centralized and maintained will help create consistent results for your team and stakeholders.?

5. How Will You Validate ChatGPT Outputs?

How often do humans answer questions correctly? How does it compare with ChatGPT?

One study found that InstructGPT, which is used by ChatGPT, was truthful on 41% of questions while human performance (with access to the internet) was 94%. If you’ve experienced ChatGPT inaccuracies and hallucinations first hand, this probably seems familiar.

When it comes to training and learning content creation, accuracy and completeness are key.??One of the biggest ChatGPT limitations in an instructional design context is its lack of content credibility. As a language model, ChatGPT imitates content made by humans, including ones based on misconceptions, bias, or falsehoods. While the information generated is often impressive and helpful, ChatGPT cannot consistently cite sources for the information provided.?

Some specific steps you should consider for validating your learning content include:

• Ensure content is reviewed and verified by subject matter experts
• Cross-reference information from different sources to ensure that the information is consistent and accurate
• Ensure training content aligns with your learning objectives?
• Verify content meets your organization’s standards and style guide
• Review training content for accuracy, relevance, and completeness
• Check that the training content is free of inconsistencies and biases
• Ensure the tone and style is appropriate for your audience
• Have other instructional designers or trainers review the content and provide feedback
• Conduct user testing of your learning content and use the feedback to make improvements

Depending on the content you are creating, you may not need to complete all of the steps listed, but these are areas to consider.

6. How Will You Evaluate ChatGPT Performance?

Using the measures of success you previously set, you can create an evaluation plan to ensure ChatGPT is producing the expected results. Time savings is often a big benefit of ChatGPT, but it’s also important to consider the time spent with ChatGPT when you compare metrics. This includes prompt development, prompt adjustments, response wait time, system down time, and time spent editing and validating outputs.

Comparing these metrics can help you evaluate the effectiveness of using ChatGPT in your process and continuously improve the process.?

In creating this article, I used ChatGPT to brainstorm and outline some content. ChatGPT also provided some of the example prompts used in the article which were tested and refined.