Defensibility of Prompt Engineered Knowledge Validation Questions

Defending the approach and output of Knowledge Validation Questions is vital for Prompt Engineering. Luckily, by using good source material and sound prompt engineering principles, you can create clear and concise prompts that will reduce the possibility of hallucination by the AI. As the prompt engineer, you are responsible to ensure that the output is reasonably defensible, correct answers can be found in the supplied learning programs, and wrong answers are actually wrong. Simple modifications to your prompting logic can force the LLM to demonstrate and share its reasoning with you.

Below, find steps for a complex and mature prompt engineering approach that instructs the model in a more intentional manner. Note the full prompt is supplied at the end of the article.

1. Output any intermediate list building or select from list steps:

2. Output the list of themes
3. Select a theme at random and output the selected theme        

2. Share reasoning on meeting any stated criteria required.

4. Review the material and validate that there is enough material to 
make 3 related multiple choice questions related to this theme and 
output your reasoning for why you can proceed.         

3. Ensure answers can be found in the educational programming.

3. Validate that you can find the answer in the materials        

4. Share reasoning on decision making around question inclusion.

Question Set Rationale: why is this question related to the 
theme selected?
Question Rationale: what is your reasoning why is this a good question?        

5. Share justification and reference for the correct answer.

Question Correct Answer Rationale: why is this the correct answer?
Question Correct Answer Reference: where in the supplied reference 
pages is this answer?        

These modifications force the LLM to think and rationalize in decision making and, most importantly, to account for this process in the output. Again, the full input and output are below.

For further reading, see my articles Applying Prompt Engineering Principles While Creating Knowledge Validation Question and Applying Prompt Engineering Principles to Create Question Sets for SaaS Education.

Good luck with your exploration and follow for more educational content on creating educational content!

User
You are going to assume the role of a teacher.
You are going to review a page of content and the associated sub pages.
From the pages reviewed you will identify a theme that can be used 
to create a set of 3 multiple choice questions that are all related. 
The multiple choice questions should be related but do not use 
answer text from one question in the question portion of another question. 
You are going to present the questions in the format  provided in 
the backticks. 
You are going to follow these steps when creating question sets

1. Review the page of content and identify several themes that are 
either important or repeat
2. Output the list of themes
3. Select a theme at random and output the selected theme
4. Review the material and validate that there is enough material to 
make 3 related multiple choice questions related to this theme and 
output your reasoning for why you can proceed. 
5. Follow the steps for creating individual questions .

For each question perform the following steps:
1. Consider the theme and the pages referenced.
2. Create a question related to the theme 
3. Validate that you can find the answer in the materials
4. Validate that wrong answers are not obviously wrong
5. Output question 1 in the format specified within the four backticks. 
6. Repeat steps 1 through 5 two more times 

Page of content and associated sub pages to review:
https://docs.dremio.com/cloud/sonar/reflections/

````
Question Set Rationale: why is this question related to the theme selected?
Question Rationale: what is your reasoning why is this a good question?
Question: this is the question? 
Question Answers: the list of answer formatted as a bullet list
Question Correct Answer: the correct answer
Question Correct Answer Rationale: why is this the correct answer?
Question Correct Answer Reference: where in the supplied reference 
pages is this answer?
Question Difficulty: as compared to the other questions, 
how hard is this question? 
````

GPT
The page from Dremio Documentation discusses "Accelerating Queries with
 Reflections." Reflections are optimized materializations of source 
data or queries, similar to materialized views, derived from existing
tables or views. They help Dremio's query optimizer to accelerate 
queries against tables or views by partially or entirely satisfying 
the query using these reflections, instead of processing the raw data.
 This approach involves evaluating reflections that have at least 
one table in common with the query references, generating a 
cost-effective query plan using reflections, and executing this plan 
for faster query results. For more detailed information, please 
refer to the [Dremio Documentation]
(https://docs.dremio.com/cloud/sonar/reflections/).

Next, I will identify several themes from this page and the 
associated subpages.

After reviewing the Dremio Documentation pages on "Accelerating 
Queries with Reflections," several key themes can be identified:

1. **Types of Reflections**: This includes raw, aggregation, 
external, and starflake reflections, each with its specific
characteristics and uses.
2. **Creating and Editing Reflections**: The process and best practices 
involved in creating and editing both raw and aggregation reflections.
3. **Reflection Execution Strategy**: The strategy for executing
reflections, focusing on optimizing either the time needed to 
refresh or minimizing the number of files for better read performance.
4. **Benefits of Using Reflections**: The various advantages of
using different types of reflections, such as accelerating queries 
on unoptimized data, reducing resource use on production data 
sources, and improving the performance of complex queries.
5. **Reflections and Query Optimization**: How reflections are 
used by Dremio's query optimizer to improve query performance, 
including aspects like partitioning and incremental refreshes.

For the creation of the question set, I will randomly select one of 
these themes. Let's select the theme **"Types of Reflections"** 
for developing a set of related multiple-choice questions.

### Question Set Creation

Question 1
Question Set Rationale: This question focuses on understanding 
the different types of reflections available in Dremio and 
their characteristics.
Question Rationale: It's a fundamental question that tests basic 
knowledge of the types of reflections.
Question: What are the two primary types of reflections in Dremio?
Question Answers:
  - A) Raw and Starflake
  - B) External and Aggregation
  - C) Raw and Aggregation
  - D) Aggregation and Starflake
Question Correct Answer: C) Raw and Aggregation
Question Correct Answer Rationale: Raw and Aggregation are the two 
primary types of reflections in Dremio. Raw reflections consist of 
all rows and selected fields of the underlying table or view, 
while Aggregation reflections are used for BI-style queries 
involving groupings.
Question Correct Answer Reference: [Types of Reflections]
(https://docs.dremio.com/cloud/sonar/reflections/types-of-reflections.html)
Question Difficulty: Easy

Question 2
Question Set Rationale: This question delves deeper into the specific type
of reflection, focusing on the raw reflection and its attributes.
Question Rational*: It aims to assess a deeper understanding of 
raw reflections and their use cases.
Question: Which of the following is a benefit of using Raw 
Reflections in Dremio?
Question Answers:
  - A) Accelerates queries by pre-joining data from multiple sources
  - B) Stores pre-computed aggregations for dimensions in tables
  - C) Accelerates BI-style queries involving aggregations
  - D) Directly uses derived tables in a connected data source
Question Correct Answer: A) Accelerates queries by pre-joining data 
from multiple sources
Question Correct Answer Rationale: One of the benefits of using Raw
Reflections is that they can accelerate queries by pre-joining 
data from one or more sources, optimizing query performance.
Question Correct Answer Reference: [Types of Reflections]
(https://docs.dremio.com/cloud/sonar/reflections/types-of-reflections.html)
Question Difficulty: Medium

Question 3
Question Set Rationale: This question focuses on the application 
and benefits of Aggregation Reflections, another primary type of
reflection in Dremio.
Question Rationale: Understanding the specific use cases and advantages 
of Aggregation Reflections is essential for effective query optimization.
Question: What is the main use of Aggregation Reflections in Dremio?
Question Answers:
  - A) To optimize the performance of scan-intensive workloads
  - B) To accelerate queries that perform complex joins
  - C) To store pre-computed aggregations for dimensions in 
tables or views
  - D) To allow Dremio to execute most analytical queries without 
touching the data source
Question Correct Answer: C) To store pre-computed aggregations for 
dimensions in tables or views
Question Correct Answer Rationale: Aggregation Reflections are primarily
used to store pre-computed aggregations for dimensions in tables 
or views, thereby improving the efficiency of GROUP BY statements 
and SQL aggregation functions.
Question Correct Answer Reference: [Types of Reflections]
(https://docs.dremio.com/cloud/sonar/reflections/types-of-reflections.html)
Question Difficulty: Hard

These questions collectively cover the theme "Types of Reflections,"
testing basic understanding, specific features, and applications of
different types of reflections in Dremio.