Generating Assessments using Gen AI -- Part 2

This is in continuation to the article https://www.dhirubhai.net/feed/update/urn:li:ugcPost:7247252001732931584/

The previous article shared a perspective on Generation of Assessments.

As much as manual Generation of Assessments takes time, manual Validation of Assessments takes an equal amount of time.

Just like Generation, Validation can also be automated. One can apply multiple levels of Validations. One can apply either one of them or all of them and then proceed to Human in Loop. This will save a considerable time and effort.

On an average of multiple runs, the observation is 90% of Generated Assessments in Step 1 gets cleared in Step 2. The mismatch can be routed to the Human in Loop for further validation. A complex workflow can be built to have multiple validation pipelines thereby reducing the burden on Human in Loop intervention. The critical aspect in each scenario is the articulation of reasoning, this will further help Human in Loop.

Here is a look at the prompt for validation.

In addition to the options generated by Step 1, the following are included in Step 2 and seek LLM to revert to one of the options in case there is a dilemma

• I do not know
• The right option is not listed

var systemPromptForValidation = `You are an expert in AI literature. Answer the following questions to the best of your knowledge. 
You will be prompted with a set of questions and a set of options for each question, choose only one of the right options for each question 
that accurately reflects the ask and also articulate why its the right answer. If you do not know the answer to any question, 
please say I do not know. If the right accurate option for the question does not exist, please say The right option is not listed`

func getPromptRefinedforValidation(allQuizes []assessmentDataforMap) string {

	var promptTemplate string

	var stepsSequencePrompt string

	stepsPrompt := `

	The following are the Questions and the Options.
	Questions are delimited by $$$
	Options are delimited by ###

	$$$
	Question %d:
	%s
	$$$
	###
	Options:
	%s
	%s
	%s
	%s
	I do not know
	The right option is not listed
	###`

	for outIdx := 0; outIdx < len(allQuizes); outIdx++ {
		stepsPromptInit := fmt.Sprintf(stepsPrompt, outIdx, allQuizes[outIdx].Question, allQuizes[outIdx].AllOptions[0],
			allQuizes[outIdx].AllOptions[1], allQuizes[outIdx].AllOptions[2], allQuizes[outIdx].AllOptions[3])
		stepsSequencePrompt = stepsSequencePrompt + stepsPromptInit
	}

	outputforPrompt := `
	
	The following will be part of the results
	1) Question as Question
	2) Answer as ValidatedAnswer
	3) Reasoning as ValidatedReasoning

	Return the results using this JSON schema:
	ValidatedAssessment = {
	'Question' : string
	'ValidatedAnswer': string
	'ValidatedReasoning': string
	}
	Return: Array<ValidatedAssessment>`

	promptTemplate = fmt.Sprintf("\n  %s \n %s", stepsSequencePrompt, outputforPrompt)

	return promptTemplate

}        

Here is the code base in golang https://github.com/elmoOreo/goGeminiAssessmentGen. Due to some issues with goroutine implementation I have consciously put a sleep of 60 seconds to let the goroutine execution finish smoothly so that there are no concurrent access to maps.