A test harness for #genAI generated code: Using the automated output refinement pattern to enhance and evaluate models

I am using this both in my teaching and in Erdos research Labs to help low code developers to evaluate models

Background

To test AI generated code, I am creating? a test harness using the following strategy which is an adaptation of the automated output refinement pattern from Yi Zhou ’s comprehensive book on Prompt design patterns

The automated output refinement pattern as discussed in the book is itself a variant of the? Self-Refine: Improving Model Quality Under Distribution Shift via Self-Refinement

Notes:

1) while the approach is prompt based, it needs an understanding of the machine learning process and workflow

2) This is not exactly a test harness I know - but you could adapt it as one - by introducing a more structured approach and testing code in stubs. Besides, I liked the pic of huskies running in the harness :)

Objective

There are many approaches to validate AI generated text but not many for AI generated code

My objective is a to create a general test harness for AI generated? code?

I am using this for ML code but I think it should work for anything

Process

Its best to explain it in code

basic prompt is (after running mnist)

Now, I want you to act as an expert AI developer/ Engineer. to iteratively refine the code. At each stage provide a focused, constructive critique, explain the critique, implement it in the code and then provide the output. Run this refinement iteration process three times.

The only problem is because it does not have keras it cannot run the code but still provides indicative output

See the full conversation and output here ?

If you found this useful, you can sign up for my book

If you are a non developer and want to learn AI with me, please see Erdos Research Labs

You can meet me and our team at our Oxford AI summit

If you would like to study with me, see our courses

Low code AI course at the university of oxford? for non developers

AI and digital twins

Thanks to Anjali Jain Aishwarya Naresh Reganti for their feedback

Image source: huskies