Metaprompt: guide an AI's behavior and improve performance

What is metaprompt

Crafting an effective system message, also known as a metaprompt, is crucial for guiding an AI system's behavior and enhancing its performance. This article describes what metaprompt is and guides you through a structured framework with example templates offering a foundation to increase the accuracy and grounding of responses generated by Large Language Models (LLMs).

System Framework

LLM system message framework from Microsoft encompasses four key concepts to tailor the AI's functionality to specific needs.

Firstly, defining the model's profile involves outlining the tasks it should perform, the intended users, and the expected inputs and outputs, ensuring the model operates within its capabilities without reliance on external tools unless specified. Secondly, the model's output format should be clearly defined, considering language, syntax, and any desired formatting to enhance readability and user interaction. To further refine the AI's behavior, providing examples that demonstrate the intended model behavior is recommended, especially for complex prompts. This approach helps the AI understand the desired outcomes and the reasoning process behind them. Additionally, establishing safety and behavioral guardrails is crucial to mitigate potential harms. By incorporating specific lines and examples into the system message, developers can guide the AI to avoid harmful content, copyright infringements or manipulative behaviors.

Limitations

It's essential to understand that while these templates and guidelines can improve interaction with LLMs, validation of the AI-generated responses remains important. The effectiveness of a metaprompt in one scenario may not necessarily translate to broader set of examples.

Example of metaprompt for a retail company

Now, let's have a look how the system message framework can be used in practice. I am using the example of system message for a retail company shown below and defining my system message and prompt in Azure OpenAI Studio -> Chat playground. This is a great way to test different system messages and see how behaviour changes afterwards.

First, define a system message in the Setup pane:

Next, let's try a simple prompt and ask about tents I can use in the winter. The system message helped me to guide the AI to provide me specific examples highlighted in bold and provided a short description. I would need to provide it a product documentation with examples to guide me further to specific URLs.

As part of my system message I asked not to answer any questions other than product information my retail company offers on the website. When I ask it to solve a simple math problem it politely refuses to do so:

Metaprompt offers an easy but powerful way how to guide your conversational AI and also ensuring additional safety checks are applied that might be relevant to you / your business. Metaprompt should be a go to way how to personalise you AI service before going for advanced, difficult and expansive technique like fine-tuning.

This content draws inspiration from existing materials and practices. As an employee of Microsoft, I want to clarify that the views and interpretations presented here are my own and do not necessarily represent the official policies or positions of Microsoft. This is intended for educational and informational purposes only.