An LLM maturity model

Here's a simple LLM maturity model framework along two dimensions: model selection capabilities and model governance. The governance needed largely depends on the capabilities used. An illustrative analogy is that different rules and guardrails are needed to drive different types of cars. Range Rovers are easy to drive, safe, and good in most weather conditions. Sports cars, however, require more skill and are higher risk. Supercars are expensive, are only for skilled drivers, and are inappropriate in inclement weather. Lastly, custom hot rods are high-maintenance and require a team of experts to build and operate. Here's the framework:

Level 1. Model selection: Pre-trained model APIs and UIs. Model governance: red-teaming. Car analogy:

Level 2. Model selection: Prompt engineering, retrieval augmented generation (RAG), and few-shot training. Model governance: Define and measure accuracy and cost. Car analogy:

Level 3. Model selection: Fine-tune pre-trained models. Model governance: Define and measure biases. Car analogy:

Level 4. Model selection: Full training of LLMs. Model governance: Continuous, real-time measurement. Car analogy:

The levels of model selection capabilities are:

1. Pre-built models: Using APIs or UIs for pre-built models. These may be multi-tenant, single-tenant or private. Examples are GPT-4 and GPT-3.5 from OpenAI; the same models from Azure; Anthropic’s Claude and AI21’s Jurassic from AWS; and Falcon, Mosaic ML, Llama 2 and Mistral-7B from Hugging Face.?
2. Prompt engineering: Testing different types of prompts on different models to select the ones with the best accuracy versus cost tradeoff. Here we include testing different prompt sizes for retrieval augmented generation (RAG) and few-shot training in general.
3. Fine-tuning: Adding your own corpus to the training of a pre-trained model and changing the model’s parameters. Reinforcement learning may also be used here to continuously fine-tune based on human feedback.
4. Full training: Completely training your own LLM.

Levels of model governance are:

1. Red-teaming: Periodic structured testing to find flaws and vulnerabilities (This was highlighted in the Executive Order on Safe AI, Oct. 31, 2023).
2. Measure: Define and quantitatively measure the accuracy and cost of an LLM use case. This may be done periodically like red-teaming or continuously to detect flaws as early as possible. There are a wide variety of current and emerging accuracy measurement techniques including end-user feedback (??,??), human-in-the-loop evaluation of a subsample of responses, and using an ensemble of other (hopefully uncorrelated) LLMs to evaluate an LLM (e.g., use GPT-4 to evaluate GPT-3.5 or Jurassic and Claude to evaluate Llama 2).
3. Biases: Define, prioritize and quantitatively measure the biases of concern to an LLM use case. Different use cases will have different biases that are important. For example, the biases of concern in a healthcare use case might be very different from those in a logistics use case. Bias measurement may be done periodically like red-teaming or continuously to detect flaws early.
4. Real-time monitoring: Monitoring accuracy, cost and biases continuously in near real-time.

You don't need a custom hot rod for a run to the store for milk, especially in a snowstorm. Which is best for a use case will depend on your time, budget, and risk appetite. And one size doesn't fit all, so if you have lots of use cases you'll have lots of combinations. How seamlessly does your AI platform support that?

#ai #genai #LLMs