"LLMOps" - Operations for Large Language Models (LLMs), is equally important as traditional MLOps. We need to reconsider how LLMs fit into the traditional software development life cycle.
Here are the key implications in the development cycle for LLMs:
- Access to LLMs - Developers access the models in various forms, such as APIs (from proprietary cloud providers or vendor APIs) or customised open-source models through fine-tuning. In most cases, developers start with APIs, and for better results, they might end up with a customized model. In the case of a customized model, serving infrastructure that demands large memory and GPUs with efficient resource management will become a new challenge.
- Prompts are the new code: Developing natural language prompt templates with sufficient context and examples becomes the new focus for solving specific business problems. This introduces the need for prompt development tools and chaining frameworks like Langchain and Promptflow. These templates require treatment as first-class software artifacts, including building, packaging, testing, and deployment.
- Knowledge base: Since prompts need to be enriched with domain-specific context, the need for a new database has emerged - Vector DB.This demands new additions to the data ingestion pipeline toolings and database management
- Observability: Unlike traditional machine learning, observing the quality and performance of LLMs is different, as they deal with unstructured text.This brings the need for new instrumentation frameworks and integration with observability tools
- Security: Because LLMs generate responses based on pre-trained large datasets and user-provided text prompts, measuring and implementing security measures in the new world of LLMs can be challenging. There will be a need to incorporate sophisticated and customisable guardrail frameworks.
- Human feedback: Human feedback is critical for evaluating and measuring LLM performance and quality. Capturing human feedback is essential in the new development process.
- Prompt Reuse: Maintaining the text prompt templates in a discoverable way helps developers quickly build the logic without reinventing the wheel.Solutions like Prompt Hub will help centrally manage and version the prompts.
In conclusion, LLMOps need to be carefully designed and integrated into the software development lifecycle by choosing the right tooling. This area needs to evolve and be well-defined, similar to traditional MLOps.