Inference Engine Spaghetti

Building enterprise-grade generative AI applications presents a unique set of challenges that extend far beyond the initial development phase. The allure of easily accessible large language models, such as OpenAI's ChatGPT released in November 2022, has underscored both the potential and the pitfalls of generative AI in the enterprise context. While these models offer unprecedented access to advanced machine intelligence, they also create a misconception that generative AI applications are simple to construct and maintain. Here I hope to guide enterprise application developers through the intricacies of crafting robust, scalable generative AI applications, focusing on design considerations, development guidelines, and the management of technical debt.

Design Considerations

Success in developing complex generative AI applications hinges on incorporating subject matter expertise throughout the design and development process. Unlike traditional software projects, these applications require a blend of conventional programming and the crafting of natural language prompts. Experts play a pivotal role in generating and refining these prompts to ensure the output meets high-quality standards.

Architectural planning should prioritize modularity, facilitating the independent evolution of components such as history and context management. The choice of technologies for prompt enhancement, whether integrating with existing enterprise systems or leveraging advanced tools like Vector Databases, must be aligned with specific use cases. Additionally, developers should create reusable modules for common functionalities and devise strategies for managing API access to large language models (LLMs), considering potential rate limits.

A pivotal design principle is ensuring the application's prompt engine is stateless and headless, decoupling backend processes from front-end interfaces. This approach treats each request as an isolated transaction, devoid of client session state on the server, enhancing scalability and maintainability.

Development Guidelines

Developers should be wary of three major areas: content preparation, prompt writing/testing, and model selection/change.

• Content Preparation: Quality data is the foundation of any enterprise application. Developers must diligently curate data sets, removing redundancies, ensuring the use of vetted content, and maintaining version control. Processing tasks like expanding abbreviations and converting non-textual information are critical for creating clean, usable datasets.
• Prompt Writing/Testing: Effective prompts are the cornerstone of generative AI applications. Establishing a "prompt playground" for experimentation allows developers to refine prompts based on performance and user feedback. A cyclical process involving "generator" and "evaluator" roles facilitates the production of outputs that meet predefined goals and quality standards. Evaluators can be based on predefined rules, machine learning models, or a combination of both. In some systems, the evaluator's role is to ensure that the generated content meets certain quality or safety standards before it is presented to the user or used for further processing.
• Model Selection/Change: The behavior of prompts can vary significantly across different models or model versions. Developers must be prepared to re-evaluate prompts with each model update and make informed decisions about model usage based on cost, performance, and safety considerations. Be prepared to retest the entire system when making a model change!

Technical Debt Issues

Maintenance is inevitable in ensuring application performance and output quality and developers must continuously monitor outputs. Six key practices will help you in understanding when testing and rework is necessary:

1. Continuously screen output for toxicity, truthfulness, and relevance
2. Actively manage versioning and have controls for staging, rollback and tracking training data
3. Monitor all system components for latency and dropped requests - usually this will be an API issue calling an LLM but in some cases applications can become CPU bound as well
4. Token and other model usage metrics should be tracked in realtime in order to intervene before cost or throttling become issues
5. Guardrails for auditability, access control, and filtering should be built into the application architecture
6. Provide a mechanism for user feedback to report inaccuracies or offensive content

The adoption of generative AI applications in large enterprises will grow rapidly in 2024, offering substantial benefits in efficiency, cost reduction, and quality improvement. However, these advantages can only be sustained with a solid foundation in application design and development. Hopefully this provides a helpful starting point to navigate the complex landscape of enterprise generative AI applications.