To Build a GenAI-Powered Application, You need more than an LLM

If you believe the headlines, large language models will magically reinvent how genAI-powered business applications get built: "Just jam your knowledge into a fine-tuned, vector embedding-ed, prompt engineered large language model from one of the big players."

It just isn't so. That simplistic approach won't scale, won't be cost-effective, won't fully protect you, and won't put all your proprietary knowledge to work. You need way more than a customized large language model to build and operate a genAI-powered business application.

To find a better way we stepped back, examined best practices, and talked to 15 of the largest service providers about how they are using genAI themselves and how they help Global 2,000 companies build genAI-powered business applications. With that as input and after months of discussion, we identified a more comprehensive GenAI application architecture we call layers, gates, pipes, and loops (see Figure):

• Layers of intelligence that bring together internal and external capabilities.?Intelligences include a staggering array of capabilities, including general-purpose, specialized, and embedded genAI models and tools from suppliers as well as your own knowledge models, for example, one that captures your knowledge for a customer service chatbot or gives your sales teams tools to incorporate your products, value proposition, and competitor battle cards into their pitch presentations. You should also incorporate capabilities from your ML and AI models, software, and more.
• Input and output control gates to protect people, the firm, and the models themselves.?Input gates turn away bad requests and turn vague requests into answerable prompts. An HR chatbot for employees shouldn’t be generating software code, but it should be great at answering detailed benefits questions. Input gates also keep models safe by rejecting requests that could lead them astray. Output gates of many kinds validate outputs for security, regulatory compliance, nonsense checks, bad-content filters, brand compliance, and more — before releasing the response (often with a person as the final decision-maker).
• Application pipes to prompt, embed, and orchestrate capabilities to generate responses. You’ll use a long-established development technique called pipes to use APIs to tap all the resources from your layers of intelligence, each prompted in turn. The orchestration comes from making sure the API calls flow seamlessly end-to-end, just like steps in a production line on a factory floor. This API-centric, loosely coupled architecture does the heavy lifting. It also provides the platform to implement your dashboards and capabilities to monitor, manage, and operate the application.
• Testing and learning loops to test apps, monitor results, and make adjustments.?Feedback loops are the fourth part in the architecture. They are vital to establishing validity, trust, and confidence in the application. GenAI-powered apps are pulsing, vibrating organisms. Just like a power plant or Jurassic Park, they need constant care. You’ll not only?test the heck out of a genAI-powered application?before turning it on; you’ll also watch it and?retest it continuously?to make sure it doesn’t get out of line. You’ll monitor for quality, performance, costs, and drift.

The language builders and hyperscalers are starting to talk about their platform capabilities to do this, but they can't solve your internal data or compliance challenges or help you put all your knowledge to work. Let's elevate and have a pragmatic discussion about the genAI application architecture to take full advantage of generative AI in enterprise apps.

?