Part 1 - Generalized Architecture for LLM API's in Client Applications

The below diagram presents a generalized inference architecture designed for the integration of Language Model (LLM) APIs within a client application. The primary motivation is to establish a repeatable, component-based design that remains agnostic to the underlying LLM use cases.

The architecture is divided into 2 parts

• Inference lane
• Training lane

This article focuses on the Inference lane.

Client Application Interaction

The client app communicates either in real-time or asynchronously via Kafka topics to the Python-based backend. Asynchronous processing is typically required for batch-based inference, such as computing risk levels for all patients in a nightly job and alerting the APCs if they fall into 'Higher' risk brackets. Client app is responsible for sending the request which contains at least 2 things

• The LLM request - comprising the use case name
• The prompt variables - derived from business objects

Backend Connectivity

The backend connects to various layers as described below:

1. LLM API Layer:This layer abstracts connections to different LLMs, such as GPT 3.5/4, LLAMA 2, or Mistral models based on specific requirements.
2. Rate limiter: Most LLM's have limits on no of requests/min per day. Hence the client apps need to be considerate of how much the LLM's could be hit in a given time frame
3. Postgres DB:Holds the LLM JSON configuration for a specific use case. The configuration includes LLM source, prompt parameters, request-to-prompt variable mapping, RAG (Retrieval Augmented Generation via vector DB's) index, sync/async API details, etc.
4. Connections:Most LLMs have a secret key that needs to be stored in a KeyVault with controlled access.
5. Vector DB:Depending on the use case, a vector DB like Pinecone is required when using RAG. Vector indices are created beforehand using the source data and referenced via the configuration.
6. Redis Cache Layer:This layer can cache the LLM response if the same or similar prompt is requested to be executed. This avoids the round trip to the LLM servers.
7. Tools:Different extension toolsets like Langchain and the newly launched OpenAi Agent framework can be utilized based on the use case and configured accordingly.

General Inference Architecture Considerations

Any general inference architecture for LLMs must account for the 3 Rs:

• Reliability: How accurate are the responses from the LLM for the use case?
• Repeatability: Are responses idempotent?
• Responsiveness: Can we impose upper/lower bounds on response timings?

In the next part, we will see more details on the Training lane.