FsOpenAI: A GPT 'chat' app for Internal Organizational Data

#chatgp #azureopenai #semantickernel #semanticsearch #fsharp

Unsurprisingly, the demand for accessing Large Language Models (LLMs) is high because of the promise they hold. I will not belabor their benefits here to keep this post brief and focus on the key point that organizational employees are clamoring to apply the power of GPT-style models on their own internal data.

Many organizations have directives that discourage employees from using the public OpenAI endpoints with internal data - which just adds fuel to the fire.

Today, the Azure OpenAI service is the only option for privately accessible GPT-style models. However, beyond models we also need services to securely store and index internal data so that it can be used effectively with the Azure deployed models. Azure Cognitive Search service can fulfill this need (although other options are also available),

FsOpenAI is meant to be deployed as an Azure web app. The goal of FsOpenAI is to provide a chat UI that can be quickly deployed to securely utilize the aforementioned Azure services. The configuration options are kept limited to reduce the IT deployment overhead.

Question-Answer Interactions

FsOpenAI has two chat or interaction modes:

1. Basic chat with pre-trained GPT-style models
2. Question and answer sessions over data stored in Cognitive Search - in conjunction with Azure OpenAI models

Most are familiar with the first. The second is of primary interest here. The diagram below shows the data flows that enable the Q&A interactions.

The right-hand side shows the data flows associated with creating the search index, while the left shows the flows associated with a typical Q&A interaction. Internal document data is 'shredded' into ~1K, (possibly overlapping) chunks. An embedding vector is generated for each chunk. The chunk text, vector, and any associated metadata are inserted into an Azure Cognitive Search index. The embedding vector captures the semantics of the text chunk.

When a user inputs a query (question or instruction) against the index data, the query is first vectorized using the same embedding model that was used for indexing. The resulting vector is then used to perform a nearest neighbor search of the index to find the most relevant text chunks. The retrieved chunks are combined with the original query and sent to the deployed chat model for generating the answer to the query - contextual to the search results. See example below:

Application Features

The FsOpenAI interface design was informed by the experienced gained from performing Q&A interactions using script code over several internal and public datasets. The following are the main features worth mentioning:

• Each chat has its own tab. Multiple concurrent chat sessions are supported.
• The chat parameters (temperature, selected models, etc.) are specific to each chat. One can easily experiment with how the settings effect the same basic query.
• The chats can be saved into local browser storage and are loaded when the site is first opened.
• To the extent possible, the model responses are streamed back to the UI in real-time. This is key because some models, e.g., GPT-4 and GPT-4-32K, can take a while (up to a minute) to generate the full response.

Features to be supported in the future:

• Custom prompts.
• OpenAI plugins and functions to support more complex interactions.
• Ability to index and query documents on the fly

Summary

FsOpenAI exists to support Q&A interactions with GPT models in the context of internal organizational data. Beyond the infrastructure setup required for Azure OpenAI (model deployments) and Azure Cognitive Search (vector indexes), FsOpenAI requires minimal infrastructure for deployment - just the Azure Web App and Key Vault services.

FsOpenAI source is available under a liberal MIT license. An online version of the app - with no backend services - is available here for a limited time. You may use your own OpenAI key to quickly test the base chat interface with publicly deployed OpenAI models.