Building Resilient AI: Crafting a Sustainable CustomGPT for Dynamic Q&A Applications

Generative AI scene has exploded in every industry, despite being a rapidly evolving field. Generative AI has become the top priority for C-Suite as it has sparked tremendous innovation beyond traditional models. This article dwells upon Gen AI's potential to revolutionise the education sector and focus points for education businesses implementing this tech.

According to a report by G2 published in October 2023, Generative AI software, particularly AI chatbots, have gained maximum popularity in the education industry in the APAC region.

Simulating human conversations to achieve personalised responses and content is top education use-case.

Another report by eLinC identifies six key trends for 2024 involving AI in education, focusing on the opportunities and risks.

1. AI-powered intelligent learning assistants
2. Smart content creation
3. Revolutionising AR, VR, and MR experiences
4. Fostering inclusion and accessibility
5. Supplementing analytics for personalised education
6. Accelerating grading and feedback

More Than 80% of Enterprises Will Have Used Generative AI APIs or Deployed Generative AI-Enabled Applications by 2026 - Gartner

Emerging Trends:

Generative AI within EdTech is characterised by several developments and emerging trends:

1. Content Creation: Generative AI can create educational content, including quizzes, assignments, and even entire courses with images and tables. Imagine this being done via an AI model trained on large troves of educational content owned by a knowledge platform organisation.
2. Adaptive Learning Systems / Personalised Learning Paths: Generative AI can create adaptive learning systems that tailor educational experiences based on individual student performance, preferences, and learning styles. Leading to enhanced engagement, improved learning outcomes and instant personalised assessment feedback and analysis.
3. Virtual Tutoring and Chatbots: Generative AI can power virtual tutoring systems and chatbots to assist learners in understanding concepts, answering doubts / queries, and providing real-time additional support outside traditional classroom.
4. Content Summarisation: Generative AI models are employed to summarize lengthy educational materials, making it easier for students to grasp key concepts efficiently.

Types of AI Models

All existing generative AI models can be loosely classified into two types:

• Closed-sourced models are developed by large technology companies, as OpenAI, Microsoft, Google. Their application programming interfaces (APIs) are available for a fee. These models allow fine-tuning the learning capability and performance using your data. Breakdown of the broadly associated cost heads follows in this article.
• Open-source models have their source code, training techniques, and sometimes even the training data available for public use and modification. The cost of such generative AI models will thus include computing costs and the expenses associated with model training.

Most of the Generative AI pioneers in non-tech industries like Education / Learning & Development, prefer to integrate the closed-source model i.e. off-the-shelf products like OpenAI’s ChatGPT, Google Bard, Claude, and Synthesia with their applications using APIs. The integration process is fairly straightforward, and so is the generative AI pricing. The downside? Limits customisation options and creates dependencies for vital business tasks, like handling customer support queries or producing visual content.

Build and train private ChatGPT sustainably

With the rise of Large Language Models (LLMs) like OpenAI's ChatGPT and GPT-4, training a private ChatGPT with organisation owned copyrighted content and customer data, requires robust architecture and understanding of content structure including data splits and joins, to create “your private ChatGPT” that leverages the data.

A. Risks to consider during fine tuning a LLM with your own data: Fine tuning (training) as a solution for adding your own non-structured data on top of a pre-trained language model for the purpose of Question Answering (QA), has major risk of hallucinations.

An AI hallucination is when an AI model generates incorrect information but presents it as if it were a fact. Why would it do that? AI tools like ChatGPT are trained to predict strings of words that best match your query. They lack the reasoning, however, to apply logic or consider any factual inconsistencies they're spitting out.

A new hallucination index developed by the research arm of San Francisco-based Galileo, which helps enterprises build, fine-tune and monitor production-grade large language model (LLM) apps, shows that OpenAI’s GPT-4 model works best and hallucinates the least when challenged with multiple tasks.?

Other Common risks to consider when you fine tune a LLM include;

1. Factual correctness and traceability, where does the answer come from
2. Access control, effort-intensive to limit certain documents to specific users or groups
3. Costs, new documents require retraining of the model and model hosting

B. Separate your knowledge from your language model: Users need accurate responses. This mandates the separation of your language model from your knowledge base. This approach is known as grounding the model or Retrieval Augmented Generation (RAG). This approach allows leveraging the semantic understanding of deployed language model while also providing your users with the most relevant information. All of this is possible in real-time, and no model training is required.

The system architect and programatic prompt engineering team, needs to handle the limitation that feeding all documents to the model during run-time might not be feasible due to the character limit (measured in tokens).

Character Limit Note: GPT-3.5 supports up to 4K tokens; GPT-4 up to 8K or 32K tokens, depending upon the subscription taken; Since pricing is per 1000 tokens, using fewer tokens can help to save costs as well.

The RAG approach would be as follows:

1. User asks a question
2. Application finds the most relevant text that (most likely) contains the answer
3. A concise programmatically engineered prompt with relevant document text is sent to the LLM. Read more on Programatic Prompt Engineering
4. User will receive "An Answer" or ‘No answer found’ response

The following diagram shows the conceptual flow of using RAG with LLMs

Now that we have an understanding of high-level architecture required to start building such a scenario, it is time to dive into the content management technicalities.

C. Retrieve the most relevant data

1. Context is the key. Ensuring that the language model has the right information to work with, build a knowledge base that can be used to find the most relevant documents through semantic search.
2. Chunk and split your data: Since the answering prompt has a token limit, ensure to split documents in smaller chunks.
3. Share multiple relevant sections and generate an answer over multiple documents. Simply start by splitting the document per page, or by using a text splitter that splits on a set token length.
4. Create a search index that can be queried by providing it with a user question.
5. Store the original source and page number to link the answer to your original document. Store additional metadata that can be used for access control and filtering.

Estimating GPT Costs

Publishing companies, Educational Institutions, Organisations evaluating the possibility of utilising their owned and generated data for creating reporting GPTs need to conduct a thorough cost-benefit analysis, considering the unique needs, goals, and budget estimations.

It's essential for C-Suite team to understand costs and assess impact to weigh it against the potential benefits in terms of enhanced learning experiences and outcomes.

1. Cost of Implementation: Integrating ChatGPT into educational platforms or services involve upfront costs for development, customisation, and integration. This can be a significant factor for educational institutions as well as publishers, especially smaller ones with limited budgets.
2. Operational Costs: The ongoing operational costs, including usage fees for accessing and utilizing ChatGPT, can impact the overall budget of educational institutions. The costs are often associated with the number of interactions or usage time, so organisations need to carefully manage and optimise their usage to control expenses.

For the purpose of this article, I am taking ChatGPT pricing to articulate the scenarios.

ChatGPT Usage: ChatGPT’s pricing is based on the number of tokens used for both input and output, which are individual units of text, such as words or characters, that the model processes. The cost of using ChatGPT depends on the number of tokens processed and the pricing plan selected.

Refer : https://openai.com/pricing

ChatGPT API Cost: ChatGPT provides an API for developers to integrate its features into their applications. The API cost is also based on the number of tokens processed and the pricing plan selected. ChatGPT offers a range of plans including free plan, standard plan, turbo plans.

To estimate the cost of using ChatGPT’s API, Technology leaders can use the bottom-up estimation technique. They can break down their project into smaller tasks, estimate the number of tokens required for each task, and calculate the cost based on the pricing plan selected.

Tokens play a crucial role in Natural Language Processing (NLP) to represent a unit of meaning in document. Tokenisation is the process of breaking down a text document into individual tokens for further analysis or processing.

In English text, one token is roughly equivalent to four characters or 0.75 words. As a point of reference, the collected works of Shakespeare consist of approximately 900,000 words or 1.2M tokens.

To learn more about tokens and estimate their usage, experiment with the interactive Tokeniser tool.

ChatGPT Usage Breakdown

Let’s assume a developer wants to integrate ChatGPT’s API into their application to provide chatbot functionality to their users. They estimate that on average:

• A user will interact with the chatbot for about average 3 messages per conversation.
• Each message (averaging input and output) consists of 100 tokens (approx. 75 words).

• The number of tokens per conversation would be 300 tokens (3 messages x 100 tokens per message), for each user.

• If 1000 users interact in a month, the total number of tokens used per month would be 300,000 tokens.

Using ChatGPT’s pricing scheme, the technology team should estimate the monthly cost of using the API:

• The cost of using the API is $0.003 per token.
• Therefore, the estimated monthly cost for the developer’s application would be $900 per month (300,000 tokens per month * $0.003 per token).

Note: The above only specify the ChatGPT test token costs, and there will be associated costs of resources, hosting, server usage etc, apart from Visual Data Costs, all these need to be considered while computing the overall operational costs of a Custom GPT QA Application.

Since the Educational content (HigherEd, Academic, Research, K-12, K-8) is usually elaborate, content-heavy and image-intensive, the possibility of consuming higher number of tokens is a reality, thereby causing a financial strain on the business.

Therefore, Content Management and its splitting process might hold the key to a lower operational costs in the scenario. Subscription Management or Limiting Daily responses might also work, but will impact the user experience, and will not be accepted well in the market

Achieving Sustainability

Building a sustainable CustomGPT for a Question-Answering (QA) application involves careful planning, development, and ongoing maintenance. Here are key points to consider:

1. Define Objectives and Use Cases
2. Cost Modelling & Estimation
3. Data Collection and Preprocessing
4. GPT Model Selection
5. Domain-Specific Knowledge Integration
6. User Feedback Mechanism
7. Fine-Tuning and Adaptability
8. Data Security and Privacy
9. Scalability
10. User Interface and Experience
11. Performance Metrics and Monitoring
12. Legal and Ethical Considerations
13. Documentation and Support

The above is based on my understanding and knowledge gained from various approaches discussed during AI Workshops, Articles and my experience of delivering GPT Products during my previous full-time employment as well as consulting assignments.

For the purpose of this article, I have dwelled upon RAG approach, there are more methods which can be discussed and evaluated basis the actual business use-case.

Coming up next in this segment:

AI in Human Resource Capital Management business use cases.

Benefits of Human in the Loop (HITL) approach in AI Deployment

Please feel free to engage on this and much more, by writing to me on [email protected] or [email protected].