Fine-tuning GPT-3 Using Python to Create a Q&A Assistant

Using an OpenAI model for FAQ by training it on a custom dataset

What is Fine-tuning?

Fine tuning allows one to train a pre-trained LLM on a smaller, domain-specific dataset to learn specific patterns and relationships within this dataset and generalize its knowledge for the task at hand.

Fine-tuning, in essence, allows us to adjust the pre-trained model to a customized dataset and optimize its performance for the task at hand (NLP application). Fine-tuning in GPT-3 entails modifying the model's parameters either manually or by training it on a dataset tailored for the given task.

According to the official documentation, fine-tuning allows in getting more out of the GPT-3 models by providing:

• ???Higher quality results than prompt design
• ???Ability to train on more examples than can fit in a prompt
• ???Token saving due to shorter prompts
• ???Lower latency requests

OpenAI models

OpenAI utilizes a range of LLM models with varying capabilities and costs mentioned below:

• GPT3: the class of davinci, curie, ada, and babbage models having Natural Language understanding capabilities. These are the only OpenAI models that can be fine-tuned
• GPT3.5: models with the ability to comprehend and produce both natural language and code. This series of gpt-3.5-turbo, gpt-3.5-turbo-0301, text-davinci-003, text-davinci-002 and code-davinci-002 are used as Chat models, that is, they are specially designed for chat applications.
• GPT4: a multimodal model having advanced reasoning capabilities and broader general knowledge that can solve challenging problems. Like the 3.5 series these models are also optimized for chat, but available on a limited beta license.

OpenAI's GPT-3.5 model does not support fine-tuning. GPT-3.5 is designed to be a highly performant and efficient language model that is already pre-trained on a wide range of natural language processing tasks, making it well-suited for many real-world applications without the need for further fine-tuning.

Dataset preparation

For fine tuning OpenAI GPT3 models, a set of training samples that contain a single input (prompt) and its corresponding output (completion), separated by a fixed separator (->) indicating the end of the prompt and the beginning of the completion.

This blog uses a custom dataset prepared after manually segregating question-answer pairs paragraph-by-paragraph from a 35 page PDF document. In the dataset shown below, the question is listed as the prompt while the answer is the completion.

Important considerations

1. Prepared data must be in JSONL?file format
2. Each completion should begin with a whitespace character and terminate with a stop sequence such as \n or ### indicating the conclusion of a completion
3. A prompt must not include a leading whitespace and the separator (mentioned above) indicates the end of the prompt, therefore, there must not be any punctuation marks or whitespaces trailing the separator.

Creating the dataset

In case the training dataset is not in the required file format, OpenAI’s CLI tool can be used to convert CSV, TSV, XLSX and JSON files to the required JSONL format as implemented below:

!pip install openai?? 
openai tools fine_tunes.prepare_data -f <LOCAL_FILE>?        

Once a JSONL document has been prepared, run the following command to create a file for fine-tuning the model

Creating a fine-tuned model

There are three methods for fine-tuning a GPT-3 model:

• ???Manual fine-tuning using the OpenAI CLI
• ???Programmatic fine-tuning with the OpenAI package
• ???Using the finetune API endpoint

Programmatic finetuning and endpoint based finetuning are beyond the scope of this implementation. The steps for OpenAI CLI implementation are listed below.

Step1:

Set OPENAI_API_KEY environment variable by including the following line in your shell/terminal:

?????? export OPENAI_API_KEY="<OPENAI_API_KEY>"        

Step2:

To begin the fine-tuning process, specifying the GPT-3 model to be used with the "-m" flag. The 'ada' model was chosen since it is the lowest cost model.

?openai api fine_tunes.create -t test.jsonl -m ada --suffix "custom model name"        

It may take some time to finish a fine-tune work once you've begun it. There is a possibility that the current process may be queued behind other jobs on the system, depending on the model and dataset size, training can take minutes or hours. If for some reason the event stream is disrupted, resume it by running:

? openai api fine_tunes.follow -i <YOUR_FINE_TUNE_JOB_ID>        

When the job is done, it should display the name of the fine-tuned model.

The convention followed for the model name is <base-model>:<suffix>:<timestamp>, in our example the name assigned to the fine tuned model is ada:ft-personal-2023-05-12-11-18-17

Using the model

We will use our trained fine-tuned model for question answering by logging in to Playground and choosing our model name from the list of models provided by OpenAI.

Below the model name you can assign completion parameters to the trained model. Each parameter (described below) shown below Fig.5 can influence the output of a generative model.

1. Temperature: governs the unpredictability of the generated text. A temperature near 0 makes outputs more deterministic, while temperature near 1 makes the answers more random
2. Maximum Length: limits the length of the generated text to a certain number of tokens (1 token = ? characters in English language). You can find token limits here <insert link>
3. Stop Sequences: a list of phrases or words that, when generated, signal the model to stop generating text. This can prevent the model from generating irrelevant or undesirable content.
4. Top P: probability threshold for generating the most likely terms. Lower numbers provide more conservative and predictable text, whereas higher values produce more diverse and unpredictable material
5. Frequency Penalty: punishes the model for generating frequently recurring words or phrases. This can be beneficial in encouraging the model to generate more distinct and diversified text
6. Presence Penalty: penalizes the model for creating text that contains specific words or phrases. This can help to keep the model from producing undesirable or offensive stuff
7. Inject Start Text: a custom starting text for the model to generate from
8. Inject Restart Text: a custom restart text from which the model will begin creating
9. Show Probabilities: displays the probability distribution of the model's predicted words, giving insight into the model's confidence in its predictions.

Result

Conclusion

In the field of Natural Language Processing (NLP), fine-tuning GPT-3 and other large language models (LLMs) has become the ideal method for developing solutions that are specialized to a given domain, resulting in impressive performance gains. We can create extremely effective question-and-answer (Q&A) assistants that excel at comprehending and producing human-like responses by harnessing the power of these models and fine-tuning them.

At CrossML Private Limited, we leverage generative AI and GPT to build applications like Personalised Customer Service Bots, Automated Marketing Content generation, Knowledge Search etc. Reach out to our expert AI team for a free consultation at [email protected]

Follow us on LinkedIn https://www.dhirubhai.net/company/crossml

More articles