Generative AI - Learnings 2023

This year 2023 has been the year of Generative AI using Large Language Models both closed source and open source.

Like many of us I have been learning via blogs, courses and using prompt engineering for building Generative AI apps and key takeaways are here:

1. Use LLM as a thought partner

• It’s a new way to find creative information but be careful to check on incorrect information (hallucinations)

2. Examples of tasks LLMs can carry out

• Writing - Brainstorming, Press Release, Translation
• Reading - Proofreading, Summarizing, Reputation Monitoring (Sentiment Analysis), Topic Modelling, Extract entities, Moderation for harmful content
• Chatting - Specialized customer service chatbot with internal company data

3. What LLMs can and cannot do?

Ask this question, and if the fresh grad can do the following task, then an LLM can also do it.

4. Prompting best practices

• Be detailed and specific
• Give sufficient context for LLM to complete the task
• Guide the model to think through the answer
• “Chain of thought reasoning”, is the step-by-step reasoning to give the model time to think to get to a final conclusive answer 1. Fast thinking vs slow thinking for complex thinking 2. Give Steps 1, 2, 3, etc and how you want the answer and in which format
• Experiment and iterate - No perfect prompt for every person or situation Instead develop a process for improving prompts through iteration
• Be careful with confidential information which you post as part of your prompt - Check how the prompt provider deals with the privacy of the information you post
• Double check if you can trust the output of the LLM

5. Iterative Prompt Engineering

• Idea
• Iteration 1
• Implementation
• Experimental result, Error Analysis
• Iteration 2 - Repeat

6. Design considerations

Lifecycle of a Generative AI project

• Scope project
• Estimate volume of tokens for Cost estimation
• Choose a model (based on size, closed or open source, and cost) - Refer list of LLMs in the Leaderboards section below
• Build/improve system
• Internal Evaluation
• Deploy and monitor
• Repeat Internal evaluation
• Repeat Build/improve system

7. Advanced Technologies – Beyond Prompting

- Retrieval Augmented Generation (RAG) - Ground model on additional internal proprietary data

- Part 1

Take Knowledge base, break to chunks

Create embeddings for each chunk

For each chunk -> store the embedding with the corresponding chunk in a Vector Database

- Part 2

Take a prompt and create its embedding

Compare the embedding with the embeddings stored in Vector Database

Get the chunks for the matched embedding and send the chunks as part of the prompt to the LLM

- 3 stages of LLM training

• Pretraining - via Supervised Learning on trillions of tokens on all public data on internet
• Fine-tuning - via Supervised Learning Adapt LLM to your task by fine tuning on high quality data
• Reinforcement Learning from Human Feedback (RLHF) - via Classification, Reward Model and Reinforcement Learning

- Cost Considerations - Think through the Cost considerations of using a cloud based LLM to power software applications.

8. Process of building an application

• Tune prompts on some examples
• Add additional "tricky" examples
• Develop metrics to measure performance on examples
• Collect randomly selected set of examples to tune to (development set, hold out cross validation set)
• Collect and use a hold out test set

9. Challenges with LLM

• Ambiguous inputs and outputs
• Hallucination vs facts
• Compatibility amongst models
• Maintaining Data Privacy
• Safeguarding against Prompt Injection

Code

Refer my Github repo for few GenAI projects on

1. https://github.com/mahtabsyed/ChatGPT-API
2. https://github.com/mahtabsyed/Building-Systems-ChatGPT-API
3. https://github.com/mahtabsyed/LangChain-for-LLM-App

Leaderboards

Check these Leaderboards to compare against various LLMs like OpenAI GPT, Meta LLaMa, Google PaLM, Microsoft Phi-2, Mistral SC56

• Best Models Leaderboard - https://huggingface.co/collections/open-llm-leaderboard/llm-leaderboard-best-models-652d6c7965a4619fb5c27a03
• Least Hallucination Leaderboard - https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard

Acknowledgement:

1. Andrew Ng and DeepLearning.ai for so many wonderful courses
2. Chip Huyen for her blogs in https://huyenchip.com/blog/

?

Mahtab Syed - Melbourne - 21 Dec 2023