ChatGPT - How we got to where we are today. A timeline of GPT development

There has been a large amount of attention focused on ChatGPT and before that, DALL-E-2, but it's important to understand that both of these are based on the same Generative Pre-Trained Transformer language model called GPT-3. Before we can understand where all this is leading us in the future, it is useful to understand what GPT-3 represents, and how rapidly we arrived at where we are today.?

What is a GPT ??

A Generative Pre-Trained Transformer (GPT) is a type of language model developed by OpenAI. It uses a transformer neural network architecture and is pre-trained on a large corpus of text data. The pre-training allows the model to learn general language patterns and representations, which can then be fine-tuned for a variety of natural language processing tasks, such as language translation, text summarization, and question answering.?

The GPT models are generative, meaning they can generate text that is similar to the text in the training data. But a GPT like GPT-3?is capable of more than generating text. Due to their transformer architecture and pre-training on a diverse text corpus, GPT models have the ability to generate not just text but also other types of data such as images and audio, given the right fine-tuning and the input format.

For example, GPT-3 has been fine-tuned to generate images, music, code, and even perform simple tasks like scheduling and translation, this is possible due to the large amount of knowledge it has been pre-trained on, but it still requires fine-tuning on specific task and/or input format.

A Timeline

2017 - Pre GPT-1

There were earlier examples of models that used similar concepts to GPT-1 before it was introduced, but GPT-1 was the first model to be called GPT and introduced by OpenAI.

For example,?the Transformer model was introduced by Google in 2017. The Transformer model is the architecture used in GPT models and it introduced the attention mechanism that GPT models use to understand the relationships between words in a sentence.

Additionally, other models such as ELMO and BERT, introduced in 2018 by Allen Institute for Artificial Intelligence, also used the concept of pre-training on a large dataset and fine-tuning on specific tasks, which is a key aspect of GPT models.

2018 - GPT-1?

GPT-1 was the first version of the GPT series of language models developed by OpenAI. Released in 2018, it was trained on a dataset of about 40GB of text data and have 1.5 billion parameters.?

It was capable of generating human-like text, completing tasks such as translation, summarization, and question answering with a high degree of accuracy.

2019 June - GPT-2

GPT-2 is a more advanced version of GPT-1, and it has several key differences and improvements over its predecessor. Some of the main differences between GPT-2 and GPT-1 include:

• GPT-2 was trained on a much larger dataset than GPT-1, with about 570GB of text data. This allows GPT-2 to have a more diverse and comprehensive understanding of natural language.
• GPT-2 has significantly more parameters than GPT-1, with 1.5 billion parameters. This allows GPT-2 to have a more complex and powerful model, which is better able to generate more human-like text.
• GPT-2 is able to generate more coherent and fluent text than GPT-1. GPT-2 is also able to generate longer paragraphs of text, which makes it more useful for tasks such as writing essays and articles.
• GPT-2 has a better understanding of natural language than GPT-1, it is better able to answer questions, summarize text, translate text and do many other natural language processing tasks with a high degree of accuracy.
• GPT-2 can be fine-tuned to specific tasks more easily than GPT-1, which allows it to achieve better performance on a wide range of natural language processing tasks.

GPT 3 June 2020

GPT-3 is the most recent and advanced version of the GPT series of language models developed by OpenAI, and it has several key differences and improvements over its predecessor GPT-2.

• GPT-3 was trained on a much larger dataset than GPT-2, with about 570GB of text data. This allows GPT-3 to have a more diverse and comprehensive understanding of natural language.
• GPT-3 has significantly more parameters than GPT-2, with 175 billion parameters. This allows GPT-3 to have a more complex and powerful model, which is better able to generate more human-like text.
• GPT-3 is able to generate more coherent and fluent text than GPT-2, and is able to perform a wide range of natural language processing tasks such as writing essays, articles, poetry and even coding with a high degree of accuracy.
• GPT-3 has a better understanding of natural language than GPT-2, it is better able to answer questions, summarize text, translate text and do many other natural language processing tasks with a high degree of accuracy.
• GPT-3 can be fine-tuned to specific tasks more easily than GPT-2, which allows it to achieve better performance on a wide range of natural language processing tasks.
• GPT-3 can perform tasks that involve multiple modalities, such as image and text understanding, image captioning, and even generating images from text.

It is perhaps the multi-modal capabilities of GPT-3 that distinguish it the most, because it allowed the development of DALL-E-2.?

The arrival of DALL-E-2 and ChatGPT did more to democratize the use of AI than anything that has gone on before. For the first time, everyone from students, to artists and business people could make use of the GPT-3 capabilities.?

DALL-E-2 - January 2021

DALL-E 2 is the second version of DALL-E, which is a model for generating images from text descriptions. DALL-E-2 builds on the capabilities of the first version of DALL-E, which was released in December 2020, and it is able to generate a wider variety of images, including ones that are more abstract and less predictable

This model is considered as a major advancement in the field of image generation and AI, it can generate images that are realistic and diverse, and it's able to generate images of things that do not exist in the real world.

November 2022 - ChatGPT?

Finally, ChatGPT was introduced and and an even wider audience could easily interact with the GPT-3 model.?

ChatGPT is a variant of the GPT-3 model that was fine-tuned for conversational language understanding and generation. ChatGPT is not a standalone model, like DALL-E-2, it is built on top of the GPT-3 model.?ChatGPT is a chatbot, but unlike most chatbots, ChatGPT remembers previous parts of the conversation. In this manner, it mimics a human conversation.?

So, why is this time-line important to understand ? In 2016, an AI computer program called AlphaGo defeated Lee Sedol, a professional Go player, who at the time was considered to be one of the best players in the world. It was clear at the time that machine learning based AI, was going to outperform humans but only in very specialized areas.

A year later, OpenAI introduced the first GPT, and what many thought would take a decade or more has happened in the last 5 years. What we are seeing now, at the end of 2022 are more generalized AI capabilities, a long way from AGI, but a step closer. Its only the beginning, and the pace at which change is happening is likely to accelerate not slow down.

#ChatGPT #AI