A Primer on Generative AI Technology

Oh, ChatGPT! Oh, ChatGPT! Almost a year on the market!Setting aside this past week’s chaos at OpenAI, what is Generative AI Technology?

Generative AI refers to a category of AI systems that can generate new, original content in forms such as text, images, music, and more, unlike traditional AI systems that may follow predefined rules or rely on explicit programming for specific tasks.

It has created an ecosystem that has emerged all on its own, with lists of prompts, tips, APIs, use cases, extensions, podcasts, success stories and failures.?

OpenAI’s GPT(Generative Pre-trained Transformer) ?is powered by a?Large Language Model, which is a subcategory of a class of models, called Foundation models.

?

"Foundational models" refer to large-scale models trained on massive amounts of a broad range of data in an unsupervised manner, allowing them to learn general patterns and features present in the data enabling them to create new content that is similar to the original data.?These models serve as a starting point or “foundation” for a variety of specific tasks without the need for extensive task-specific training. They can be fine-tuned to perform tasks like text classification, language translation, and question answering without requiring a large amount of new data.?

Traditional AI models?are designed for very specific tasks and require a lot of labeled data to train. If we need to solve ten tasks, we need to train ten separate models, which can be time-consuming and resource-intensive.

Because of the huge amount of data that those foundation models have seen, by the time that they're applied to small tasks, they can drastically?outperform a model that was only trained on a few data points.?

?

Let’s dig in, just a bit deeper into the Transformers type of neural network.

?

Consider a language model that is trying to predict the next word based on the previous ones. If we are trying to predict the next word of the sentence?“the clouds in the …”. In this case where the difference between the relevant information and the place that is needed is small, the neural network model can learn to use past information and figure out the next word for this sentence as “sky”

Let’s take another example, let’s say we want to predict the last word of the text “I grew up in India. My primary education was in a city called … I speak fluent…” . For this case, the model will need more context. Recent information suggests that the next word is probably a language, but if we want to narrow down which language, we need context of India and the city, that is further back in the text.

For translating sentences like that, models like Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been used to figure out these sort of dependencies and connections.

?

In this example, each input is set as the word in that text. The Recurrent Neural Network passes the information of the previous words to the next network that can use and process that information.

These models may also need to selectively remember or forget things that are important and not so important.

Like everything else, GenAI has?its limitations.

?One is the high computing cost, which might use up multiple GPUs at a time just to host the models with a couple of billion parameters. This makes it challenging for enterprises to train foundation models on their own or vet start-ups that create theirs.

Another limitation is trustworthiness, especially in the language domain due to the vast?amounts of training data scraped from the internet. Given the size of the data, it's impossible for human annotators to vet each data point to ensure it is unbiased and free from harmful content.

We are still at the beginning of GenAI, can’t wait to see what the future looks like!