GPT3: technological breakthrough

Every once in a while, a major technological advance brings to mind the Arthur Clark’s observation: “Any sufficiently advanced technology is indistinguishable from magic.” Just recently it was demonstrated by OpenAI’s GPT3. The WOW effect generated a lot of discussions, but they seem to mostly focus on the HOW: how it works, how to use it, how it will affect the future. I think that it's equally as important to understand the WHAT - what technological breakthrough underlies its impressive performance?

In fact, it’s a combination of achievements that make it so remarkable.

Fundamentally, GPT3 is a text matching engine: it matches the input text with an output text. A search engine, such as Google, is also a matching engine. However, unlike the older generation of matching engines, GPT3 does not match pre-existing pieces of content - it creates the best match instead.?

At the core of every matching engine is a mathematical matching function that defines how the matches are computed. In the case of GPT3, it’s a mastodon of a function containing 175 billion variables. Based on billions of sentences written by humans (training data), GPT3 was able to compute the optimal values for these variables, which are then used to generate new text. Effectively, GPT3 extrapolates patterns learned from human-generated text to produce new content. NB: GPT3 will generate new text every time, even when a better one already exists among its training data.

In other words, GPT3 has captured the collective wisdom of billions of human-produced pieces of content and encoded this wisdom in a highly complex mathematical function. This collective wisdom can be broken down into three main components:

1. Linguistic expertise

GPT3 writes perfectly, from syntactic point of view, in 46 natural languages. The key achievement here is that it does so without any constraints - outside of narrow contexts such as spell check or translation (which it can also do).

2. Knowledge

GPT3 has also captured the knowledge contained in the training data. The capture is not perfect this time – it makes many mistakes. Nevertheless, the accuracy level overall is very high and in some domains the knowledge capture is near perfect. This is a real breakthrough for information retrieval and shows a way to improve search, both on the Web and within organizations. This improvement can come in two ways: through direct question answering or by helping to extract structured data from unstructured content.

3. Human thinking & behavior

Human-produced content necessarily also reflects human thinking & behavior, and such patterns did not escape GPT3. Here the accuracy is not so high, but still, GPT3 is quite polite, can express emotion on demand, and is able to carry out many logical and mathematical operations. This means that machines can now learn to mimic human behavior as well as extract and reuse logic from situational descriptions in natural language.

It's important to understand that GPT3 is a phenomenological model – it directly models the output of a complex system (human writing) without modeling any intermediate steps. This means it has no notion of language structure, knowledge, logic or behavior – all it does is detect and reproduce patterns. It’s truly remarkable that this approach, relatively simple from conceptual point of view, is able to perform so well. The biggest achievement of GPT3 is precisely in demonstrating this to a wide audience, including the general public.