General AI, Offline AI and RL

Artificial intelligence has always been an exciting and promising field, as it holds the potential to revolutionise numerous industries and change the way we live our lives. The idea of having an AI tool that can play hundreds of video games at a supreme level is incredible, but what if we could take this concept further and create a robot that is not only great at gaming, but also capable of optimizing microchip designs and used for general-purpose, industry-level robotics?

This is the concept of a general-intelligence robot, and it is something that we have never seen before. However, recent advancements in AI research, particularly in the field of reinforcement learning, have brought us closer to this possibility than ever before.

To be fair, it's only natural for many people who have only recently become aware of AI to think that it is limited to generative AI tools like ChatGPT. While language models like GPT have certainly been a significant breakthrough in AI, the real innovation lies in the fact that we have reached a new frontier for AI with natural language models.

The real transformation we have obtained in AI is building AI language models that perform reasonably well in multiple scenarios, allowing the creation of solutions like ChatGPT. However, the reality is that AI is much, much more than pre-trained transformers like GPT. Actually, they are only the beginning.

AI is Not Only Generative AI, It's Much More

Unbeknownst to many, AI fields like Computer Vision or Offline RL have insane potential. However, while the former already has several use cases where it's already actively used, the latter has consistently lagged behind over the years.

Reinforcement Learning, or RL, is a multi-step process that requires "interaction." For each step in the process, the model acknowledges its state (its situation in the environment), performs an action, and if the action implies an approximation to the desired final state, it receives a reward.

For each action the model makes in the game, it understands the impact of that action, potentially receiving a reward and reshaping its parameters to maximise those rewards. This way, the model learns what actions yield rewards and defines the policy - the strategy - it will follow to maximise them.

However, despite the impressive qualities of RL, it has a problem; it is expensive to train. The cost of doing "online" training (making an AI model interact with the real environment to learn) is very high. For that reason, for a long time, AI scientists have wondered if there was a way to pre-train these models in "offline" environments (from a dataset of data instead of learning by interacting with the real environment) using generalised datasets.

That way, you minimise the costs while also allowing AI engineers to have a solid initiation point, a good backbone which more particular, tailored solutions can be trained upon. In other words, what AI scientists have been trying for a long time with RL is to reach the point we've just achieved with NLP with pre-trained transformers: a high-quality initiation point for new AI solutions. In other words, generalistic RL models that can be easily trained into specific utilities in a cost-effective and highly efficient way.

Offline RL, Making the Most Out of RL

Offline RL is regarded as one of the holy grails of AI, as the answer to a long-coveted desire: How do we make RL, a field we know is the real deal for decision-making, affordable, scalable, and accessible?

The potential implications of this breakthrough are huge. One of the major challenges in AI has been developing a general-purpose intelligence that can perform well across multiple tasks, rather than being limited to a narrow set of applications. This new development in offline RL brings us much closer to that possibility.

Imagine a robot that can not only play hundreds of video games at a high level, but can also optimize microchip designs and be used for general-purpose, industry-level robotics. This is a level of general intelligence that has never been seen before.

The implications for industries like manufacturing, transportation, and logistics are immense. Robots with this level of intelligence could revolutionize the way these industries operate, improving efficiency and productivity while reducing the need for human labor. They could be used to optimize supply chains, operate autonomous vehicles, and carry out complex manufacturing tasks with ease.

But the potential applications of this technology extend far beyond industry. It could also be used to create smarter, more efficient personal assistants and other consumer-facing applications. For example, imagine an AI assistant that can not only understand natural language but also learn from your behavior to anticipate your needs and preferences.

Of course, there are also potential downsides to this technology. As with any advanced technology, there is always the risk of it being misused or exploited for malicious purposes. There are also concerns around job displacement, as the development of highly intelligent robots could lead to widespread automation and the displacement of human workers.

Despite these concerns, the development of offline RL is an important milestone in the field of AI. It brings us closer to a future where intelligent robots are capable of performing a wide range of tasks with ease, improving efficiency and productivity in a wide range of industries while also making our lives easier and more convenient.