Democratized Research Collectives are Popping up in the A.I. Community

Rise of the Decentralized A.I. Labs

If you enjoy articles about A.I. at the intersection of breaking news join AiSupremacy?here. I cannot continue to write without community support. (follow the link below). For the price of a cup of coffee, Join 89 other paying subscribers.

https://aisupremacy.substack.com/subscribe

What is the Farama Foundation and the future of RL?

Hey Everyone,

Way more impactful than the hype around Generative-AI in the last year, the uptick in new kinds of A.I. labs that are global, decentralized and democratizing accessibility to A.I. is really impressing me.

I wanted to do a short overview while introducing the new one to the club. The fact is the A.I. community is incredibly collaborative and global, despite the political and supply-chain de-coupling we are seeing in some industries on the international and geopolitical stage. Progress in research is just more efficient when top talent work together, no matter where they live.

In a remote work global context, this is now possible. New AI Labs are popping up every month now. This is then the A.I. meta trend nobody tells you about.

To make A.I. really more accessible and democratized, researchers are getting around BigTech’s hold on it by creating more autonomous decentralized A.I. labs and collectives.

Its A.I’s fundamental global and collaborative nature of research that’s also showing an acceleration since 2021 in the entire industry.

At the heart of this movement are these new kinds of A.I. labs, startups and focused (and somewhat decentralized) research collectives. I’m trying to cover some of them on this Newsletter.

The rate at which these A.I. labs, startups and organizations are a forming is now higher than ever before as we head into 2023.

I consider this a fairly important development in the last few years in the?global A.I. community. From Hugging Face to so many other non-profit A.I. Labs, it’s ushering in a new era for A.I. research.

This is one of the reasons I like to cover new endeavors in my “Prospectus” column. When I covered my summaries of the State of AI Report 2022, this was one of the more interesting parts of A.I.’s development in 2022.

What is Farama Foundation?

This is a tweet below:

Farama Foundation – a new nonprofit organization designed in part to house major existing open source reinforcement learning (“RL”) libraries in a neutral nonprofit body. We aim to provide standardization and long term maintenance to these projects, as well as improvements to their reproducibility, performance, and quality of life features. We are also working to develop key pieces of missing software for the open source reinforcement learning ecosystem.

The Farama Foundation is a new nonprofit dedicated to maintaining and standardizing open source reinforcement learning projects (e.g. Gymnasium, old Gym)

GitHub:?https://github.com/Farama-Foundation

Their mission is to develop and maintain open source reinforcement learning tools, making reinforcement learning research faster and more productive, and reducing the engineering workload required to apply RL in both research and industry.

• Open-source RL Learning tools
• Making RL research faster and more productive
• Reducing engineering workload required to apply RL

What is RL in Machine Learning?

Reinforcement learning is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward

Rise of Decentralized Research Collectives

Many of these A.I. labs start off as non-profit and then have to pair up with bigger partners. DeepMind with Google, OpenAI with Microsoft for example, Google with co:here - more funding on the way.

The global community via hubs like Hugging Face manage to do important work in new collaborative global movements.

This even as existing BigTech talent at places like Google Brain, Meta AI and DeepMind pivot to found their own companies.

There is then this new breed of A.I. startup. More specialized than ever before.

All of this accumulates over the 2020s to make A.I. move faster in the 2030s. We are already seeing more applications for example of robotics in the real world.

Will RL Usher in an Age or Robots?

• Jasper robots?assemble fresh meals?for?nearby apartment dwellers.
• Laundry folding?robots are?improving in their abilities.
• Tesla promises to make a?general purpose?humanoid robot called Optimus.
• NATO deployed its own?“killer robots” as well recently.

Others view A.I.’s culmination with RL as an existential risk. In 2021, Google’s DeepMind said?RL was enough?to reach AGI, or artificial general intelligence.

What has Reinforcement Learning Achieved so far?

Reinforcement learning has been able to achieve human level performance, or better, in a wide variety of tasks such as?controlling robots,?playing games, or?automating industrial processes. Reinforcement learning has also been responsible for some of the greatest achievements of AI in recent history, such as?AlphaGo,?AlphaStar, and?DOTA2.

• Reinforcement learning is conceptualized as a loop where the agent observes the state of its environment, and then takes an action that alters that state.

Farama Foundation

How will A.I. evolve in the RL Loop?

• At the time of receiving the next observation, the agent also receives a reward associated with the most recent action.
• Some academics wonder when A.I. gets more evolved how humans might keep A.I. safe given this loop.
• Curiously according to the recent?Spiceworks Ziff Davis State of IT survey, which polled 1,400 tech professionals in North America, Europe, Asia and Latin America, found 49% of respondents said innovations in AI could lead to human extinction, mirroring opinions espoused by Tesla founder Elon Musk and theoretical physicist Stephen Hawking.

Is A.I.s here (Dangerous or Useful) yet?

Primer on RL

In supervised learning, the basic software stack typically only has three components: the dataset, preprocessing of the dataset, and the deep learning library.

• In reinforcement learning, the software stack is much more complex. It starts with constructing the environment itself, usually a piece of software like a simulation or a video game. The base environment logic is then wrapped with an API that learning code can be applied to.

Reinforcement learning’s reliance on standard software environments as opposed to datasets creates a lot of unique problems for the field. Increasingly synthetic data and synthetic AI is used.

The Standard API Problem and The Origins of The Farama Foundation

In order to have standardized environments and modular RL code in general, there needs to be a well-designed and easy to use standard API for accessing reinforcement learning environments.

For most use cases, this already exists through a Python library called Gym. Gym was originally created by OpenAI 6 years ago, and it includes a standard API, tools to make environments comply with that API, and a set of assorted reference environments that have become very widely used benchmarks. It’s been installed more than?43 million times?via pip, cited more than?4,500 times?on Google Scholar, and is used by more than?32,000 projects?on GitHub. This makes it by far the most used RL library in the world.

The Farama Foundation effectively began with the development of?PettingZoo, which is basically Gym for multi-agent environments. PettingZoo was developed over the course of a year by 13 contributors.

Read their full Announcement blog?here.

The core?team?of contributors maintaining Gym and PettingZoo dramatically grew into a massive international team, and our greater group of contributors, now known as the Farama Foundation, currently spans 14 timezones.

Farama Foundation?is therefore one of these new A.I. decentralized, global and community based A.I. labs. Think of these organizations like DAOs of the machine learning community.

• Check out their?project standards.
• Their goal is to offer long term maintenance, standardize environments and add key quality of life features. Some of the most important quality of life features they’re working to add are consistent detailed documentation websites, easy installation, support for multiple architectures and operating systems, type hinting, docstrings, and improved rendering functionality.

Supervised deep learning is used so often in our daily lives that it’s hard to comprehend.

Reinforcement learning, on the other hand, is rarely used in application right now, and usually requires massive teams to deploy.

They are likely going to stimulate awareness and research in RL.

Current reinforcement learning algorithms can already control nuclear fusion reactors, robot balloons in the stratosphere, real F-16s fighter jets in flight, or?layout production semiconductor chips. Reinforcement learning can also achieve superhuman performance in what are extremely challenging games such as StarCraft 2, DOTA 2, Go, Stratego, or Gran Turismo Sport on real PS4s.

Thus the?Farama Foundation?likely has a bright future.

If you enjoy my content?and want to support while getting access to more posts, please be my guest. I’m very grateful for any support. This publication has nearly hit 8,000 readers in the first 10 months, with over 90 paid subscribers.

With premium you get twice as much content at about $0.50 per post. Much less in fact with a yearly subscription.

If you enjoy articles about A.I. at the intersection of breaking news join AiSupremacy?here. I cannot continue to write without community support. (follow the link below). For the price of a cup of coffee, Join 89 other paying subscribers.

https://aisupremacy.substack.com/subscribe