An OpenAI spinoff has built an AI model that helps robots learn tasks like humans

In the summer of 2021, OpenAI quietly shuttered its robotics team, announcing that progress was being stifled by a lack of data necessary to train robots in how to move and reason using artificial intelligence. Now three of OpenAI’s early research scientists say the startup they spun off in 2017, called Covariant, has solved that problem and unveiled a system that combines the reasoning skills of large language models with the physical dexterity of an advanced robot. In this edition of What’s Next in Tech, explore how this new AI model helps robots learn the same way humans learn: through millions of observations.?

For a limited time, subscribe and save 50% to explore our 10 Breakthrough Technologies list and receive a FREE tote bag. With your subscription, you’ll unlock unlimited access to unparalleled reporting, insights, and analysis through the lens of MIT.

Can this new multimodal AI model help robots graduate from the lab to the warehouse floor?

Covariant’s new model, called RFM-1, was trained on years of data collected from the company’s small fleet of item-picking robots that customers like Crate & Barrel and Bonprix use in warehouses around the world, as well as words and videos from the internet. In the coming months, the model will be released to its customers. The company hopes the system will become more capable and efficient as it’s deployed in the real world.?

So what can it do? Covariant’s cofounders showed MIT Technology Review reporter James O'Donnell how users can prompt the model using five different types of input: text, images, video, robot instructions, and measurements.?

For example, show it an image of a bin filled with sports equipment, and tell it to pick up the pack of tennis balls. The robot can then grab the item, generate an image of what the bin will look like after the tennis balls are gone, or create a video showing a bird’s-eye view of how the robot will look doing the task.?

If the model predicts it won’t be able to properly grasp the item, it might even type back, “I can’t get a good grip. Do you have any tips?” A response could advise it to use a specific number of the suction cups on its arms to give it better a grasp—eight versus six, for example.?

This represents a leap forward in robots that can adapt to their environment using training data rather than the complex, task-specific code that powered the previous generation of industrial robots. It’s also a step toward worksites where managers can issue instructions in human language without concern for the limitations of human labor. Read the story.

AI has become mainstream, fundamentally transforming the nature of work for people and the organizations that employ them. Join MIT Technology Review’s editors and reporters for a free LinkedIn Live session about the future of AI enabled work on March 26.

Get ahead with these related stories:

1. This robot can tidy a room without any help — A new system helps robots navigate homes they’ve never seen before with a little help from open-source AI models.
2. Watch this robot as it learns to stitch up wounds — The robot was able to sew six stitches all on its own… and has lessons for robotics as a whole.
3. Large language models can do jaw-dropping things. But nobody knows exactly why. — And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

Image: Covariant