Robot Operations Meets ChatGPT

I get it, the headline sounds like clickbait, but bear with me for a bit. This article was written by a human, so that has to count for something these days.

There's a ton of confusion out there about robots, artificial intelligence, chatbots, "robotic process automation" and a bunch of loosely related topics that are often just thrown into the same bucket by people with a limited understanding of the underlying technologies. I'm going to try my best to avoid that and hopefully bring some clarification to these points for non-technical readers.

In my role at InOrbit.AI I work with a lot of robot developers that are creating different types of smart robots. These are real-world, physical machines powered by software that use sensors to perceive the environment in which they operate. Some examples include autonomous mobile robots (AMRs) used for material handling, cleaning or inspection tasks, as well as vision-driven robotic arms and collaborative robots (cobots.)

The software that drives these smart robots often uses Machine Learning (a branch of AI) such as computer vision to identify objects and obstacles, as well as path planning to optimize their movement to achieve a stated goal. More advanced uses of AI may include reinforcement learning to enable a robot to "teach" itself how to best achieve a task, whether in the real world (IRL) or in simulations (known as Sim-to-Real).

On the other hand, the latest craze -- which is not really new, but has become the current hype amongst investors and entrepreneurs -- is generative models, a type of unsupervised learning?techniques in machine learning. Whether it's large language models (LLMs) for text generation or deep convolutional generative adversarial network?(DCGAN) for creating images, the idea is to train a model on a large body of data to then generate similar yet unique data that seems "real".

These technologies (robot navigation and generative models) share some common ML foundations, but have evolved to achieve different goals. They are now finally at a level of maturity where applications combining the two is just becoming possible and may open up new possibilities.

Generative models can be trained on domain-specific data, including robot operations data. For instance, platforms like InOrbit capture large amounts of operational data, from highly specific data sources to overall mission performance. This is done at scale across hundreds or thousands of robots and is typically presented to trained operators using configurable dashboards.

Using ChatGPT or equivalent LLMs trained on these data sets, it may be possible to generate a conversational interface, allowing non-technical users to ask questions and get answers without having to understand dashboards. Likewise it may be possible to generate narrated reports from the data. Since RobOps platforms like InOrbit can also be used to direct the behavior of robots, it may be possible to trigger behaviors through a conversational chat UI.

For the time being, systems like ChatGPT suffer from what is known as hallucinations: because these models don't actually understand the world, they can often make up facts or concepts that are nonsensical. For instance, ChatGPT may be convinced that a robot teleported from one location to another if the intervening data is missing. Therefore, it's still critical to have humans directing the work and ensure that the information presented actually checks out.

It may also be possible to recreate images of the robot environment from camera and sensor data. This would make it possible to create views from arbitrary points in space and time, covering the gap between real-life image capture and siumulation.

As generative models evolve and hallucination issues are addressed, it will enable a much broader set of people to interact more naturally with robots. Picture for instance a retail worker, at a store with multiple robots: a floor scrubber, an inventory robot, a restocking robot and perhaps others in the future. Simple commands like "send the cleaning robot to aisle 14" or "tell the inventory robot to skip the cereal section while it's being restocked" are a natural way of interacting with robots. Moreover, roboteers will require less training and will be able to ramp up sooner.

There is significant potential ahead for these technologies as they converge and are used together to enable the next level of robot operations at scale.