Down the rabbit hole of Robotics and reinforcement learning AI

Some say the term "robotics" was coined by Isaac Asimov in 1941, but the concept of automated machines can be traced back to the Greeks and Romans. Hephaestus was responsible for crafting the palaces and thrones, and the metalworking god also had a workshop on Mount Olympus. Hephaestus constructed automatons to help him in this workshop, automatons which could use the forge's bellows, and also work the metal in the fire.?

The term "robot" itself has a more dramatic origin, stemming from the Czech word for forced labour, and was popularized through Karel ?apek's play featuring humanoid robots. The real-world definition of “robot” is just quite hard to box into one category.?

Ask 10 experts and you’ll get 10 different answers, but what we can say is that a robot is an intelligent, physically embodied machine. A robot can perform tasks autonomously to some degree. And a robot can sense and manipulate its environment.

Sort of. Let me explain.

Back in the 70s, we find ourselves in the midst of an industrial revolution, where robots were being used to simplify production automation. Robots were rudimentary, often bulky machines, confined to the realms of science fiction and highly specialized industrial applications. They were more a novelty than a necessity, a testament to human ingenuity rather than a tool for societal transformation.

The car industry played a pivotal role in this development, with 通用汽车 adopting robotic automation in 1962, specifically for welding in the automotive industry. This marked a major milestone in the history of robotics, as UNIMATE ROBOTICA robot arms revolutionized manufacturing, performing precise tasks with efficiency.

Today, there are lots of robots in the automotive industry, which is where industrial robotics started. In fact, the entire production line is automated. Just think of the Tesla Gigafactory concept.?

To simplify, it starts with taking metal sheets from stacks and feeding them to the press lines. Then, the parts are transferred and moved to the body shop. In the body shop, the parts are assembled and welded using various joining technologies such as spot welding and laser welding. After that, they go into the paint shop where the body is painted. Then comes the “marriage” stage where the powertrain and body are put together, creating a complete car. The final assembly and trim involve adding the dashboard, seats, and wheels. This is where most of the robots were developed already in the '80s and '90s.

However, after 2000, robotics expanded into many other areas. Robots are now used in the food and beverage industries for picking, packing, and palletizing tasks. There has also been significant development in using robots for grinding, polishing, and assembling components in the consumer electronics industry. More recently, robotics has been employed in logistics, restaurants, and hospitals. The story of robots is just at the beginning.

Yet, we still don’t have home robots that can assist with the bloody dishes or the laundry. There are robots like iRobot Roombas for home use which are great, but we haven't seen a general-purpose home robot that can do everything. However, it’s probably just a matter of time, because robots are already impacting our daily lives.?

The most common type used for industrial applications is the serial kinematic robot, which has a series of joints and links on top of each other that resemble an arm. Another type is the parallel kinematic robot, often referred to as “spider robots”, which can move very quickly. There are also scatter robots with different mounting axis configurations. The right choice of kinematics depends on factors such as reach and payload, which are crucial for industrial applications.

As we moved into the 21st century, the field of robotics began to evolve beyond industrial applications. Many people are familiar with Boston Dynamics , a robotics company that gained a lot of attention with amazing videos. While their robots may be more visually appealing and tend to be full-bodied, industrial robots like 阿西布朗勃法瑞公司（ABB） ones look different and are more widespread.?

A new robotics landscape.?

The rise of service robotics, powered by advancements in artificial intelligence, has opened up new possibilities and applications. Today, robots are not just used in factories but are also found in hospitals, homes, and even on the streets in the form of self-driving cars.

The integration of AI into robotics has been a game-changer. AI has enabled robots to learn from their environment and adapt to new tasks without explicit programming. This has led to the rise of collaborative robots (cobots) working alongside humans, handling tasks such as assembling, drilling, and welding, allowing humans to focus on high-value tasks like quality assurance.

Healthcare is another huge field of application. Robots are now used not only in the operating room but also in clinical settings to support healthcare workers and enhance patient care. For example, hospitals and clinics are deploying robots for a much wider range of tasks to help reduce exposure to pathogens during the COVID-19 pandemic.

The use of robotics and automation also extends to research laboratories where they are used to automate manual, repetitive, and high-volume tasks so technicians and scientists can focus their attention on more strategic tasks that make discoveries happen faster.

Streamlined workflows and risk reduction provided by medical robotics offer value in many areas. For example, robots can clean and prep patient rooms independently, helping limit person-to-person contact in infectious disease wards. Robots with AI-enabled medicine identifier software reduce the time it takes to identify, match, and distribute medicine to patients in hospitals.

As technologies evolve, robots will function more autonomously, eventually performing certain tasks entirely on their own. As a result, doctors, nurses, and other healthcare workers will be able to spend more time providing direct patient care.

But perhaps one of the most prominent fields is Agriculture. Most people don’t realise it, but there is a tremendous workforce shortage. We're truly risking not having food. If you haven’t seen it yet, Clarkson’s Farm show on Amazon Prime is a good glimpse at the complexity behind food production. According to the UK Office for National Statistics, almost 3 in every 10 farmers in England are aged 65 years or older. Similarly, in the EU, almost 31% of farms are managed by persons aged 65 or over.?

The ageing of rural populations is a global phenomenon, and a report by HelpAge International notes that the proportion of older farmers is increasing in low and middle-income countries as well.?

Can you imagine that? No farmers on the horizon. So, there's definitely something we must do about it. Works that traditionally were carried mostly by humans are challenging. They require competence, mobility, vision, grasping, and connectivity. The building blocks are there, and there are several startups doing great jobs.?

In five years, the next big thing is going to be robots in agriculture, weeding, seeding, and harvesting, also enabling urban farming. All in all, to gain productivity, save water, and have better food at better prices. The things that have happened in automotive may also happen in this food revolution. Still, in food, there are huge challenges in food processing that are very tough and risky, such as in butchery and cutting processes. Boning processes are also very complex and may require a lot of assistance. These processes will probably be enabled by robots.

If we go to the other extreme when we're thinking about service robots, now we see Spot from Boston Dynamics or ANYmal from ANYbotics helping to work around civil works, construction, and inspections. Robots are doing inspections and surveillance. They are also guiding us in places like museums. We can foresee that robots will finally come into unstructured environments, and this will happen through the visual perception part, through more advanced locomotion and mobility. All that put together will enable a lot of new opportunities for robots to support humans.

The whole AI wave that's happening right now is really going to open up a lot of these opportunities that used to be just impractical to be accessible to robots. But now, all of a sudden, if they can understand what's around them and react to it, there are countless applications that could benefit us as humans.?

There are already many robots out there performing various tasks for us. It's important to recognize that there are two main types of robots: those that excel at repetitive tasks and those that can make autonomous decisions and move independently. Most of the robots currently in use fall into the first category, where they excel at repeating a single task that has been programmed into them.?

For example, as explained above, robots in car manufacturing plants or electronics manufacturing sites are often repetitive stations that perform the same action over and over again. However, robots have not been as successful in making autonomous decisions, perceiving the world around them, and learning and adapting as they go along. This is the part where robots have struggled to scale.?

The gap between the two types of robots primarily exists as an AI challenge. While the hardware and mechatronics of robots are already developed, the AI side is still lacking. This requires AI scientists and researchers to build advancements that enable robots to move and make decisions autonomously in the real world.

Bridging this gap is one of the main reasons why companies like Covariant exist. Covariant is an AI robotics company that was founded in a UC Berkeley lab with the world's premiere researchers, whose work gave rise to modern AI. In fact, Covariant key people were part of OpenAI . The company's mission is to apply cutting-edge research to building AI robotic systems, starting with warehouse operations.?

The Covariant Brain is the company's universal AI platform that delivers value on Day One, empowering robots to learn everything so they can pick anything. The platform is trained on millions of picks from Covariant robots in warehouses around the world, enabling robots to autonomously pick virtually any SKU or item on Day One. Covariant robots learn general abilities such as robust 3D perception, physical affordances of objects, few-shot learning, and real-time motion planning.

In 2017, Covariant's founding researchers set out to take their advancements from a lab to the real settings in need of AI Robotics. Five years later, the company's growing team around the world continues to set a new standard for what's possible in AI and the industries it's applied to. Leading warehouse integrators and customers worldwide select the Covariant Brain as their AI Robotics platform to automate robotic picking.?

The company is improving how the world works every day by building robots with the ability to see, think, and act, that’s why this is potentially a huge game-changer.

Logistics is one area where AI and robotics play a significant role, although may not be highly visible in our everyday lives. Logistics refers to the movement and manipulation of goods, and it impacts our lives in various ways.?

In modern warehouses, automation has significantly improved the movement of goods. Technologies such as conveyor belts and mobile robots are used to automate the movement of standardized containers within warehouses. However, the manipulation of goods remains a challenge because items vary in terms of packaging materials, geometry, and other factors.?

This is where Covariant focuses its efforts currently. They aim to develop AI that can manipulate a wide range of items efficiently, regardless of their characteristics. The manipulation tasks include order picking, parcel induction, and put wall operations. Order picking involves the selection and packing of items for customer orders placed online. Parcel induction focuses on the efficient handling and routing of packages in express courier and mail delivery services. Put wall operations entail picking and sorting tasks in warehouses.

These manipulation tasks require advanced AI capabilities because of the diversity of items and the need to handle tens of thousands of different kinds of items in typical scenarios. Furthermore, the items can change over time as trends shift and new products are introduced. This complex problem requires the development of advanced AI algorithms to handle the diverse range of items encountered in logistics operations.

The answer to this complex problem? Reinforcement learning.

It is a concept that revolves around decision-making and the fundamental goal of reinforcement learning is to make intelligent decisions that can lead to desired outcomes. Unlike supervised learning, where explicit labels are provided, reinforcement learning operates on a reward-based system. Instead of specifying the exact actions to take, the system is given a reward that measures the outcome of its decision-making process.?

It is believed that the road to a sentient machine will be paved by reinforcement learning.

The system explores different actions, and through a combination of successes and failures, it learns which actions lead to favourable outcomes. This outcome-driven decision-making framework is employed in various domains, including robotics, where the objective might involve tasks such as successfully picking up an object or manipulating it efficiently.

What sets reinforcement learning apart from other learning methods is its ability to adapt to new situations and generalize from past experiences. Unlike supervised learning, where the system relies on labelled examples, reinforcement learning allows the system to explore and discover optimal actions by trial and error.?

This aspect of exploration enables the system to handle new scenarios and unknown items effectively. Additionally, reinforcement learning can be applied to a wide range of problems beyond robotics. It is a versatile method applicable whenever decision-making is required, and it is challenging to specify the best course of action explicitly. Examples of reinforcement learning applications include training AI agents to play video games and optimizing energy consumption in data centres.

The beauty of reinforcement learning and the AI software developed by Covariant, referred to as the "brain," lies in its capability to be deployed on different robot hardware configurations, making it a generalized AI platform. The system leverages fleet learning, where the experiences and data collected by robots across different locations and scenarios are shared and used to improve the overall performance. This collaborative decentralised learning approach allows robots to handle new items even if they have never encountered them before.

When evaluating the performance of AI-driven robotic systems, a performance-based assessment approach is gaining prominence. This approach involves conducting tests or exams to measure the intelligence and capabilities of the system. Just as humans are evaluated through exams to assess their mastery of a subject, robots are evaluated through tests to measure their intelligence and competence.?

These assessments provide a statistically significant measurement of the system's performance, enabling a more accurate evaluation. Performance-based assessment helps determine the system's ability to generalize and handle diverse scenarios effectively. It ensures that the AI solution is reliable and can perform well across various tasks and environments.

In conclusion, reinforcement learning is a decision-making framework that focuses on achieving the best outcomes through decentralised exploration and learning from successes and failures. It finds applications in robotics and beyond, allowing AI systems to adapt to new situations and generalize from past experiences. Performance-based assessment plays a crucial role in evaluating the intelligence and competence of AI-driven robotic systems, ensuring their reliability and effectiveness.

This gives AI the ability to make decisions as opposed to just understanding – fundamentally changing its impact on society.

Need more digging? Here you can find additional resources

Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots https://arxiv.org/abs/2103.14295

Reinforcement Learning in Robotics: A Survey: https://www.ri.cmu.edu/pub_files/2013/7/Kober_IJRR_2013.pdf

Scalable Deep Reinforcement Learning for Robotic Manipulation https://ai.googleblog.com/2018/06/scalable-deep-reinforcement-learning.html

The Ingredients of Real World Robotic Reinforcement Learning

https://bair.berkeley.edu/blog/2020/04/27/ingredients/