How do you train robots to adapt to dynamic and uncertain environments?

Reinforcement learning (RL) is learning through trial and error. The formal framework of RL involves: Agent: learner that interacts with the environment and takes actions Environment: system that the agent faces and responds to the agent’s actions State space: set of possible situations that the agent and environment can be in Action space: set of possible actions that the agent can take Reward: numerical feedback that indicates how well the agent is doing Policy: strategy that determines what action the agent should take in each state Value function: that estimates how good it is to be in each state or to take each action in each state Model: function that predicts what the next state and reward will be given the current state and action

Reinforcement learning is similar to how humans learn a new skill - practice, feedback, and improvement. Consider the game of chess, where each move's consequence is immediate feedback. Over time, the player adjusts strategy to maximize winning chances. Similarly, RL algorithms guide robots, like making a chess move, with the environment offering feedback, rewarding success, and penalizing failure. This iterative process equips robots to make optimal decisions, even in complex, uncertain scenarios.

Two simple examples to understand human robot interaction are: 1. Using a voice assistant like Siri or Alexa to ask questions, play music, set reminders, or control smart devices. 2. Playing with a robotic toy like a Furby or a Lego Mindstorms that responds to touch, sound, or movement. Additionally, HRI also presents many difficulties and unresolved issues for researchers and practitioners. For example, how can we ensure that humans and robots can cooperate effectively and safely in complex and dynamic environments to achieve common goals?

To understand HRI, it's crucial to draw comparisons to human-human interaction. Robots not only need to learn to interact, but also to empathize and interpret nuanced behaviours, akin to a person reading body language. For example, a hospital robot needs to differentiate between a patient's discomfort and a mere change in posture. Similarly, in education, a robot might adjust its teaching style in response to a student's boredom or confusion.

To illustrate the importance of this topic, let me give you two examples of how transfer learning can enhance HRI: 1. Emotion recognition for rehabilitation: Researchers applied transfer learning to extract emotional signals from the facial features of patients with traumatic brain injury (TBI) and deployed an emotion-recognition model on Pepper robot to support their rehabilitation process. 2. Motor difficulty classification for pHRI: Researchers from UT Arlington developed a CNN model to estimate motor control difficulty using surface electromyography (sEMG) from the human upper limb during a physical human-robot interaction (pHRI) task and used transfer learning to adapt the model to new subjects.

Transfer learning plays a pivotal role in practical applications like robotic waste sorting. For instance, if a robot has been trained to identify and sort plastics, it can leverage this existing knowledge to identify different plastic types (e.g., PET, HDPE), without additional extensive training. This not only saves resources but also empowers the robot to adapt quickly to new sorting requirements, demonstrating the efficiency and flexibility of transfer learning in real-world scenarios.

Meta learning (ML) draws comparison to the human cognitive process, where we continually update our knowledge base and adapt to new situations. In practical terms, imagine a warehouse robot: traditionally programmed, it might falter if a box is misplaced. However, with ML, the robot could recognize the anomaly, drawing on past experiences of handling similar issues, and take appropriate action - thus improving efficiency and robustness. This flexibility gives ML-trained robots an edge, ensuring they remain competent even in the face of data variability or unexpected circumstances.

Curriculum Learning (CL) could be compared to how a human child progresses from basic algebra to more complex calculus, building on earlier knowledge for higher-level tasks. An apt example would be the use of CL in training AlphaGo, where initial lessons involved learning rules of Go from simpler board states before gradually escalating to complex games. Thus, CL helps robots manage learning overload, similar to pacing studies in human learning, promoting a healthier learning progression and more robust skill acquisition.

Venturing into the world of robotics, self-supervised learning (SSL) emerges as a potent way to train robots for unpredictable environments. This technique empowers robots to learn from their own data, sans external labels or assistance, allowing them to predict, reconstruct, or fill in gaps in their observations. Not only does SSL aid in enhancing their understanding of tasks like classification or detection, but it also bolsters their capability to assess their own confidence and manage uncertainty.

With the rise of the robots-human interaction in the future, it is important that we consider the ethical implications. We need to prepare and equip the workforce and consumers with the necessary knowledge so they know how to interact and work with robots. So if we had robots as co-workers, how are we going to interact? How will management evaluate the human performance compared to the robots? We also need to study and analyse the social impacts and put mitigations in place.

I think we need advice and inputs for robots to also grow emotionally. I'd bring in Dr Daniel Goleman and Dr Travis Bradberry to discuss ways in which we could give them human strengths of facing adversity, expanding on our creative ideas, responding to us in human ways. If we're making them and wanting them to evolve, wouldn't we want them to be more and more relatable to us?