Project GR00T: Training Robots through Large-Scale Simulation Frameworks

Welcome to the summary of the nineth lecture of the LLM Agents course conducted by University of California, Berkeley. Refer to this link for the summary of the previous lectures.

The success of NLP is traced back to the inception of specialist models capable of solving the key functions such as sentiment analysis and information retrieval. On top of this, the generalized models (Ex: ChatGPT) are built to resolve any tasks given the prompt, while the specialized generalist models (Ex: travel planning, coding) are derived by fine-tuning and distilling the generalized models. Following the success of NLP and inspired by how humans continually learn and adapt in the open world, the objective of GR00T is to build the embodied AI for the humanoids. It is guided by three principles namely, data pyramid, the matrix? and foundation agent. Most of the current robot systems are specialists requiring special hardware and dedicated pipeline for each use case. The main challenge in transforming the robot systems is it is very difficult to collect the data required for training the robots. To accelerate the data collection, the data pyramid is built on the data from the real robot (teleoperating robots through omniverse cloud), simulation (running simulations on GPU) and internet (for training foundation models). According to the matrix principle, it is efficient to train the robots from the simulation data as it is easier to simulate a problem than to solve it. MineDojo is an open-source framework to build generally capable agents through simulations. ?

Two use cases of simulation are the reinforcement learning and imitation learning to train the robots. HOVER (Humanoid Versatile Controller) is a model trained by reinforcement learning, and distilled to teleoperate the robot, collect data and control the whole body movement (kinematic position tracking, joint angle tracking) of humanoids. While the imitation learning ?takes more time to collect the data through human demos, the data is multiplied through text-to-3d, stable diffusion, and LLMs for generating hand-made objects, scenes and tasks, respectively. RoboCasa and MimicGen are the large-scale simulation frameworks used to augment the human demos for training the generalist robots in kitchen environments and diverse machine-generated tasks, respectively.

The third principle of foundation agent emphasizes building a foundation model capable of mastering different embodiments, skills, and tasks. The robotic systems are mapped into three coordinates: embodiments (types of robots), skills, and reality. Metamorph is developed as a single neural network used to control 1000 different robots (graph of joints). MimicGen is a method to train the robot across multiple skills. To further automate this process, Eureka is built as a dual-loop system involving an LLM in the outer loop to generate the reward function, while using reinforcement learning in the inner loop to achieve the target task (pen spinning simulation). To transform the simulation into reality, DrEureka uses LLM to implement the domain randomization (varying the physical parameters such as gravity, friction) to overcome the imperfections in simulation. Due to this, the complex tasks, such as a robot dog walking on a yoga ball, were transferred zero shot to the real world. The outcome of this project leads to the NVIDIA OSMO for orchestrating the training of robots using small amounts of human demonstrations.