AI at Meta的动态

Transcript

I think our demo, based on what I've seen is the first demo which operationalizes human robot collaboration. Largely there were like these three challenges, a mix of software and research. The first challenge is that we want to make sure humans behave differently, everyone behave differently in the real world, right? So we want to bring that concept into the simulators as well. And so we designed the system that could be, that would allow both for realism and also for like very fast speed so that we could train our robots in simulation in a scalable way. And so a lot of the work that we did was to build this human models based on real world data, based on real scans of humans, like 3D scans that were allowed to have different human shapes and. And feel appearances, but also based on recordings of real human emotions. How do we even get the raw information from the human perspective at all? How do we find a sensor? So we found Quest 3 for that. We needed a sensor not only to get this information, but also show the human what the planner is thinking, what the robot is doing. I'm thirsty, can you get something to drink? And that too on a limited resource hardware device, we thought would make our research more explicit and understandable. Another thing that we showed was once a human. Action is detected as a human picked up cup. We show that so that the user understands that ohh, their action has been registered and they can get that feedback right, just like traffic lights. I think we have a lot of moving components that will contains the human sensors, the mixed reality visualization app and also the robust skills. When you have many components, moving components together in our central stack, that creates a lot of challenges. So you need to have a way to quickly debug your assistance as quick as possible. And so we come up with an approach where we can, sometimes we can fake a certain components. Black component is not ready, but still the other components can be either input or output from like components to work and so that everyone in the team can work in parallel without like blocking each other. We put all of it together and we get this real world. It's like magic moment. People put on quest and they say, oh, the robot is working and your quest can actually tell you what the robot is going to do or what the robot is thinking about right now. And that's just not a modality in which we think or which we have interacted with robots. A lot.