VR teleoperation
For my master’s at UT Austin, I built the VR interface our lab used to teach a humanoid robot household tasks. Wearing the headset, you could see through the robot’s stereo cameras and move its hands with your controllers. That’s how we filmed the ramen demo above.
I wrote the interface in C++ so we could use the same headset with our MuJoCo simulation and the real robot. In simulation, asking the headset where your hands were took about 70 milliseconds per round trip. Streaming the poses let the simulator read the latest one in about 0.1 milliseconds.
The red and blue arrows show where I’m asking each hand to go. The gap between an arrow and the gripper shows how far the robot still has to move. Imitation learning
We recorded 250 demonstrations per task and trained policies to repeat them. On the real robot we ran ten trials of each task in a row. The robot took the cap off a spray can nine times and put a tool back in its box eight times.