The app, called Tracking Streamer, tracks a person's movements, including head, wrist and finger movements, and transmits that data via a Wi-Fi connection to a robot on the same network. (Image credit: Younghyo Park via X)
Researchers have developed an app for Apple's Vision Pro mixed reality headset that allows users to control a robot using head and hand gestures. The app could be used to remotely control devices in a variety of situations, from pranks to navigating disaster zones.
The Tracking Streamer app records human movements—specifically head, wrist, and finger movements—and transmits that data via Wi-Fi to a robot on the same network. The connected robot then interprets that data into appropriate movements.
The researchers published their findings in a short paper published on March 9 on GitHub, an open-source platform. Their system tracks 26 points on the hands and wrists, as well as specific points on the head. It also records spatial data, such as height above the ground.
The app's developer, Younghyo Park, a graduate student at MIT, posted a video on the X website, formerly known as Twitter, showing the system in action.
In a short video, study co-author Gabe Margolis, a graduate student at MIT, controls the quadruped robot using arm and body movements.
🥽 Want to use your new Apple Vision Pro headset to control a robot? Want to record yourself moving around/manipulating the world to teach a policy? I made a VisionOS app that can stream your head/wrist/finger movements over Wi-Fi that you can subscribe to on any machines using… pic.twitter.com/l5CbeINWWfMarch 9, 2024
Margolis gestures to tell the robot to walk up to a closed door, turn the handle with its grip, and enter. In another sequence, he gestures to the robot to lift the plastic lid of a trash can and throw it in the trash. At one point, he also bends over — prompting the robot to mimic him.
Apple released its Vision Pro virtual reality (VR) headset in February 2024, and footage has since emerged of people using the device in their daily lives. Shortly before its launch, scientists presented a study that looked at how well VR headsets work in everyday settings.
They found that the devices had latency, limited peripheral vision, and caused distortions similar to a funhouse mirror. They cautioned against wearing VR headsets while driving or walking down stairs.
Park and Margolis suggested in their study that some users might wear the Apple Vision Pro constantly, like glasses, and that constant use would provide more data that could be used to teach robots how humans move.
Keumars Afifi-SabetSocial Links NavigationChannel Editor, Technologies
Kumars is the technology editor at Live Science. He has written for a variety of publications including ITPro, The Week Digital, ComputerActive, The Independent, The Observer, Metro and TechRadar Pro. He has worked in tech for over five years, previously serving as features editor at ITPro. He is an NCTJ-qualified journalist and has a degree in biomedical science from Queen Mary, University of London. He is also a registered charter member
Sourse: www.livescience.com
