[New software developed at the University of Michigan not only expands the potential community of people who can use virtual and augmented reality systems but seems likely to enhance the experience of presence for all users. This story is from the University’s Michigan Engineering News, where the original includes a 1:27 minute video (also available on YouTube). Coverage in Heise includes this additional detail: “Currently, ‘HandProxy’ exists only as a demo app, but in the long term, the technology could also be incorporated into real robotic arms. A patent has already been applied for, and the team is looking for partners for a possible market launch.” –Matthew]
[Image: HandProxy touches objects inside a demo app. The hand is guided by voice commands. Credit: Marcin Szczepanski, Michigan Engineering.]
This digital hand enables hands-free virtual reality
More than just a stand-in, the AI-powered agent can complete tasks by following simple voice commands that don’t include nitty-gritty details.
By Derek Smith
October 13, 2025; updated October 15, 2025
A digital, voice-controlled hand could improve the convenience and accessibility of virtual and augmented reality by enabling hands-free use of games and apps. The prototype software was developed by computer scientists at the University of Michigan.
The researchers’ software, called HandProxy, allows VR and AR users to interact with digital spaces by commanding a disembodied hand. Users can ask the hand to grab and move virtual objects, drag and resize windows, and perform gestures, such as a thumbs up. It can even manage complex tasks, such as “clear the table,” without being told every in-between step, thanks to the interpretive power of GPT-4o, the AI model behind ChatGPT.
The hand’s ability to independently parse complex tasks on the fly makes it more flexible than current VR voice-command features, which are limited to simple, system-level tasks, such as opening and scrolling through menus, or predefined commands within an app or game.
“Mobile devices have supported assistive technologies that enable alternative input modes and automated user-interface control, including AI-powered task assistants like Siri. But such capabilities are largely absent in VR and AR hand interactions,” said Anhong Guo, the Morris Wellman Faculty Development Assistant Professor of Computer Science and Engineering.
“HandProxy is our attempt to enable users to fluidly transition between multiple modes of interaction in virtual and augmented reality, including controllers, hand gestures, and speech,” said Guo, who is also the corresponding author of a study describing the software, published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.
Enthusiasts praise VR for its immersion. Users want to be inside a virtual space, not just viewing it from the outside. The benefits, they claim, range from making games more exciting to training doctors and surgeons without risking lives.
Maximizing physical realism is key for suspending disbelief, so the industry has moved toward tactile control with hand-tracking cameras and gloves. But the focus on life-like hand motions isn’t the ideal method for certain people and situations. VR users in cramped spaces might not have room for complicated gestures, and AR users may want to navigate small displays while their hands are full with cooking or cleaning.
A strict reliance on hand gestures becomes even more cumbersome for users who have motor impairments or other disabilities. People with muscular dystrophy and cerebral palsy have difficulty using VR, Scientific American reports. Tactile motions can even dissuade some users with chronic illness from even trying VR. One Redditor shared that a chronic illness prevents them from enjoying games with repetitive swinging motions, and they were skeptical that VR would be right for them. HandProxy could help make VR more comfortable and approachable.
Leave a Reply