[Facebook is developing a technology that displays an external view of the eyes of the user of a virtual reality headset, creating a presence illusion for other people in the room and reducing the sense of isolation of the user. The current prototype is described in the Road to VR story below, where the original includes two more images and two short videos. –Matthew]
Facebook Researchers Show ‘Reverse Passthrough’ VR Prototype for Eye-contact Outside the Headset
By Ben Lang
August 3, 2021
Researchers at Facebook Reality Labs today published new work showcasing a prototype headset which has external displays to depict the user’s eyes to others outside of the headset. The goal is to allow eye-contact between the headset wearer and others in an effort to make it less awkward while wearing a headset and communicating with someone in the same room.
One of my favorite things to do when demoing an Oculus Quest to someone for the first time is to put on the headset, activate its ‘passthrough view’ (which lets me see the world outside of the headset), and then walk up and shake their hand to clearly reveal that I can see them. Because Quest’s cameras are at the four corners of the visor, it’s not easy to imagine that there would be any way for the user to see ‘through’ the headset, so the result from the outside seems a bit magical. Afterward I put the headset on the person and let them see what I could see from inside!
But this fun little demo reveals a problem too. Even though it’s easy for the person in the headset to see people outside of the headset, it isn’t clear to people outside of the headset when the person in the headset is actually looking at them (rather than looking at an entirely different virtual world).
Eye-contact is clearly a huge factor in face-to-face communication; it helps us gauge if someone is paying attention to the conversation, how they’re feeling about it, and even if they have something to say, want to change the topic, or leave the conversation entirely. Trying to talk to someone whose eyes you can’t see is uncomfortable and awkward, specifically because it robs us of our ingrained ability to detect this kind of intent.
But as VR headsets become thinner and more comfortable—and it becomes easier to use passthrough to have a conversation with someone nearby than taking the headset off entirely—this will become a growing issue.
Researchers at Facebook Reality Labs have come up with a high-tech fix to the problem. Making use of light-field displays mounted on the outside of a VR headset, the so called ‘reverse passthrough’ prototype system aims to show a representation of the user’s eyes that’s both depth and direction accurate.
In a paper published this week for SIGGRAPH 2021, Facebook Reality Labs researchers Nathan Matsuda, Joel Hegland, and Douglas Lanman, detailed the system. While to external observers it appears that the headset is very thick but transparent enough to see their eyes, the apparent depth is an illusion created by a light-field display on the outside of the headset.
If it was instead a typical display, the user’s eyes would appear to float far away from their face, making for perhaps a more uncomfortable image than not being able to see them at all! Below researcher Nathan Matsuda shows the system without any eyes (left), with eyes but no depth (middle), and with eyes and depth (right).
The light-field display (in this case a display which uses a microlens array), allows multiple observers to see the correct depth cues no matter which angle they’re standing at.
What the observers see isn’t a real image of the user’s eyes however. Instead, eye-tracking data is applied to a 3D model of the user’s face, which means this technique would be limited by how realistic the model is and how easy it is to acquire for each individual.
Of course, Facebook has been doing some really impressive work on that front too with their Codec Avatars project. The researchers mocked up an example of a Codec Avatar being used for the reverse passthrough function (above), which looks even better, but resolution is clearly still a limiting factor—something the researchers believe will be overcome in due time.
Facebook Reality Labs Chief Scientist, Michael Abrash admits he didn’t think there was much merit to the idea of reverse passthrough until the researchers further proved out the concept.
“My first reaction was that it was kind of a goofy idea, a novelty at best,” Abrash said in a post about the work. “But I don’t tell researchers what to do, because you don’t get innovation without freedom to try new things, and that’s a good thing, because now it’s clearly a unique idea with genuine promise.”
– – — – –
It might seem like a whole lot of work and extra hardware to solve a problem that isn’t really a problem if you just decided to use an AR headset in the first place. After all, most AR headsets are built with transparent optics from the outset, and being able to see the eyes of the user is a major benefit when it comes to interfacing with other people while wearing the device.
But even then, AR headsets can suffer from ‘eye-glow’ which obstructs the view of the eye from the outside, sometimes severely, depending upon the optics and the angle of the viewer.
AR headsets also have other limitations that aren’t an issue on VR headsets, like a limited field-of-view and a lack of complete opacity control. Depending upon the use-case, a thin and light future VR headset with a very convincing reverse passthrough system could be preferable to an AR headset with transparent optics.