[This story from Road to VR evaluates a new presence-evoking technology announced by the company Varjo, which is promoting its Reality Cloud platform as “allowing the ultimate science fiction dream, photorealistic virtual teleportation, to come true.” Here’s a little of what Varjo says in a blog post announcement:
“With Varjo Reality Cloud and Varjo XR-3, anybody will be able to capture their surroundings in 3D and then invite somebody else to join that same exact reality, see and hear others, and share their experience in absolute immersion. Instead of jumping on a plane, you will be able to teleport to any location and back instantly, making your actual physical location completely irrelevant. It won’t just be a simple hologram of an object or a static reality capture; we’re talking about a fully immersive, photorealistic real-time 3D stream, all the way down to the last pixel. And what’s best, you’ll be able to edit and modify the shared 3D reality together as you would edit a photo or a video. Want to add in new objects or tweak existing parts of the reality? Easy. It’s just bits.”
See the Road to VR story for two short videos and the blog post and coverage in VentureBeat for more details. –Matthew]
Hands-on: Varjo Reality Cloud is a Platform for Capturing & Sharing Physical Spaces in Real-time
By Ben Lang
June 24, 2021
Varjo today announced its new Varjo Reality Cloud platform, a device-agnostic VR meeting platform which can scan and share physical spaces in real-time using the depth sensors on its latest XR-3 headsets. The company doesn’t have a specific release date for the tech just yet, but is presenting a glimpse of what it plans to deliver in the future. Road to VR got an early look at the new tech.
Varjo Reality Cloud
Varjo is building a virtual meeting platform not unlike we’ve seen before, but with one key difference: the company plans to leverage the wide field-of-view depth sensors on its Varjo XR-3 headset to allow users to easily capture their surroundings and use it as the basis for virtual meetings. What’s more, beyond just a static capture, the Varjo Reality Cloud platform will continuously update the portion of the environment that’s in the headset’s view while the meeting is happening. That means if there’s something real and relevant in the host’s environment—like a book, product, or even another person—the virtual viewers will be able to see that thing moving and updating in real-time (as long as the host is actively looking at it).
The idea melds well with XR-3’s existing high-quality passthrough capabilities. During normal use of the headset it’s easy to toggle on the headset’s pass-through view to see the environment around you, which means if you were sharing your local environment through the Varjo Reality Cloud, it could seem like others in the meeting were standing right in the same room as you.
Varjo thus likens its Reality Cloud platform to ‘teleportation’, though I wouldn’t say it goes that far just yet.
Hands-on With the Prototype
I got to see an early prototype of the Varjo Reality Cloud in action during a meeting with the company in Silicon Valley. Using the Varjo XR-3 headset, I was shown a pre-recorded example of a Varjo Reality Cloud meeting space with a person standing in the center of the room talking, gesturing, and showing me some objects from around the room. While most of the room around me was static, the person was essentially being ‘filmed’ by an XR-3 headset, which meant their movements (and anything in a certain area around them) were being updated in real-time.
To be clear, the environment I was seeing wasn’t just flat or even 180 footage, it was an actual volumetric space, and so was the person that was standing inside the room. And while I could definitely make out the specific person I was looking at and the room around me, in this prototype phase the fidelity leaves a lot to be desired. The room scan and the person in front of me were assembled from a splotchy point-cloud of colored dots—far from the incredible quality of several of Varjo’s photogrammetry demos that I’ve seen in the past.
While it’s almost certain that the Varjo Reality Cloud won’t look as good as careful pre-captured photogrammetry any time soon, the company says that what I was looking at is merely a proof of concept, and that improvements in fidelity are expected as they move forward with development.
One important part of that ongoing development will be moving the whole thing into the cloud. While the demo I saw was a pre-recorded example of the Varjo Reality Cloud, ultimately the company plans to stream the captured environments from the cloud to any participants in the room, leaving the bulk of the computing to be done in the cloud. To do so at the highest possible quality on its ultra-high resolution headsets, the company says it has developed a foveated compression algorithm to cut the stream down to just “single megabytes per second.” My understanding is that that algorithm specifically takes advantage of the eye-tracking that’s built into Varjo headsets.
Device Agnostic
But Varjo headsets aren’t the only devices that will be able to join the Varjo Reality Cloud. While it’ll take an XR-3—with its equipped depth-sensors—to capture and stream the environments, the company says that it’s taking a device-agnostic approach to participants. The company expects that participants joining Varjo Reality Cloud sessions could be on computers, smartphones, tablets, and other VR headsets too.
Cool, but Revolutionary?
Varjo seems really excited about the Varjo Reality Cloud. Earlier this week the company’s Chief Product Officer teased that it is “probably the biggest announcement ever by Varjo.” But I’m not sure I get the hype. It’s not that the fidelity isn’t great in the proof of concept—I trust the company when they say it will get better—it’s just that I don’t see the major advantage of having virtual meetings in scanned environments, unless the environment itself is specifically of interest. Although you can see the host’s environment, it isn’t like you can interact with it. Beyond a ‘wow this is cool’ factor, I’m not yet convinced of the value.
When I asked Varjo’s executives about this, they brought up a theoretical example where a session could be held near a conveyor belt on a factory floor which has some sort of issue that needs to be diagnosed by remote technicians. Ostensibly you could have someone on the ground at the factory scan the area, then throw on their XR-3 headset so that others joining the meeting could get a look at the conveyor belt issue. But the idea that you’d bring a high-end PC down to the factory floor and hook up your tethered headset in order to facilitate the meeting seems… clunky in practice compared to simply sharing a livestreamed video from a smartphone.
The company also thinks its Varjo Reality Cloud could have even further reaching implications down the line. Not just for high-end enterprise applications, as Varjo serves now, but also in day to day life.
“We believe that Varjo’s vision for the metaverse will elevate humanity during the next decade more than any other technology in the world,” said Timo Toikkanen, CEO of Varjo. “What we’re building with our vision for the Varjo Reality Cloud will release our physical reality from the laws of physics. The programmable world that once existed only behind our screens can now merge with our surrounding reality—forever changing the choreography of everyday life.”
A decade is surely a long time for Varjo to change my mind, but again, I’m just not quite seeing the paradigm-shifting value that the company does, even in a theoretical future where this tech is available to consumers. Like, it would be great to invite a friend to my virtual home—at least so that we’re in a familiar space—but it isn’t like they could join me on my couch or sip wine poured fresh from the bottle. So why not just meet in one of many synthetic virtual spaces that are already available today?