[This VentureBeat story describes how Volvo is using Varjo’s high-resolution HMDs to let car designers test drive real cars with virtual features: “With this kind of technology it will eventually be difficult for you to tell if you are sitting in a faux car in an office or in a real car in a parking lot or really driving a mountain road in the woods.” See the original version for more pictures. –Matthew]
[Image: For safety, a passenger in the back seat can warn a driver in the front seat during Varjo’s testing. Credit: Varjo.]
Varjo teams up with Volvo to enable safe driving with AR headsets
Dean Takahashi
May 29, 2019
Varjo has announced a partnership with Volvo to enable safe driving with an augmented reality headset that you could eventually wear while operating a vehicle. It can give you visual alerts, using computer graphics to highlight hazards on the road.
Volvo is using Varjo’s XR-1 developer edition headset, which is like a VR headset but with pass-through video viewing so you can see what is going on in the outside world through cameras attached to the headset. Sweden-based Volvo is also investing in Varjo.
Wearing a headset on your face while you’re driving may sound nuts. But Volvo’s Casper Wikman, technical leader for R&D, said in an interview that the company has already done tests with people driving on roads in Scandinavia while wearing the headsets. The company is showing the tech at the Augmented World Expo in Santa Clara, California this week.
The developer edition is aimed at engineers, researchers, and designers who can use it to experiment with AR and mixed reality — the combination of digital headsets and physical locations. The goal is to create mixed reality applications that are indistinguishable from the real world so you can’t tell what is real and what is virtual.
“We have been in close collaboration with Volvo for over a year,” said Urho Konttori, founder and chief product officer at Helsinki-based Varjo, in an interview with VentureBeat. “We press a button and you see the wire frame. Then we press another button and it becomes a real car. You can see a semi-transparent heads-up display that Volvo is testing.”
I tried it out yesterday and it was pretty amazing. I sat in a chair in an office, put on the headset, and then watched an augmented reality car form around me. At first, it was just the wire frame, or blueprints, of the car that came into my view within the headset. Then the Varjo folks flipped a switch and the full realistic version of the car materialized. I felt like I was sitting in a car, only I knew it couldn’t have been a real one because my head was bumping the virtual ceiling of the car.
I looked out the window and saw desks in the office. (You can see what that experience is like in this video, this video, and this video). But then they switched the view again. All of a sudden, my car was in the middle of a street in Venice. I looked around and saw brick buildings. I could see a heads-up display on the windshield of the car that Volvo was only testing in digital form. And then a moose walked across the street, and my AR glasses identified it as a road hazard that required my attention.
“We can create a scene that is fully immersive and realistic with a full field of view,” Konttori said.
It was pretty realistic, without many of the artifacts you typically see in AR demos. But I did not get a chance to drive a car with headsets on it, and it may be a long time before many people are able to do that.
With this kind of technology, it will eventually be difficult for you to tell if you are sitting in a faux car in an office or in a real car in a parking lot or really driving a mountain road in the woods. That’s the pretty cool idea that inspired the engineers at the Swedish carmaker to work on the technology.
The VR headset started shipping earlier this year. The add-on camera module and the headset will ship to developers in the second half of 2019. Pricing will be disclosed later, but this is one more example of an AR/VR company attacking the enterprise market first, where precision technology is more important than a low price for an initial product.
The resolution of the headset displays is much better than what is in the market today in the form of Oculus or HTC Vive VR headsets. It has two 12-megapixel cameras that can create images with a resolution of more than 4K per eye, Konttori said. This is what enables the photorealistic image quality. The headset has integrated eye tracking so it can properly render the images from the correct point of view.
It can handle this kind of resolution because the headset is connected via wire to a powerful PC with a Nvidia GeForce RTX 2080 graphics card. If you put this headset in the car, it would have to be wired via a Thunderbolt 3 cable to a PC in the backseat or the trunk. The headset itself weighs about 2.6 pounds.
The XR-1 upgrades Varjo’s human-eye resolution headset (VR-1) with a front plate featuring the cameras with video pass-through mode. The device uses cameras to digitize the world in real time and then multiplexes it inside the graphics processing unit (GPU) with the virtual content to show the combined result to the user. The Varjo VR-1 is the only device with resolution high enough to make the seamless visual blending possible, and the XR-1 has the only camera technology capable of producing high resolution with unperceivable latency (less than 15 milliseconds), according to Konttori.
Unlike competitor devices that are delivering mixed reality with ghostlike, semi-transparent renderings in a limited view, the XR-1 enables photorealistic mixed reality with a 90-degree field of view. That’s not bad, but we actually see the world with a 220-degree field of view, which means your peripheral vision is cut off in this headset.
To me, that’s not so cool for driving conditions. I was in a car wreck once because I didn’t see a car coming straight at me from the side. But Wikman said the field of view is good for seeing the road ahead of you, and Volvo is investigating what can be done to make the driving experience safer by showing you alerts and other computer overlays in digital form.
With the XR-1, virtual objects appear as real as anything in the physical world and can themselves cast shadows or even illuminate reality. Black objects appear truly black, opaque objects indeed block real or virtual light, and semi-transparent objects will refract light from the real world behind them. With the what the company touts as the industry’s most advanced color-matching technology, the colors are perfectly rendered for product trials and showcases.
For the first time, users can also switch seamlessly between mixed reality and full virtual reality modes. The depth sensors in XR-1 allow mapping of real-life objects and environments to build natural occlusion. Examples of industrial applications for the device include UX design, collaboration with photorealistic 3D models, training, and simulation, as well as unique research and data analytics across real and virtual environments with XR-1’s 20/20 Eye Tracker.
Volvo is using the device to test-drive virtual car designs on the road, which is something you can’t do in a car simulation, Wikman said. Volvo can test these ideas for things like the heads-up display many years before they exist in physical form. Volvo has used XR-1 prototypes for this new workflow since the summer of 2018.
By adding virtual elements to the interior of the car, Volvo designers are able to perform design studies of future cars before they are even built. Volvo Cars engineers have also been test-driving a real car with the driver wearing an XR-1 headset. Adding photorealistic virtual elements or complete features to the test scenarios while driving (world-first) enables UX concepts to be quickly iterated. The car designers can add virtual automobiles or unexpected road hazards (such as a virtual moose) to the road for realism, which results in faster turnaround times and ultimately safer cars.
Volvo has been impressed enough by Varjo’s technology that the two are no longer just partners. As of May 2019, Varjo will receive an investment from the Volvo Cars Tech Fund, the Swedish carmaker’s venture capital fund that invests in high-potential technology startups. Founded in 2016, Varjo has about 145 employees.
“From the very beginning, our vision has been to create a product that can seamlessly merge the real and the virtual together,” said Varjo founder and CEO Niko Eiden in a statement. “The incredibly advanced ways in which Volvo Cars uses the XR-1 show that Varjo’s technology enables things that have been previously impossible. Together with Volvo, we have started a new era in professional mixed reality.”