[This long piece from Backchannel provides an optimistic but realistic status report on the evolution of presence technologies, including details on Google’s new stand-alone VR headset; the original version of the story includes more images. –Matthew]
Inside Google’s Slow-Mo VR Moonshot
Clay Bavor knows immersive computing is a long-term project. Here’s what he’s doing to make it happen faster.
Steven Levy, Editor of Backchannel
May 17, 2017
No one in Silicon Valley loves virtual reality or believes in its future as much as Clay Bavor. As vice president of VR and AR for Google, he’s a passionate advocate for the technology, with which he has been obsessed since he was a teenager. In Bavor’s three years of involvement with the company’s efforts in artificial realities, he has taken a populist approach, introducing accessible mobile phone-based products such as the dirt-cheap Cardboard viewer and the more recent $79 Daydream viewer.
Today, at Google’s big I/O developer conference, he’s announcing new moves that edge the company away from the VR dollar store—if not quite into the high-rent district. The splashiest of this bunch is an instant-on, standalone headset — think of it as a Daydream viewer with the phone built in, optimized for VR. Google has built a prototype “reference model” of this headset with Qualcomm’s help, and in the coming months Lenovo and HTC VIVE will release sleek commercial versions. The price is expected to fall in the mid-hundreds range—similar to the higher-end VR rigs sold by Oculus and HTC, but without the $1,200 or so supercharged computer that those products require.
Yes, you get what you pay for. The computation in these new devices is more akin to that of the phone than the supercomputer. But Google will announce at an I/O session tomorrow that it has come up with a scheme — codenamed Seurat, after the painter — to produce graphics that look as good as those from much higher-priced systems.
Google has other news: an augmented-reality version of its Expeditions classroom application, and advances in its Tango phone-based navigation system.
But just as significant, and maybe more so, is something that Bavor isn’t introducing in his presentation. It’s… an essay. Bavor plans to publish it the moment he hits the I/O stage to introduce Google’s new advances. Though it reflects his optimism about the field, it deals frankly with a problem that resonates throughout the entire bespoke-reality business: The hype about VR might be driving unrealistic expectations.
For months now, people have been asking him, What’s taking so long? Where are the transformative applications in virtual reality (which replaces your sight and hearing with an artificial world) and its cousin augmented reality (which layers artificial objects on top of the real world)? Speaking of which, doesn’t it seem like everyone is now talking about AR, maybe because VR isn’t moving fast enough?
Our expectations have been stratospheric. “It’s hard to read Techmeme or Wired or TechCrunch, especially in the first half of last year, without predictions that VR would solve all known problems in the immediate future,” Bavor says. These technologies were supposed to change our lives. They were touted as the next stage of computing. So far, it’s just not-so-cheap thrills and big promises.
Bavor addresses those questions in an epic post — around 2,800 words — about the future of what he calls immersive computing. In one sense, it’s a pep talk, reflecting Bavor’s continued enthusiasm for VR. Indeed, he tells me that Google is still investing heavily in the field, with “many hundreds of people” in his group working hard on virtual and augmented realities. But Bavor is also making a plea for patience.
It couldn’t be more well timed. We’re at a moment when the mixed-reality movement, after a few years of steady hype and billions of dollars invested, is now expected to deliver compelling results — even though the technology is not ready to do so, at least not at the level for which people are clamoring. Bavor put it gently to me, saying that the VR movement is “getting ahead of its skis.” Which is, of course, a circumstance that inevitably results in a snow crash.
And it isn’t the first time.
In the early 1990s, even though hardly anyone had experienced it, virtual reality was a thing. A huge one. A thick file of articles and manifestos declared it to be The Future (including my own 1990 contribution to Rolling Stone). People wrote books about it. Even more significant, companies began trying to develop it, most notably the 3D-graphics firm AutoDesk. It was supposed to be the next revolution.
And then, suddenly, the headsets went dark. VR was too expensive. The computers were way too slow. The latency gave you vertigo and could even make you barf. So the articles stopped and the investment money dried up. We entered a long VR Winter during which people pretty much forgot about the whole thing. The internet boom, followed by mobile phones and social networking, gave people plenty of other things to think about.
But in 2010, when Palmer Luckey hacked together a headset that showed off how powerful computers had gotten in the interim — powerful enough to address some of the problems of the original VR bubble — investors and insiders began to rediscover the thrills of virtual worlds. (One big fan: Clay Bavor.) And in 2014, when Facebook purchased Luckey’s Oculus for an astonishing $2 billion, the boom commenced.
All the major tech companies are participating in this second wave of VR, and each of them is pursuing it to forward its respective mission. Facebook sees it as the next social platform. Microsoft focuses on business productivity. Though Apple has yet to acknowledge it, conventional wisdom is that the company has a huge contingent of people working on some form of augmented reality. If so, whatever Apple plans to do with it, we will hear the company boast that immersive computing has finally been done right.
Under Bavor’s leadership, Google has adopted its own “VR for everyone” ethic. Cardboard was introduced in 2014 as a brilliant hack, lowering the price of entering VR so dramatically that the New York Times could afford to include a unit with the delivery of the Sunday paper. Maybe the smartest thing about it was that it looked cheap, and it didn’t strap onto your head. You held it to your face, like a pair of binoculars, and then set it down. It sent the message that VR was a fun thing to experiment with, but not something you had to plan your future around. “It had to be almost free, because there wasn’t much to do it with it,” says Bavor.
To Bavor, the populist approach is all part of a deliberate process — and a necessary one, because even the higher level of VR that comes from more immersive headsets has not yet reached the price-performance standard that would satisfy many millions of consumers. “Could we have built the mother of all headsets that cost $2,000 and was amazing?” he asks. “Of course. We have those in the lab, but it didn’t make sense to try and push the product.”
That’s because we haven’t actually invented a lot of the technology that will help VR fulfill its promise of becoming the mainstream technology for entertainment, communication, and computing. In Bavor’s blog post, he says it straight out: “Everything needs to get better.” By his description, the current generation of VR is a hacked-together solution built out of components repurposed from smartphones. “It’s like we’re building airplanes from bicycle and car parts,” he writes. Both Google and Facebook, as well as other companies, are actively engaging in the research that will yield better displays, dedicated chips, and a deeper understanding of how to engage the brain, so that digital creations seem as real as physical ones, through devices that we can wear as casually as eyeglasses. But it’s a process that will take years. Probably a decade.
In the meantime, there’s been something of a VR letdown as of late. The numbers of actual VR units sold are unimpressive. Though Facebook’s demos of future work in social VR have been amazing, CEO Mark Zuckerberg has expressed disappointment with Oculus’s performance, and has replaced the company’s management. (Zuckerberg has a 10-year view on this, and is also asking for patience.) Hollywood studios have embraced the idea of VR, but movie makers are still trying to crack the problem of storytelling in the new medium. One of the most exciting companies in immersive computing — Magic Leap, the secretive augmented-reality firm funded in part by a nine-figure Google investment — has reportedly suffered delays in heading to market. And though we’ve seen a lot of exciting experiences in games and VR software, there has yet to be a truly killer app or breakaway hit.
Yes, it’s magical to put on a headset and get your stomach fluttering from the view from a high ledge. But the life-changing stuff is still far away. And that’s OK. Bavor compares the development to that of the smartphone, where the first devices appeared almost 30 years ago with the first portable phone. It took 20 years from there to get to the iPhone, which a decade ago launched us on our smartphone-addicted course. (Of course, when that first brick-size mobile phone showed up, it was treated like an odd and expensive curiosity, and no one was predicting it would transform our workplaces and culture.)
To get to ubiquity, Bavor says, Google and others need to reduce the friction of setup and using VR, improve the core technology, and broaden the set of experiences that developers can create. Certainly the standalone headset will reduce the hassle of using your phone as a VR device. “You don’t need another phone, you don’t need a PC, you don’t need to plug it into anything,” he told me in an early preview. “Everything that you need for VR is contained in the headset itself.” Because the new device doesn’t have to strap a phone to the visor, the weight is distributed more evenly, and it feels more comfortable. To be blunt about it, it’s a pain to repurpose your phone as the VR engine for an uncomfortable headset. The standalone device puts you into VR with the touch of a button.
When I tested it out at Google HQ — in a clunkier version than the upcoming commercial units — I found it a much smoother experience than the Daydream viewer or the Oculus-powered Samsung Gear. The secret, says Bavor, is what Google calls “WorldSense”—a means to track head movements with low latency that not only allows flexibility in what you see, but also lets you move around in a virtual scene.
Google has also done some interesting graphics work to make up for not having a powerful computer driving the simulations. Its new Seurat technology uses a computational sleight of hand that dramatically cuts the rendering time of certain objects from hours to milliseconds. Indeed, when I tested this out earlier this week, the Rogue One scene I was plunged into seemed as dense as a high-end VR experience. No wonder: Google’s engineers told me that the original hi-res scene, as rendered by Lucasfilm, consisted of 50 million triangles, an order of magnitude more than even a powerful PC can handle. Seurat’s optimization tricks faithfully create the scene with only 70 thousand triangles, so it can easily run on the mobile Daydream platform in its full 3D glory.
Google is also bullish on augmented reality. Bavor totally rejects the idea that the two technologies are rivals, instead arguing that they are two varieties of a single movement he calls immersive computing. He’s even got a shorthand way to describe the difference: “VR can put you anywhere; AR can bring anything to you.” But basically the tools are similar — in both cases, the idea is to convince your brain that stuff is there when it’s not physically present.
“VR is AR with the lights turned off,” says Johnny Lee, who is engineering director for Tango, the technology that that precisely triangulates where your device is. Google calls this a “visual positioning service,” which is kind of a GPS under rooftops. VPS maps indoor locations and identifies the location of a mobile device within a mapped place, in real time. Where GPS can give you directions to a museum, VPS can give you the precise location of a single artifact in some obscure corner of a gallery.
Lee, who formerly was involved with Microsoft’s Kinect, has long been obsessed with spatially aware computing. “One day, phones, wearable devices, and drones will all understand physical space,” he says. (That darned “one day!”) Combined with a dynamic indoor mapping system, Tango is able to augment the view from your camera to embellish your environment with virtual labels, notes, and other annotations that you can view through your camera. (By the way, the WorldSense tracking technology in the standalone headset is powered by Tango’s mapping prowess.) Google has pre-mapped a number of Lowe’s stores, where employees (and soon, you?) will be able to specify a specific screw-head and be led by virtual pointers to the exact row, aisle, and bin — artificially labeled —to locate it. Tango will also reportedly give you a hell of a tour of the Detroit Institute of Arts Museum. But because Tango is currently only implemented in one Lenovo phone, with a second (the Asus ZenFone AR) to be added this summer, it still is more of a demo than an everyday technology.
I also liked Expeditions AR; while the VR version of Expeditions uses Cardboard for a group of students to share a teacher-controlled VR field trip, the AR variation puts phones on a selfie stick so that kids can examine virtual objects — the planets, a strand of DNA, or the Roman Colosseum — right in the classroom. But like most of the technology today — either VR or AR—testing it just whetted my appetite for the main event. In this case, what if the virtual objects were actual people dropped into the classroom — either visiting lecturers, students out sick, or Yo-Yo Ma giving a cello lesson?
That’s the story with today’s immersive computing: As impressive as it is, we can’t help but compare it with what it’s promised to become. It’s cool to have a standalone headset. But that’s not exactly delivering alternate worlds from contact lenses.
Bavor does not have to look far to see how wrong things can go when hype over-promises what a product can do. I’m talking, of course, about Google Glass, the company’s misbegotten wearable tech experiment. “The vision was right,” Bavor says, “but there was a mismatch between where you could see it going and the current technology — and maybe [it] was trying to do a bit too much,” He also thinks that Glass worked for so long in stealth that the company was blinded to the backlash once people got their foreheads on the product. “One of my tenets in leading this team and in VR is learning by doing and building, and by having real people use real products,” he says.
In the meantime, Bavor is asking for patience. “I wanted to make sure people knew that it would take some time,” he says. “This isn’t going to happen overnight, and it’s important that people understand the rate at which this unfolds.”
Bavor feels certain that the payoff will be worth it. For a few years now, Bavor has been talking about what he calls “the dark cave of possibilities.” It sounds kind of dystopian. But he means it in the sense of one of those old adventure games where surprises lurk down every corridor. As we make the technology advances that make fully immersive computing more powerful and accessible, Bavor imagines applications that will be as important to us as the ones on our phones are today. These will allow Google, he believes, to transform the way it delivers the information that is at the center of the company’s mission. People won’t only access the tsunami of information that Google provides — they will also experience it, whether they are using Google to work, learn, or sample a version of Street View (which might become Street Visit). What’s more, he and his colleagues still believe that this stuff will become everyone’s digital reality, just as the PC, then the graphical interface, and now the phone did. “It will be the way we perceive the world, move through the world, and interact with the world,” says Jon Wiley, Google’s director of immersive design.
Bavor rejects the idea that dashed near-term expectations could lead to a second VR winter. “In the ’90s there was enthusiasm and energy about VR, but I don’t think anyone saw a VR experience they would call completely transporting,” says Bavor. “Today, we’re crossed that threshold. The energy is [similar to the ’90s], but it’s a fundamentally different level of technology.”
The hype, of course, is just the same.