[This story from Wired reviews the evolution of NextVR’s coverage of football and basketball and includes implicit and explicit references to presence, discussion of the potential for memory confusions between 3-D VR and in-person game experiences, and the obstacles to widespread adoption of live experiences of sports via VR. See the original story for several more images.—Matthew]
The NFL’s New VR Highlights Take Football to a Whole New Level
Tim Moynihan
12.11.16
Deep in the second quarter of the New York Giants’ win over the Chicago Bears on November 20, Giants quarterback Eli Manning slung a short pass to running back Rashard Jennings, who was forced out of bounds at the 28 after a five-yard gain. Nothing about the play, which lasted all of six seconds, stood out as particularly noteworthy. Except for one thing: Safety Deon Bush and linebacker Danny Trevathan shoved Jennings out of bounds right in front of NextVR’s bug-eyed sideline camera. Watching the play on a VR headset made you instinctively want to get out of the way. The “you-are-there” sensation was strong.
OK, time out. You’re rolling your eyes, with good reason. Video on a VR headset is usually pretty disappointing. The novelty of watching something from a cool virtual vantage point erodes quickly. There’s a “screen door effect” when seeing pixels on a magnified screen, shattering the illusion of being there. The content typically lacks narrative, offering nothing to direct your attention or provide urgency or drama.
But NextVR’s remarkable work with the NFL and the NBA provides a virtual experience that is far more engaging and dramatic. The slick highlights packages they produce for the NFL are released after the games, but NextVR is uniquely equipped to do live VR video. The company uses the same crew, gear, and processes to broadcast one live NBA game each week. They use eight camera rigs, hop between camera angles, and include onscreen graphics and play-by-play announcers.
We aren’t talking 360 video, either. There’s no confusion about where to look, as NextVR’s rigs use a 180-degree field of view to record only the action in front of them. The hardware is more 3-D camera than VR camera, with processing tricks that add depth and boost the sense of realism. Each double-barrel RED rig captures 6K video for each eye. That’s tens of millions more pixels than any headset can display now, but NextVR wants to future-proof its content.
The result represents the dawn of a new kind of VR video, an experience that truly combines the best parts of watching a game on TV and being in the stadium.
Monster Truck
Peek inside NextVR’s production truck and you’ll see a director, line producer, editors, audio technicians, and banks of monitors above racks of expensive gear. “We have a ton of engineers, because the tech side is so heavy,” says coordinating producer Josh Earl. “We have more engineers than actual broadcast people. I’ve been in broadcast for 15 years, and I still don’t understand how everything in here works yet. It’s just different.”
The on-field cameras each use a pair of super wide-angle 8mm lenses to send two fisheye views of the action to the production truck, where the producers watch it on the monitors. You’ll find several Gear VR headsets in the truck, but no one is wearing them during a game. They’re there just so the crew and on-air talent can give the NFL VR experience a spin before blasting it out to the masses later on. “With the NBA, it just goes out live,” Earl says.
The barrier to live NFL coverage goes beyond inking deals. The execution is already as polished as anything you’ll see on TV, but NextVR is still feeling its way through some details of shooting sports. Something as simple as panning is a no-no, because it may make people sick. Zooming is also off-limits, as it chops off the field of view. Both limitations make it tough to capture fast breaks in the NBA and long bombs in the NFL. When switching between cameras, NextVR producers favor a slow fade between angles, often when the action slows down.
The company has optimized its NFL field coverage by mounting cameras under each crossbar, placing manned cameras on the sideline of each end zone, and having four camera operators roam the sidelines. The setup favors action in the red zone over plays between the 30s, where each team has its bench. “The hard part about doing football is that with the 8mm cameras, you need the action to come to you,” Earl says. It’s a bit easier with basketball, where most of the action happens in front of the NextVR camera under each hoop. “Anything that happens, we have a great angle of it,” he says.
That said, NextVR is all but ready to flip the switch on live football. The crew essentially produces complete games already, they just don’t broadcast them. “The reason we do the NFL highlights the way we do it is because we want it to be exactly the same,” says VP of content Danny Keens. “We don’t want there to be any loss of quality, any loss of resolution, any of that stuff. It’d be easier to just piece that show together and record it bit by bit. But we go start to finish and do live graphics and replays in real-time.”
3D + VR = OMFG
VR is only now moving into the mainstream, but NextVR has been at it for two years. What started as a simple one-camera setup at midcourt during an exhibition game evolved into the current multi-camera setup with all the fixins. Earlier this year, NextVR announced it would carry live NBA games each Tuesday as part of NBA League Pass. It marked a huge step forward for virtual reality.
“For the first time ever, we announced a production schedule,” says company CEO Dave Cole. “We had more than 500 hours of live VR production under our belt before the NBA deal, but they were all one-off productions. That’s not the type of thing you can get viewers to schedule. I liken it to going to Best Buy and buying a television and the salesperson saying, ‘Well, there’s probably going to be a broadcaster for this device someday.’”
NextVR’s secret sauce is its 3-D effects, which date to its origin in 2009 developing 3-D television transmission tech. The company’s video-compression technology shrinks files by removing redundancies between each rig’s left camera and right camera. Because that process involves detecting the edges of objects in a scene, the same technology can create wireframe replicas of everything the cameras shoot. When the video is rendered in a VR headset, it overlays stereoscopic video on top of those wireframes, creating the illusion of volume. It’s a mix of video and video-game tech.
“It sends a hugely impactful message to your brain that you’re actually in this environment,” Cole says. “Right now, the mesh, the number of vertices in the wireframe is quite low. In the next generation camera, which is rolling out in the middle of the season for the NBA, we are quadrupling the resolution of that mesh. That sense of presence is what we’re amping up.”
Hacking Your Memory
NextVR outpaces the VR competition with the quality of its tech and scale of its deals, but it’s already eager to refine the experience. “We will have done our job right when people can’t remember whether they actually went to the game or watched it on NextVR,” Cole says. “I think that’s an achievable goal.”
Crazy as it sounds, that actually could be an achievable goal. At least in part. “My answer to whether or not that could be possible is … sort of,” says Julia Shaw, a memory expert, criminal psychologist, and author of The Memory Illusion. Shaw hasn’t studied VR’s effect on memory, but she has successfully implanted false memories into the brains of test subjects. Because our memories are unreliable, convincing VR experiences could fool our brains. But only to an extent.
“Reality is multi-sensory,” she says. “When you’re looking at something, no matter how high-def it is, if you don’t have things like proprioception, your sense of space, you don’t have smells, you don’t have taste, you don’t have temperature. These are things that we generally rely upon as markers to let us know we’ve experienced something instead of just imagined it.”
So if Cole is serious about taking NextVR to that level, he’ll have to figure out how to infuse the experience with a lot more sensory input. Things like smart thermostats synced to the action and stadium-smell simulators.
VR’s Biggest Challenges
NextVR has some more important technical hurdles to clear first, but they likely won’t be barriers for long. You can only watch its programming on the Samsung Gear VR and Google Daydream, but Cole hints you’ll see NextVR on PlayStation VR soon. And NextVR is talking with other sports leagues and entertainment companies about getting more content in the pipeline. Cole says 4k-capable phones will make the viewing experience that much better within the next year, and 5G connectivity will make accessing content easier on mobile devices.
All of this begs the obvious question: Will viewers embrace VR as a “first screen” option for sports? Already, watching a game on NextVR is a better eyeball experience than watching it on TV. But here’s the thing: VR is a solitary pursuit, one that requires clamping a headset on. Watching sports is a social endeavor, one that revolves as much around the camaraderie of the experience as it does the game itself.
So there are compelling reasons to strap on a VR headset to watch a game, but it’ll likely only happen if you’re home alone. Duncan Stewart, director of technology research for Deloitte Canada, says the solitary viewing experience is just one mainstream adoption barrier. According to his research, a few trends are shaking out in these pioneer days of VR: The medium still appeals primarily to males, hard-core gamers, and those with deep pockets. Deloitte’s global surveys show that more than 95 percent of people don’t own a VR device and aren’t interested in buying one.
“There are indeed some people who are interested enough in the VR perspective to watch sports and wear a device on their head that blocks out their wife or husband, kids, parents, friends, pets and smartphone for hours at a time,” Stewart says. “But not many.”
There are other impediments, not the least of which is VR’s illusion of immersion is shattered every time you need to use the bathroom or grab a snack. And then there’s the simple fact that watching a game isn’t as simple as flipping on the TV, popping open a beer, and flopping down on the couch. You’ve gotta hook up the gear, launch the app, and find the content. NextVR’s incredible videos may represent the future of VR, but sports fans may determine the future of NextVR.