Beyond motion capture: FX wizards may have solved VR films’ biggest problem

[Apparently the little-known company Uncorporeal has figured out how to jump across the ‘uncanny valley’ in VR to produce more effective presence experiences. This story is from Wired, where it includes another image and two videos. There’s not much there at this writing, but the company’s website is uncorporeal.com –Matthew]

An Uncorporeal 10DVR rendering

[Image: The geometric data gathered by Uncorporeal’s cameras and processing hardware results in a data mesh that can be placed in any game environment built in Unity 3D.]

These FX Wizards May Have Solved VR Films’ Biggest Problem

Peter Rubin
October 7, 2015

Sebastian Marino already has an Academy Award for his special-effects work on James Cameron’s Avatar, but the other evening he became a director, and staged a tiny shoot in his apartment. He and a small crew set up an array of cameras and filmed two actresses in front of a green screen. Only one of them delivered lines. “I imagine you’re feeling a bit like Alice right now,” she said enigmatically, “tumbling down the rabbit hole. Like her, I would suggest that you choose carefully.” She pointed in one direction, then another. And that was it; they took down the green screen, and everyone went home.

But Sebastian Marino wasn’t done. The next step was to turn that tiny shoot into the future.

Overnight, the data he’d captured with his cameras simmered; in the morning, he dropped the resulting file into the Unity 3D videogame engine and placed the actresses in a bare-bones apartment. In one room, a neon sign on one wall read WE WANT WHAT WE SEE, and a lamp against another wall cast its light across the actress’ right side. In the other room, he made the interior light red and darkened everything so you couldn’t see that the other model wasn’t wearing much. After all, when you’re making the future, you don’t want to be, as Marino said later, “overly suggestive.”

The thing is, this wasn’t a videogame. It was a movie—and, even better, quite possibly the first actual VR movie to use human beings. The actress who gave the Alice-down-the-rabbit-hole speech looked nothing like a videogame character who was waving from the other side of the Uncanny Valley; she looked like herself. She was wearing the same clothes, her face moved perfectly, and her motions were fluid and flawless. Yet, she was standing in a room that didn’t, in any real way, exist.

What Marino has managed to do, along the rest of his fledgeling VR company Uncorporeal, is solve in a very real way the cardinal problem that plagues the prospects of virtual filmmaking. And what’s even more astonishing is that he and his two co-founders—whose careers encompass decades of wizardry at Lucasfilm/ILM, WETA, the Euro Space Agency, Google X, Microsoft, and Electronic Arts—have done it in stealth mode. They provided the company’s seed funding themselves, they’ve never shown up at a VR conference, and they’ve used email accounts belonging to a holding company so that no one caught wind of the name Uncorporeal. “The number of people that have seen our stuff,” Marino says, “you can count on your fingers and toes.”

If you hadn’t noticed by now, we are way down the rabbit hole.

Caught In the Frame

As we careen headlong toward 2016 and the long-awaited launch of consumer VR, there’s still one very real problem plaguing the medium. It’s not simulator sickness, or display resolution, or optical solutions; all of those things are effectively solved by now, and while they’re not perfect, they’re unimaginably better than they were even 18 months ago. It’s not input; the three consumer systems coming (Oculus Rift, HTC Vive, and Playstation VR) have demonstrable and elegant control schemes. And it’s not content: from dozens of game titles under development to Netflix apps to virtual museum tours, a robust pipeline ensures that early adopters will have plenty to do.

The problem is movies. Specifically, live-action video capture. Consumer VR relies on external cameras to track your headset in space, and then translate its movements and positions into virtual environments. It’s called positional tracking, and in VR gaming it’s what allows you to crouch behind a desk, or peek around a boulder, or look under a table by making those movements in real life—crouching, leaning, bending. But there’s currently no way to bring positional tracking to video.

Think about it: If you position an array of cameras around a central point to act as the “eye” of the person wearing the headset, then you’re stuck with whatever 360-degree view you get from that vantage point. Of VR’s optimal “six degrees of freedom”—nodding your head, turning your head, tilting your head left/right, and actually moving through space on three axes—VR video is thus far confined to moving your body. In other words, you can look around all you want, but you can’t change your position, no matter how much you crouch and lean. (And with your brain registering physical movement without seeing movement reflected in your field of vision, you might actually feel some discomfort.)

In order to get around the positional-tracking thing, a number of VR “film” companies have opted to make Pixar-like CGI movies—using VR-optimized gaming engines like Unity 3D and Unreal 4 to create what are essentially non-playable games. (This weird hybrid model is exactly why words like “experience” and “storytelling” are supplanting the word “movie” in VR circles.) But while you get positional tracking, and all six degrees of freedom, you lose something else: realism. Trying to create photorealistic humans in a regular non-VR videogame already runs smack into the Uncanny Valley; doing it with the added processing burden of VR—stereo renders, each running at a whopping 90 frames per second—is utterly futile.

And that’s where Uncorporeal comes in. They film actors just like any other VR video company, with an array of cameras that they’ve created themselves based on camera-on-a-chip technology. However, their cameras capture the entire lightfield of the room—like a video version of Lytro’s still cameras that debuted in 2011. That means, in simple terms, that the company has so many light samples that it’s able to process that performance and digitally by creating a geometric model that with sub-millimeter accuracy. “We’ve solved not only for the geometry of the subject,” Marino says, “but then we also capture all of the light for all of those cameras, so we have every single pixel. No matter where you’re looking at the subject from.”

If that sounds like motion-capture, it’s not. In mo-cap, you make a static 3D model of a character, then animate its skeleton and other underlying layers to create the performance; it may look realistic, and it might even be based on an actor wearing a suit covered in skeletal trackers, but it’s still CG. However, Marino says, Uncorporeal’s process is more like “per-pixel motion-capture”; every pixel is motion-tracked in 3D, and so can be digitized flawlessly, but it’s still a human performer giving the human performance.

OK, but here’s the other part: That’s just the actor. Once the camera rig captures a person’s performance, Uncorporeal synthesizes all the data and spits out a proprietary codec, u3d, that contains all the geometric information of the captured performer, as well as the lightfield information—which is really just color information. The u3d-encoded file can then be dropped directly into any environment built with the Unity 3D game engine, using a Unity plug-in that Uncorporeal built. The result? A human being in a virtual environment, and one who, thanks to the game engine, will cast a virtual shadow. The lightfield information means that the integration is seamless, rather than looking like a bad Photoshop job. Marino refers to the product—indeed, to the whole process—as “10DVR”: the usual six degrees of freedom, plus three spatial dimensions and time.

To be clear, Uncorporeal isn’t the only company thinking along these lines. Earlier this year, Lytro raised $50 million to pivot toward VR applications for lightfield cameras, but we haven’t seen anything from them thus far. Another company, OTOY, has shown footage of a lightfield render of an empty room, but we haven’t seen human performance. What Uncorporeal seems to be alone in offering, however, is the marriage of image capture and visual effects. By capturing real people and their real emotions, the process preserves the intimacy and empathy of VR video; by rendering them perfectly and bringing them into an infinitely customizable, positionally tracked virtual space, the process preserves that elusive sense of presence that we can only get in CGI experiences.

Not surprisingly, Hollywood is listening. Marino confirms that Uncorporeal is actively talking to a variety of studios, but not in any high-profile way. “Since we self-funded our seed round, we didn’t need to be out there pandering for attention,” he says. “And given our contacts, we’re able to get to who we need to speak to.” And while they’ve steered clear of the usual VR events and conferences, they’ll be attending the Future of Storytelling conference, which starts today in New York. Marino will be there, armed with a DK2, ready to show people what the true future of storytelling might look like.

This entry was posted in Presence in the News. Bookmark the permalink. Trackbacks are closed, but you can post a comment.

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*
*

  • Find Researchers

    Use the links below to find researchers listed alphabetically by the first letter of their last name.

    A | B | C | D | E | F| G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z