[This story from The New York Times provides an interesting look at the development and early-days potential of Google’s Tilt Brush technology and virtual art generally; see the original for more images and two videos. –Matthew]
[Image: Roz Chast’s cartoon made with Tilt Brush. Credit: Google]
The Making of Virtually Real Art With Google’s Tilt Brush
By Frank Rose
January 4, 2017
SAN FRANCISCO — In 1949, a Life magazine photographer named Gjon Mili made a pilgrimage to the French Riviera to see Pablo Picasso. Mili had come up with a way to photograph trails of light, and he wanted to shoot Picasso “drawing” in midair with a light pen — a process that would leave no trace except on film. Picasso loved it. The result, published in Life and exhibited at the Museum of Modern Art, was Picasso’s celebrated series of “light drawings” of bulls and centaurs and the like — photographs that captured him in the act of creating the ultimate in ephemeral art.
Picasso is long gone. But some 68 years later, Google has been calling on dozens of artists, animators and illustrators with a high-tech update of Mili’s concept — a virtual reality setup that enables people to paint with light that actually stays where you put it, at least for viewers wearing a VR headset. In place of Gjon Mili are Drew Skillman and Patrick Hackett, a pair of video game developers turned virtual reality enthusiasts who live in San Francisco.
They were trying to build a 3-D chess application one night a couple of years ago when they discovered it had an unexpected side effect: As you moved the chess pieces around in virtual space, they left trails of light behind. Sensing that their bug was in fact a feature, the two dropped the chess project immediately and hurled themselves at the light trails, hoping to develop a tool for drawing in three dimensions.
In April 2015, seven months after they had cobbled together a rudimentary system they called Tilt Brush, Google bought their company for an undisclosed sum — which is how Mr. Skillman and Mr. Hackett have come to be ensconced in the company’s offices near the downtown waterfront here. With Google’s support, Tilt Brush has attracted a team of developers and evolved into a sophisticated tool for drawing, painting, even sculpting in space. It was released in April as a free add-on to the new HTC Vive, an $800 virtual reality system produced by the Taiwanese manufacturer HTC in partnership with Valve, an American video game developer. (It’s on sale as a $30 software package from Valve’s online store.) Reviewers immediately dubbed it the Vive’s killer app.
This is hardly the kind of reception the two inventors were expecting when they started working on it in Mr. Skillman’s apartment, a 400-square-foot studio in South Park, the little neighborhood that has been a hub for San Francisco’s digerati since the 1990s. “Not in our wildest dreams,” said Mr. Skillman, a slightly built 36-year-old with a neatly trimmed beard and a modest, unassuming manner.
Tilt Brush got its name because in its earliest versions, you would draw or paint on a two-dimensional surface that could be tilted in any direction in virtual space. But because the HTC Vive includes not just a virtual reality headset but also a pair of hand-held controllers and two tracking sensors that map your movement in space, the program was revamped to enable you to paint or draw anywhere within a room-sized area — no surface required.
One controller serves as a palette, with dozens of colors and effects; the other acts as a brush or pen. To watch someone use it is a bit unnerving, since the person appears to be making marks in midair, but you can’t actually see those. But put on a Vive headset and step between the sensors yourself — as I did in a windowless room in Google’s New York offices — and the illusion of delusion disappears. Instead, you suddenly see what has been produced: a phantom creation in three dimensions, something you can walk around, walk through, poke your head inside, do everything except touch.
Google’s investment in virtual reality pales beside Facebook’s $2 billion purchase of the VR pioneer Oculus. Still, Tilt Brush is part of a growing effort, one that began with the introduction in 2014 of supercheap cardboard headsets that work with smartphones, and continues with the recent release of a stunning virtual reality version of Google Earth as a free download for Vive owners.
Over the past year, Google has invited more than 60 people to try Tilt Brush and offer feedback, and this week the company is unveiling their work and participation. “You would never want to create an artistic tool with only engineers,” said Mr. Skillman. “That’s just absurd.” According to Tory Voight, Google’s Tilt Brush program manager, those who have joined this artists-in-residence program include Dustin Yellin, a Brooklyn artist known for his 3-D collages encased in layers of glass, and Jonathan Yeo, a British painter whose portrait of Kevin Spacey as the fictional American president Francis J. Underwood was exhibited last year at the Smithsonian Institution’s National Portrait Gallery.
Jeff Koons, whose art works have sold at auction for as much as $58 million, had early access, though he was not part of the artists in residence program. Bob Mankoff, Roz Chast and other cartoonists for The New Yorker have also checked it out, as has Alex Hirsch, the young animator behind Disney’s hit television series “Gravity Falls.” Last June, Mr. Hirsch posted a sample creation on Twitter along with an enthusiastic status update: “Drawing in virtual reality makes you feel like a world-devouring wizard god!”
But the first to try it was Glen Keane, a near-legendary figure who in his 37 years at Disney had brought the warmth of hand-drawn animation to such characters as Ariel (in “The Little Mermaid”), Aladdin, Tarzan and Pocahontas.
“When I left Disney in 2012,” Mr. Keane said recently in his studio, a vintage bungalow on a quiet residential street in West Hollywood, Calif., “I told them it was because I know there’s something new coming — I don’t know what it is, but I need to leave to find it.”
He was still looking when he met Regina Dugan, then the leader of Google’s secretive Advanced Technology and Products group, which was experimenting with virtual reality. First he partnered with the group to make a hand-drawn virtual reality short called “Duet,” a charming piece released in 2014 about two babies growing up that was shortlisted for an Academy Award. “Duet” broke new ground, since animation in virtual reality had almost exclusively been computer-generated. But though he’d spent a lifetime drawing on paper, Mr. Keane had always dreamed of being able to make the paper disappear. “The goal would be to animate not on paper but in space,” he said.
Enter Tilt Brush, which Mr. Keane encountered when Mr. Skillman introduced himself at a visual effects conference in San Francisco. Though still in a primitive state, it was maturing rapidly, and Mr. Keane soon became a convert. In September 2015, seven months before its release, he previewed its capabilities with “Step Into the Page,” a five-minute video. “The edges of the paper are no longer there,” he exclaimed in a voice-over as he did a loose, freehand sketch of Ariel in virtual space. “This is not a flat drawing. This is sculptural drawing.”
Scott McCloud, the graphic artist whose book “Understanding Comics” is considered the ultimate guide to the art form, got to play with Tilt Brush when Google invited him to its Silicon Valley headquarters in August. “I don’t mind saying, I’m a little bit obsessed with this program,” he said as we sat in his small, book-crammed office in a mall in the Los Angeles exurb of Thousand Oaks. “One thing that appeals to me the most is, it’s still very early. Everyone is asking fundamental questions. We’re still trying to figure out what people are going to use it for. I love it when technology is in that stage.”
So, what are people going to use it for? Mr. McCloud threw out a few suggestions: performance art, virtual sculpture, industrial prototyping. “I doodle with it,” said Mr. Yeo, the painter, who spoke by telephone from London. “I describe it as a three-dimensional sketchbook.” What it won’t produce is anything resembling comics as they currently exist in the medium some call “Flatland.”
For denizens of Flatland who have learned to work within the constraints of 2-D, this can be problematic. Mr. Mankoff, the cartoon editor at The New Yorker, came up with some characteristically Mankoff-like 3-D drawings during the two days he spent using Tilt Brush with Ms. Chast and other colleagues at Google’s New York offices. But though he professes to have enjoyed the experience, he seems unlikely to lead The New Yorker out of Flatland. “You’ll probably find a sharp divide between older and younger cartoonists,” he said. “Or it may just not work — but that’s no reason not to try it.” One suggestion the New Yorker team had: cubes and cylinders and other ready-made shapes to draw on, “so you’re not starting from zero in 3-D space.” (Google has since implemented a similar idea.) Other than that, Mr. Mankoff offered with a grin: “The advice I would have for Google is, more cats. You can’t go wrong with cats.”
Mr. Keane has a different item on his wish list. Currently, works created in Tilt Brush are as motionless as a New Yorker cartoon. They’re essentially static, and Mr. Keane is an animator. “To me, the thing to conquer is to be able to animate in real time in space,” he said as I viewed a Tilt Brush drawing called “Victory Dance” in his studio. The piece captures a brief moment in time, seconds after his daughter’s high school choir had won a national championship — but it begs to be brought to life.
“Just like you have a spatial dimension, you’ve got to have time as a dimension,” Mr. Keane continued. “There’s no reason you can’t do that. I’m not smart enough to figure it out. But Drew is.”