How Facebook is taking mind reading from sci-fi to reality

[There’s plenty of press coverage of events and announcements at this week’s F8 Facebook Developer Conference (see Wired on Facebook Spaces VR, The New York Times on AR, and The Guardian for 8 takeaways from the conference), but less attention has been given to the company’s work on brain-computer interfaces. This story is from The Verge, where it includes more images and a 0:15 minute video. –Matthew]

How Facebook is taking mind reading from sci-fi to reality

Regina Dugan: ‘This isn’t cocktail party talk.’

by Nick Statt
Apr 20, 2017

The rumblings started months ago. Through a series of peculiar job listings and key hires, it became clear Facebook was up to something unlike anything it had ever pursued. Building 8, as the company would name it, was to be a new division under famed technologist Regina Dugan, former director of the government’s Defense Advanced Research Projects Agency (DARPA). Dugan had transitioned to the tech industry in 2012, serving as the head of Google’s experimental ATAP group. Among other things, it was responsible for the promising but now defunct Ara modular smartphone project.

On Wednesday, Facebook took the wraps off Building 8 and had Dugan tell the world know what exactly her fast-growing team has been working on. At the day-2 keynote at the company’s F8 developer conference in San Jose, Dugan announced Facebook’s plans for two ambitious projects: one to develop a system for letting you type with just your thoughts, and another to let you “hear” using vibrations on your skin. This would be done through brain-computer interfaces — devices that can read neural activity and translate it into digital signals, and vice versa.

The objective: to help Facebook take the lead in the burgeoning field of augmented reality, which integrates our online and offline lives using a variety of still yet-to-be-built devices. “The goal of an [augmented reality] system is to have a much more blended physical and digital world,” Dugan told The Verge in an interview. “I break that if I have an input mechanism that is not also blended between my physical and digital world.” In Facebook’s view, the road to AR will be paved with the smartphone camera. But eventually, it leads to the brain — which is where Dugan and her team come in.

Dugan says Facebook’s goal is to develop something it calls a “brain click — a way to complete tasks in augmented reality using your mind. You could brain click to dismiss a notification that popped up on your AR glasses, for example. Researchers at Building 8, who have teamed up with medical institutions around the country, want to turn the brain into an input device, starting with letting people type with their thoughts.

Even “something as simple as yes/no, on/off” would be the “equivalent of having the actual mouse in the advent of the graphical user interface,” Dugan says. AR glasses that receive inputs directly from your brain would be ideal, Dugan says, because that would further reduce the disconnect that technology creates between people and their surroundings.

She feels similarly about Building 8’s second announced project, an effort to broadcast messages to your skin. “It’s the whisper being delivered,” Dugan says. “It wouldn’t break the social contract. If I pick up my phone while we’re talking, I have broken the connection I feel to the people here in this room.” And so Building 8 is pursuing a way to “hear” with your skin, using what amount to a series of vibrations on one’s arm that correspond to words and phrases. Eventually, you may be able to send discrete messages delivered through touch.

The work builds on the advent of Braille and a technique known as the Tadoma method, a communication format developed for deaf and blind individuals that relies on the impaired person feeling the movement of the lips and the vibrations of a speaker’s throat with their hands to make out the spoken words. Facebook wants to build on those techniques, which extract semantic meaning from touch, to create what just might amount to a new form of language. “I think the thing that is so interesting to me is how inherently wired we are to language and communication,” Dugan says. “Our cochlea works in one way. But we can interpret language through Braille, the Tadoma method… we’re just wired for it.”

The company has rigged up a working device that has allowed a Facebook employee to differentiate between three shapes, colors, and actions through just vibrations on her skin. “She learns how the word black feels and how the word blue feels different,” Dugan says. “You could imagine over time you could begin to understand how the components of the words feel.” Facebook imagines this as both a method of discreet communication for healthy consumers and a way to restore communication to people with disabilities.

They are bold goals, but Building 8 has set a concrete path for achieving them. Dugan became famous at Google’s ATAP unit by implementing an aggressive two-year deadline for each project it pursued. Dugan adapted this method from DARPA, where it was used to great effect. It was DARPA that laid the foundation for the early internet, in the late 1960s, and has been responsible for early versions of everything from GPS to Google Street View to Wikipedia.

At Google, the strategy bore less fruit. The Ara project ran into repeated roadblocks and was eventually shuttered, but Dugan has reportedly altered her approach at Building 8. According to The Information, Dugan is working more closely with product specialists at Facebook to ensure projects can exit the prototype phase and make it to market.

Building 8’s first two-year goal is to improve the rate at which people can type with their thoughts to 100 words per minute using implanted electrodes. At Stanford University, implanted electrode arrays inside paralyzed patients allow them to type with their thoughts at up to eight words per minute. To increase typing speeds, Dugan and her team are monitoring medical trials and working with partner institutions including UC San Francisco, UC Berkeley, and Johns Hopkins Medicine.

The second goal of Building 8 over the next two years is to determine whether typed thoughts can be read non-invasively — which is to say, without having to implant a device in the brain. That would lower the cost of the device dramatically, and make it available (and desirable) to a much wider group of people.

“Implanted electrodes simply won’t scale,” Dugan said on stage Wednesday. Instead, the company is exploring how optical imaging could get real-time data from the brain and translate it into words. The resulting device could be something like a neural cap worn on the head, or some type of band that stretches around the back of the skull.

Of course, it’s possible to hear about Facebook’s plans and worry the company is planning to invade your privacy in new, more dystopian ways — literally getting inside your head and under your skin. But Mark Chevillet, Facebook’s technical lead on the thought typing project, says the technology should be designed to operate only during the final part of the speech process, right before your brain tells your mouth to start moving. The thought is already formed, and you have made an explicit choice to share it.

“If you’ve dictated to Siri, imagine the same thing but without you speaking outside,” he says. “You would be speaking internally — you’re just not actually making the sounds.” Dugan was quick to point out onstage at F8 that the company is not interested in reading your thoughts. “That might be more than any of us care to know,” Dugan said. “And it’s not something any of us should have a right to know.”

Whatever their plans, Dugan and Building 8 have a significant amount of scientific challenges to overcome. Other companies, like Elon Musk’s Neuralink and Bryan Johnson’s Kernel, are also pursuing brain-computer interfaces, though with more grandiose visions of turning us into smarter, faster cyborgs. Though we know little about Musk’s efforts, Kernel has more than $100 million from Johnson, a PayPal co-founder, to advance neuroscience research.

Dugan is unfazed by competing efforts. “You can talk about it in general terms, or you can get down to business,” she says. “This isn’t cocktail party talk. This is serious.” It’s one thing to lay out a grand vision, she says, and another to build a process capable of achieving it. Building 8 has a concrete plan, Dugan says, and it is already attracting some of the best minds in the industry.

“This is a style of work I know,” she says. “If you want to look at 60 years of history and success, that’s the model you use. That’s what we do.”

This entry was posted in Presence in the News. Bookmark the permalink. Trackbacks are closed, but you can post a comment.

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*
*

  • Find Researchers

    Use the links below to find researchers listed alphabetically by the first letter of their last name.

    A | B | C | D | E | F| G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z