[To virtual and augmented reality we have to add substitutional reality, and as in this story from The Japan Times, consider its far-reaching implications. –Matthew]
[Image: Riken researcher Naotaka Fujii (right) tests a ‘substitutional reality’ system that lets people experience how a prerecorded 360-degree video can be mixed into real-life surroundings. | Courtesy of RIKEN]
Riken mind bender stays one step ahead of virtual reality
By Ayako Mie, Staff Writer
December 4, 2016
Imagine you are standing on the Grand Canyon Skywalk, a horseshoe-shaped bridge suspended 1,200 meters above the Colorado River. You are likely to get dizzy and freeze up at the thought of venturing out onto the 10-cm thick glass.
Nevertheless, you step forward and breathe a sigh of relief after realizing you are not tumbling toward the river.
But how can you be sure what you see is really there? And more fundamentally, how do we know that we live in the real world and not a virtual one?
The boundary between what’s real and virtual is blurring with the rise of virtual and augmented reality.
By wearing Oculus Rift, a type of virtual reality headset, you can transport yourself to the middle of a desert without actually being there. By using Google Glass, you can use the overlaid information in front of your eyeballs to find the nearest coffee shop without opening the maps app on your smartphone.
And now Naotaka Fujii, a researcher at the state-backed Riken institute, is taking VR to an entirely new level: “substitutional reality.”
The CEO of VR company Hacosco led Riken’s Laboratory for Adaptive Intelligence in 2012 to develop a system with the goal of manipulating reality. The so-called SR system can be used to re-create an experience similar to the one presented in the sci-fi movie “Total Recall,” in which a character played by Arnold Schwarzenegger buys memory implants to enjoy a real-life experience on colonized Mars.
Riken’s SR system does not actually implant memories, but the visual and audio effects are so realistic that the brain becomes too confused to distinguish between present and past, or fake and real.
The trick is simple. First, a head-mounted display (HMD) must be put on in order to electronically view what’s actually in front of you. Later, a pre-recorded 360-degree video shot in the same surrounding environment is loaded into the HMD.
The shift is so subtle the user does not notice the difference.
Even though Fujii is a leading figure in VR and SR in Japan, he is not an engineer. Rather, he is a social psychiatrist who researches “social brains,” a study of how the organs behave when interacting with people in different social settings.
Starting out as an ophthalmologist, Fujii began research at the Massachusetts Institute of Technology on how to restrain eye movements.
After returning to Japan in 2004 to work at Riken, he started to take an interest in social brains.
Fujii said he wanted to find out why he felt socially awkward at times. Initially, he used monkeys in his studies. But when he extended that research to include humans, he encountered a problem.
“If I say ‘good morning’ to the person being tested to see how he or she reacts, I have to say it exactly the same way to every person. But I cannot do that as my mood changes,” he said. “I needed to create a device to do that.”
That’s how the SR system emerged.
Fujii said that, in a way, substitutional reality is more real than VR because the inserted images appear to be a continuation of reality.
He said he discovered a shift in how people being tested process what they see. At first, they seem dubious the person in front of them is real. But soon afterward they accept it as real and start talking to the pre-recorded image.
“They have to believe what they see because otherwise they cannot take any action,” he said. “I found it interesting.”
In 2014, Fujii invented an affordable virtual reality device called Hacosco so he could monitor the social impact of the technology.
Priced at only ¥1,000, it substitutes expensive headgear with a cardboard viewer designed to hold one’s smartphone.
By comparison, Sony Corp.’s PlayStation VR hit shelves in Japan in October priced at ¥44,980 ($433), while the Oculus Rift costs around ¥90,000 and the HTC Vive ¥99,800.
Still, if the concepts of virtual and augmented reality are embraced in everyday life, we can expect to see a drastic change, Fujii said.
With the advance of the internet of things and artificial intelligence, Fujii said it might be possible for 360-degree videos to be projected into every corner of the space we live in. Information or images could then pop up on any medium or object available, Fujii predicted.
It’s like turning every flat medium around you — such as a wall or a table — into a display gadget to project any AR or VR program, Fujii explained. Or like having a 3-D image of someone conversing with you with the full understanding of what you like and want.
This shift is already happening through such robots as Amazon Echo and Google Home, even though there are limits to what their AI systems can do.
If AI-based devices or 360-degree images become so sophisticated we cannot distinguish fake from real, Fujii said, they will be adopted by society as a new form of communication.
One might wonder how our senses of taste, smell or touch can function in a virtual world.
Fujii said our brains will fill the gap.
In your dreams, for example, sensations can seem quite real. You can savor that juicy steak. Or feel the wind blowing through your hair.
If these concepts evolve into a “Total Recall” experience, he said, having the virtual steak will be much more affordable than consuming the real one.
The new technology will also present risks, he warned. Will people be willing to come back to the real world if they love their virtual, ideal world too much? What if those technologies are used in crimes? What if the world comes under a Big Brother government that chooses to exploit people by compensating them with rewards that can only be used in a virtual world?
“Because our brains would be tricked, the meaning of what’s right and what’s wrong and what’s true and false would change. Society will change. We might be at a turning point in history,” Fujii said.
No regulations on VR exist yet, nor does Japan have any government agencies assigned to deal with the potential risks. At this point, it is up to the companies that produce the technologies. For instance, Oculus requires that users of its products be 13 years old or older.
That is why Fujii launched the Virtual Reality Consortium of public- and private-sector representatives in 2015 to spur innovation and address issues related to virtual reality.
Fujii said the VR and SR worlds require a high technological literacy to enjoy but offer a sense of unpredictability that is also an attraction.
“I used to think I could die at any time as the world was not interesting. But I want to live and see what’s really going to happen,” Fujii said, “And after all, nobody really knows if we live in the real world.”
Key events in Naotaka Fujii’s life
1991 — Graduates from Tohoku University School of Medicine.
1991-1997 — Resident physician and graduate teaching assistant, department of ophthalmology, School of Medicine, Tohoku University.
1998-2004 — Postdoctoral fellow and senior research scientist, department of brain and cognitive sciences, Massachusetts Institute of Technology.
2004 — Joins Brain Science Institute at Riken.
2012 — Becomes head of new Laboratory for Adaptive Intelligence at Riken.
2014 — Launches Hacosco as CEO.
2015 — Launches Virtual Reality Consortium.
—
“Generational Change” is a series of interviews that appear on the first Monday of each month, profiling people in various fields who are taking a leading role in bringing about changes in society.