[This story from Geek describes a novel presence experience that explores and demonstrates the complex interaction of the visual and aural senses. The story (lightly edited for typos here) includes more images, and the link at the end is to a New York Times story that includes a 12:09 minute video and more information. –Matthew]
Virtual reality sim Notes on Blindness makes you feel like Daredevil
By Charles Battersby
Jun. 2, 2016
Superhero stories and urban legends say that blind people develop enhanced senses when they lose their vision. In the real world, it isn’t quite so simple. The subjective experience of what it’s like to regain awareness of space after losing sight is difficult to convey. When writer and theologian John Hull began to lose his sight, he recorded a series of audio diaries about what he called “a world beyond sight.” These firsthand accounts were compiled and used as the basis of a film called Notes on Blindness. To help convey the experience in a more subjective form, the filmmakers also created a virtual reality experience that uses “binaural sound” and abstract visuals to teach users how a blind person can become cognizant of the space around them.
The feature film uses contemporary interviews with Hull and his wife, interwoven with scenes where actors recreate the scenes described in Hull’s audiotapes from the period 30 years ago when he was losing the last of his sight. In the VR project, Hull’s audio diaries guide the user through a series of chapters exploring how he learned to use sound to discern what objects surrounded him.
According to the film’s writers and directors, Peter Middleton and James Spinney, “Early on in the development process it became apparent to us that only a small fraction of John’s vast audio archive could make it into the feature film. This got us thinking about how some aspects of his account could find fuller creative expression in an interactive or experiential piece. We were particularly interested in developing a project that could use binaural sound. We began working with Arnaud Colinart from Ex Nihilo, and Amaury La Burthe from AudioGaming and their team to edit down diary passages that would be suited to this technology.”
“Binaural sound” means that each headphone in the VR headset plays slightly different sounds, which mimic the nigh-imperceptible differences that a person’s left and right ears hear in the real world. This allows VR users to distinguish which direction virtual sounds are coming from. One chapter in the experience has users sit on a virtual park bench, listening to the area around them; the footsteps of passersby, wind blowing through trees, the rippling of water on a pond. As they listen, Hull explains the significance of each sound when trying to become aware of the objects surrounding the user.
In another scene, the user sits inside a house while rain falls outside. Water leaks through the roof and drips into pans as Hull explains how the sound created by these drops can reveal the space of the room. Users can turn around in their virtual chair and use the binaural sound to explore the room and the objects within.
“We were particularly drawn to passages that describe the development of John’s appreciation of ‘acoustic space’ — the mental mapping of environments through sound,” say Middleton and Spinney. “Four of the six chapters in the VR project specifically concerned with John’s auditory experience of blindness; listening to the footsteps of his children playing in park; how the intricate patterns of falling rain bring distinction to one’s surroundings; the engrossing acoustic experience of choir singing in a cathedral; or how wind and thunder can transform outdoor space.”
The virtual reality experience and the movie both use abstract imagery to represent how sound “looks.” The filmmakers say that “It was initially our intention that the interactive experience be audio-only, but early experiments found that sighted users found it difficult to engage in the material without at least some visual stimulus. So we began thinking about how to visually represent acoustic experience.”
“We were interested to learn that the visual centers of the brain are still highly active in blind individuals in conceiving space, patterns, and motion,” continued Middleton and Spinney. “Acoustic information is interpreted by the visual cortex — so in a sense our artistic challenge was analogous to the ‘visualisation’ of sound that is an unconscious neurological process in John [Hull]’s experience.”
This visualization bears a strong resemblance to how the blind superhero Daredevil’s “radar sense” is depicted in film and TV. This isn’t a coincidence, say Middleton and Spinney. “Daredevil was a visual reference that popped into Arnaud and Amaury’s head when they first saw the rainfall sequence in the short film version of Notes on Blindness — and stayed with them when they developed the art direction of the VR.”
Arnaud Colinart and his team were also inspired by imagery from Hull’s audio diaries, which were recorded decades before Ben Affleck and Charlie Cox put on the Daredevil costume. Middleton and Spinney say it was Hull’s “…recurring references to water and moving particles, in which wind moving through leaves brings trees back into existence, droplets of rain give shape to the world. This imagery seemed fitting because it is never fixed: particles converge and dissipate, suggestive of John’s distinction between visual and acoustic experience. Whereas in the visual plane things remain stable and present, in audible experience, objects come in and out of being through sound, appearing and disappearing as the sounds rise and fall.”
Hull was aware that his experiences might be seen as superhuman. In an interview with VisionAware.org, he said “One change was the experience of what they call ‘echolocation’. It dawned upon me when I was walking down the road and suddenly I sensed there was an obstacle in front of me. It’s quite striking really. This ability mystifies sighted people, doesn’t it? Sighted people think blind people have super powers.”
Notes on Blindness is not a superhero sim, though. Several sequences deal with the panic and anxiety that can come from blindness, as well as Hull’s belief that if he didn’t learn to understand blindness, it would destroy him. VR users who wish to experience it for themselves can do so when the VR app releases on June 30 of this year for Samsung Gear, and in the following weeks on Google Cardboard and iOS. The feature film premiered at Sundance 2016 and will be theatrically released on July 1 in UK and in the US later in the year. The short film is available online now.