[This is an important first-person report from Medium by an experienced presence-creator about the dangers of moving beyond the “uncanny valley” to the point where we lose the ability to suspend the belief that our experience is real (i.e., not created/mediated by technology). See the original version for a larger version of the figure below, 7 more images and 2 videos. –Matthew]
There is a second valley past the Uncanny Valley or How developing an AR horror game put me in the hospital.
Bryan Mitchell
February 12, 2019
Background: I have been developing mobile games for 10 years. I have had 5 games in the top 5. Over 20 million downloads. Over $1 million in revenue.
This hypothesis began in 2013, when I was watching the film Gravity in 3D at the cinema. There was a moment when shards from an exploding satellite flew at my face and I jumped in my seat. For a split second, part of my brain/body thought I was going to die. It wasn’t exactly a jump scare. The events unfolded in front of my eyes without too much surprise. It was that, for a moment, the experience had the quality of being too real to be enjoyable. Intense engagement in the film and a concern for the characters on screen shifted to concern about myself.
I thought about that experience for some time. I considered the idea that entertainment could, at one point, become so real that one could lose their ability to suspend belief. But the idea remained shelved until 2015 when I began working on an augmented reality survival horror game for mobile devices. A six month project I thought. The goal was to put people inside the movie Paranormal Activity. How hard could that be? My obsession with making that happen turned into a 3 year long nightmare.
Early on in pre-production, it became apparent that there existed a relationship between the understanding of the player, control of their environment and our ability to control and manipulate that player’s construct of reality. It was for this reason that we decided to design the game with the following constraints:
1. It could only be played at night.
2. It could only be played inside.
3. It could only be played with all of the lights turned off.
4. It could only be played with headphones.
The entire experience was lit only with the phone’s LED. Because we controlled the lighting in the player’s reality, the lighting of augmented elements could be matched perfectly. Instead of using 3D models, I photographed actors and composited them into the camera feed. They looked real because they were real. They matched the lighting and lens characteristics because they were photographed under identical conditions with identical hardware.
Because all of the lighting was coming from the LED next to the lens, I was able to model the light as a point source and use the intensity of pixels captured by the camera to get a general approximation of depth (using the inverse squared law). I spent the better part of a year developing a single camera depth estimation system so that the elements in the game could “walk around” in the environment exactly as they would if they were real. It wasn’t enough that augmented elements looked real. I wanted them to behave like they were real. I wanted them to sound like they were real.
I wanted them to be real.
The microphone on the device was routed to the player’s headphones. Every step they took, every natural creak or noise they made, was relayed into their headphones. Without even adding anything, the experience was unsettling.
I wanted every sound to sound as if it was coming from a real location in space. So I searched and searched for a good binaural audio engine but I could not find one. So I built my own audio engine. The reverb was adjusted based on the room characteristics gathered from the depth estimator. I adjusted how augmented elements sounded based on where they were, where you were, where the walls were. I tied the engine into the accelerometer and gyroscope so that the audio buffers had extremely subtle audio distortions based on the movement of the device. I did everything I could to ground the experience in the player’s actual reality.
I obsessed over the details.
I spent months on single effects. Distant lightning caused the LED to flicker. The player’s environment, being lit only by the LED, actually changed. The flicker was augmented on screen to account for the camera artifacts that would occur, such that only portions of the screen were brightened. The audio feed from the microphone popped from the interference caused by the electromagnetic pulse. Shadows were cast. The binaural audio representing thunder was delayed and layered. It didn’t just play in the headphones. It sounded as though it was both outside your headphones, and coming in through your microphone on a slight delay. The goal was to trick the brain such that no suspension of disbelief was required to accept the augmented reality as reality.
I obsessed over the details.
One criticism I received (while marketing this during pre-production) was that I was attempting to achieve full immersion on a tiny mobile screen. But the thing was, you weren’t supposed to forget that you were holding a phone. That too was a part of the experience. You were supposed to believe that you were inside your house. You were supposed to believe that you were filming strange events with your phone. The reality I was creating, extended well beyond the screen. Outside of the supernatural nature of the story, nothing about the experience required a suspension of disbelief.
And I obsessed over the details.
Giving the phone access to your contacts allowed it to send you really twisted text messages and calls from people you knew. Giving the phone access to your facebook allowed the game to simulate the writing out of suicide notes that were posted to your wall. Granting full Facebook access allowed the game to build a profile of the player with the intention of building up a model of what could really scare them. The theory behind the game was beyond fucked up. And I shied away from implementing a lot of really scary ideas.
And a lot of this didn’t make it to the final release for various reasons. Some should be obvious. But everything, from A-Z was designed around the player’s world, around the player’s ego/identity. All of it with the intention of erasing the line between the the game’s reality and the player’s reality. All of it having the effect of preventing the player from having the ability to suspend their belief. Outside of turning the game off, we had absolute control over the player’s reality. Hundreds of hours were sunk into the design of this monstrosity. Thousands of hours were sunk into the development. Thousands more into testing.
And then I got sick…
What the public doesn’t know (until now) is that during the last year of development, I became very sick. So sick that I couldn’t play test it any more. So sick that I could no longer work on the game without throwing up. My chest hurt. I had panic attacks all the time. And at the time, I couldn’t pin point a reason. I had worked on a handful of projects before. All of them had their stressers. But I had never in my life experienced something like this.
It could have been the hours I was obsessively putting into the project. It could have been from the stress of constantly fundraising while doing twenty different jobs. It could have been the product of the complete breakdown of the relationship between myself, my business partner, and our marketing officer. It could have been the sleep deprivation. It could have been so many things.
But I think it was the game.
AR and VR are a completely different class of media. You’re not watching someone else live out an experience. You’re not controlling a character living out an experience. In AR and VR, you’re the one living out the experience.
No matter how many times I played my game, because of the way it was designed it would still get me to jump. If I did not anticipate the scares I knew were coming, if I lost my focus even slightly, no matter how many times I played the game, part of my brain was reacting to the events in the game as if they were real. I would jump. Every time. And I play-tested this game for hours every night. For months and months. I grew anxious. A pain grew and grew in my chest and stayed there. But I kept working when I should have stopped. I was pushed by my business partner. I was pushed by myself. I was pushed way too far.
And then I had an emotional breakdown…
One day I broke down crying and started punching walls. I was persuaded by my wife to go to the hospital and they involuntarily committed me for the night.
I had to bring on another developer to help me finish the game. I’d get too sick every time I worked on it. I’d get sicker when I played it. We eventually released a game that we were all happy with. And I felt that I had mostly achieved my goal of making a game so real and so scary that it would change the industry. Unfortunately, we quickly discovered, that I had only succeeded in making a game so scary that nobody wanted to play it. A game so scary, that perhaps, nobody should have played it. Perhaps, given enough time, they would have ended up like me.
The red flags were there. During the course of development, I could not get my wife to play the game. I could not get 99% of my friends to play the game. I had set out to craft an experience that would be “so real you were afraid to play it,” and I had completely failed to consider what achieving that goal would actually mean. It would be terrible financially.
Although the game’s trailers received 100 million views online the game failed to recoup its budget. I lost a lot of money. My relationships were strained to the point that my wife and I almost divorced. I was a complete mess emotionally and mentally. My business partner didn’t understand it. From his perspective, I had fucked everything up. I was pushed out of the company, and I later sold my shares to my business partner in an acquisition deal that handed Oren Peli’s studio the IP (creator of Paranormal Activity). They created a sequel. And I no longer have anything to do with the project (which has skewed far from my original vision and has pretty much abandoned any pretense of a fully immersive experience).
Ultimately, the game was a success and a failure in all the wrong ways. And when I fail, (and I failed hard given the financial, emotional, and relationship costs I incurred on this project), I try to find the things I can take away from the experience. And that brought me back to the idea I began to form when I first watched Gravity in theaters. It brought me to writing this post about the dangers of the Canny Valley.
Let me assure you that this is NOT an advertisement. The game is no longer for sale. The sequel is awful in my opinion. And I have no plans of mentioning the title of the game here or promoting it in any way. I got sick and that’s not a selling point. This is not a “holy shit, I need to play this game” marketing gimmick. This is a warning. Because I got sick…
And I really think it might have been the game.
I see some small evidence for the Canny Valley in VR games that feature holes you can fall into and ledges you can fall off. Even though you know the floor is there in front of you, even though the better part of you knows what the reality is, part of your brain is telling you that you’re going to die if you take a step forward. Part of you is incapable of suspending that belief. And I really worry about the implications of that. I really worry about the implications of an industry headed in the direction of creating experiences that are indiscernible from reality. Because the Canny Valley exists. It exists in the transition between suspension of disbelief and the inability to suspend belief that is not treated as seriously as reality.
I worry about hostile actors/developers. I worry about incompetent developers. I worry that consumers aren’t taking the dangers of this medium seriously. AR and VR are extremely powerful mediums, and they’re here. Right now. Because so many gimmicky games are being created, because so many shitty experiences are out there, everyone is underestimating the dangers that really exist. We’re talking about the dangers of tripping and falling down. The dangers of punching a wall with the controller. But nobody is talking about the effects this medium can have on the brain. And I wanted to start that conversation here.
In AR and VR, it’s your experience and part of you is experiencing the simulation as reality. You need to be careful about which realities you choose to experience.