[This piece from Harvard Magazine summarizes some of the observations from an interview with Harvard psychology professor Steven Pinker on the topics of virtual reality and presence. See the original version (or YouTube), for the 14:12 minute video interview. A related video, “Harvard Professor Steven Pinker on Apple Vision Pro,“ is also on YouTube. –Matthew]

[Image: Steven Pinker, Johnstone Family professor of psychology at Harvard University, wearing a Meta Quest headset outside William James Hall, home to Harvard’s psychology, sociology, and social studies departments. Credit: Jason Nemirow, courtesy of Steven Pinker]

The Psychology of Virtual Reality

Harvard’s Steven Pinker on the psychology underlying the experience—and limitations—of VR

by Olivia Farrar
January 23, 2025

Virtual reality (VR) technology represents the convergence of several sets of major forces in modern postindustrial life: big tech, novelty consumer demand, and interactive digital spaces. VR headsets promise immersive experiences in gaming, work, exercise, and entertainment, but their implications extend beyond these applications. Johnstone Family professor of psychology Steven Pinker explored the psychological and social dimensions of VR after experimenting with a Meta Quest device.

At the heart of VR’s allure is its ability to create “presence,” the sensation of being physically situated in a virtual environment. Pinker, an expert in visual cognition, likens VR to stereo photography, which uses offset images to produce a three-dimensional effect. VR builds on this principle, engaging stereo vision, peripheral stimulation, and real-time motion tracking to mimic the world. The result is a vivid immersion that far surpasses the experience of viewing a laptop monitor or movie screen.

The Meta device can also replicate the property of peripheral vision, which plays a crucial role in this realism. “There’s an awful lot of information given in the contours of vision,” Pinker explains, which is an “enormously important cue to spatial orientation [and] balance.” Attempting to stand on one leg with eyes closed proves the point—yogis aside, balance is usually lost quite quickly. “Even with the inner ear, the vestibular system,” Pinker continues, “the visual system gives a huge amount of information about the body’s orientation in space.”

When a VR headset is worn, the virtual world stays still despite external movement. Simultaneously, parts of the wearer’s body move with them, because the computer updates the user’s view of the world depending on head position. This creates an illusion of peripheral vision—establishing a lifelike digital quality.

The Limits of Virtual Reality and VR Headsets: Eye Contact and Social Connection

Certain key qualities of vision—including eye contact—are lost when wearing VR headsets. Eye contact, Pinker explains, generates “common knowledge,” a shared understanding vital for social coordination and trust. In 2014, Pinker co-authored a study with graduate student Kyle Thomas, Ph.D. ’15, examining the puzzle of altruism and the psychology of common knowledge. This is the principle of “You know that I know, you know that I know that you know”…ad infinitum—or, the recursive belief state in which A knows X, B knows X, A knows that B knows X, B knows that A knows X, and so on. This social property, Pinker explains, is “[N]ecessary for coordination, for agreeing to rendezvous, for agreeing on a tech standard, for agreeing to use paper currency, for agreeing to drive on the right—but also, to agree on social relationships. “Are we friends? Are we lovers? Are we boss and subordinate, supervisor and supervising, or transactional partners?” Pinker illustrates. “These statuses and relationships are ratified by common knowledge, by two people knowing something, and knowing that the other knows that they know that the other knows it.”

One of the easiest ways to generate common knowledge is not necessarily by thinking (“I know that she knows differently”), but by establishing eye contact. “When I look into your eyes, I’m looking at the part of you that’s looking at the part of me that’s looking at the part of you—and that instantly generates this set of recursive knowledge states,” says Pinker. These knowledge states are impossible to replicate when wearing VR headsets, even when interacting with the most sophisticated avatars, which underscores the limitations of VR in replacing real relationships. Pinker gives the hypothetical image of a couple sitting together, each wearing VR headsets and conversing with simulated versions of each other. While technically feasible, this scenario feels emotionally hollow and devoid of intimacy.

On the other hand, acting as a different user in virtual reality may have numerous positive use cases—including in healthcare, education, military training, and in mental health treatment contexts. Imagine, for example, playing your favorite character in a highly realistic digital rendering of a fictional world, or acting as a different version of yourself with a setting you’ve always wanted to visit (Pinker related the story of ‘visiting’ Maasai Mara National Park in Kenya using the headset). These same principles can instead be applied to immersive virtual reality surgical simulation, in gamifying educational lessons or studying history in simulated historical words, in providing safe and realistic training environments for soldiers in digital military environments, or for therapeutic treatment (cognitive behavioral therapy for patients with anxiety, phobias, drug and alcohol use, eating disorders, and PTSD—all in highly controlled, safe simulated settings).

These examples are all made stronger by the “Proteus effect,” which describes people’s tendency to conform behaviorally to the qualities associated with their projected physical appearance (i.e. gender, race, physical attractiveness). The ability to embody an extremely appealing avatar might encourage increased confidence or improved performance. Pinker suggests that by inhabiting digital personas of different genders, ethnicities, or body types, individuals can gain fresh perspectives on others’ experiences.

[Note: The following points are not covered in the video interview. –Matthew]

What Ethical Concerns Underlie Virtual Reality (VR) Headsets?

Accompanying the promises of VR, with the headset market alone projected to become an over $100 billion market by 2030, are serious ethical challenges—particularly regarding data privacy. VR devices collect vast amounts of user data, including eye movements and emotional responses, to refine engagement and drive consumption. This level of surveillance raises concerns about autonomy and the commodification of human experience. Recent decisions by Meta, such as relaxing content moderation protocols across its platforms, highlight the tension between technological innovation and ethical responsibility.

Latanya Sweeney, director of the Data Privacy Lab at Harvard, has explored the societal implications of emerging technologies (see the feature article “When Technology and Society Clash,” November-December 2024). Her work underscores the need for robust regulations to address privacy and surveillance concerns. The impact of new tech on surveillance and privacy is yet to be fully defined and regulated; and, as VR technology advances, these ethical considerations will become increasingly urgent.