AR will spark the next big tech platform – Call it mirrorworld

[This recent Wired story from the magazine’s founding editor is a thoughtful and thought-provoking perspective on the evolution of technology that explains how and why presence illusions will increasingly occur via augmented reality, creating both important benefits and potential costs. The original version of the story is extremely long so it’s abridged here, but see the original for more information and detail. –Matthew]

AR Will Spark the Next Big Tech Platform—Call it Mirrorworld

Kevin Kelly
February 12, 2019

EVERY DECEMBER, ADAM Savage—star of the TV show MythBusters—releases a video reviewing his “favorite things” from the previous year. In 2018, one of his highlights was a set of Magic Leap augmented reality goggles. After duly noting the hype and backlash that have dogged the product, Savage describes an epiphany he had while trying on the headset at home, upstairs in his office. “I turned it on and I could hear a whale,” he says, “but I couldn’t see it. I’m looking around my office for it. And then it swims by my windows—on the outside of my building! So the glasses scanned my room and it knew that my windows were portals and it rendered the whale as if it were swimming down my street. I actually got choked up.” What Savage encountered on the other side of the glasses was a glimpse of the mirrorworld.

The mirrorworld doesn’t yet fully exist, but it is coming. Someday soon, every place and thing in the real world—every street, lamppost, building, and room—will have its full-size digital twin in the mirrorworld. For now, only tiny patches of the mirrorworld are visible through AR headsets. Piece by piece, these virtual fragments are being stitched together to form a shared, persistent place that will parallel the real world. The author Jorge Luis Borges imagined a map exactly the same size as the territory it represented. “In time,” Borges wrote, “the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it.” We are now building such a 1:1 map of almost unimaginable scope, and this world will become the next great digital platform.

Google Earth has long offered a hint of what this mirrorworld will look like. My friend Daniel Suarez is a best-selling science fiction author. In one sequence of his most recent book, Change Agent, a fugitive escapes along the coast of Malaysia. His descriptions of the roadside eateries and the landscape describe exactly what I had seen when I drove there recently, so I asked him when he’d made the trip. “Oh, I’ve never been to Malaysia,” he smiled sheepishly. “I have a computer with a set of three linked monitors, and I opened up Google Earth. Over several evenings I ‘drove’ along Malaysian highway AH18 in Street View.” Suarez—like Savage—was seeing a crude version of the mirrorworld.

It is already under construction. Deep in the research labs of tech companies around the world, scientists and engineers are racing to construct virtual places that overlay actual places. Crucially, these emerging digital landscapes will feel real; they’ll exhibit what landscape architects call place­ness. The Street View images in Google Maps are just facades, flat images hinged together. But in the mirrorworld, a virtual building will have volume, a virtual chair will exhibit chairness, and a virtual street will have layers of textures, gaps, and intrusions that all convey a sense of “street.”

The mirrorworld—a term first popularized by Yale computer scientist David Gelernter—will reflect not just what something looks like but its context, meaning, and function. We will interact with it, manipulate it, and experience it like we do the real world.

At first, the mirrorworld will appear to us as a high-resolution stratum of information overlaying the real world. We might see a virtual name tag hovering in front of people we previously met. Perhaps a blue arrow showing us the right place to turn a corner. Or helpful annotations anchored to places of interest. (Unlike the dark, closed goggles of VR, AR glasses use see-through technology to insert virtual apparitions into the real world.)

Eventually we’ll be able to search physical space as we might search a text—“find me all the places where a park bench faces sunrise along a river.” We will hyperlink objects into a network of the physical, just as the web hyperlinked words, producing marvelous benefits and new products.

The mirrorworld will have its own quirks and surprises. Its curious dual nature, melding the real and the virtual, will enable now-unthinkable games and entertainment. Pokémon Go gives just a hint of this platform’s nearly unlimited capability for exploration.

These examples are trivial and elementary, equivalent to our earliest, lame guesses of what the internet would be, just after it was born—fledgling Compu­Serve, early AOL. The real value of this work will emerge from the trillion unexpected combinations of all these primitive elements.

The first big technology platform was the web, which digitized information, subjecting knowledge to the power of algorithms; it came to be dominated by Google. The second great platform was social media, running primarily on mobile phones. It digitized people and subjected human behavior and relationships to the power of algorithms, and it is ruled by Facebook and WeChat.

We are now at the dawn of the third platform, which will digitize the rest of the world. On this platform, all things and places will be machine-­readable, subject to the power of algorithms. Whoever dominates this grand third platform will become among the wealthiest and most powerful people and companies in history, just as those who now dominate the first two platforms have. Also, like its predecessors, this new platform will unleash the prosperity of thousands more companies in its ecosystem, and a million new ideas—and problems—that weren’t possible before machines could read the world.


Augmented reality is the technology underpinning the mirrorworld; it is the awkward newborn that will grow into a giant. “Mirrorworlds immerse you without removing you from the space. You are still present, but on a different plane of reality. Think Frodo when he puts on the One Ring. Rather than cutting you off from the world, they form a new connection to it,” writes Keiichi Matsuda, former creative director for Leap Motion, a company that develops hand-­gesture technology for AR.

The full blossoming of the mirrorworld is waiting for cheap, always-on wearable glasses. Speculation has been rising that one of the largest tech companies may be developing just such a product. Apple has been on an AR hiring spree and recently acquired a startup called Akonia Holographics that specializes in thin, transparent “smart glass” lenses. “Augmented reality is going to change everything,” Apple CEO Tim Cook said during an earnings call in late 2017. “I think it’s profound, and I think Apple is in a really unique position to lead in this area.”

But you don’t need to use AR glasses; you can engage using almost any kind of device. You can kind of do this today with Google’s Pixel phone, but without the convincing presence that you get with 3D visors. Even now, wearables like watches or smart clothes can detect the proto-mirrorworld and interact with it.

EVERYTHING CONNECTED TO the internet will be connected to the mirrorworld. And anything connected to the mirrorworld will see and be seen by everything else in this interconnected environment. Watches will detect chairs; chairs will detect spreadsheets; glasses will detect watches, even under a sleeve; tablets will see the inside of a turbine; turbines will see workers around them.

The rise of a massive mirrorworld will rely in part on a fundamental shift underway right now, away from phone-centric life and toward a technology that is two centuries old: the camera. To recreate a map that is as big as the globe—in 3D, no less—you need to photograph all places and things from every possible angle, all the time, which means you need to have a planet full of cameras that are always on.

We are making that distributed, all-seeing camera network by reducing cameras to pinpoint electric eyes that can be placed anywhere and everywhere. Like computer chips before them, cameras are becoming better, cheaper, and smaller every year. There may be two in your phone already and a couple more in your car. There is one in my doorbell. Most of these newer artificial eyes will be right in front of our own eyes, on glasses or in contacts, so that wherever we humans look, that scene will be captured.


But here’s the most important thing: Robots will see this world. Indeed, this is already the perspective from which self-driving cars and robots see the world today, that of reality fused with a virtual shadow. When a robot is finally able to walk down a busy city street, the view it will have in its silicon eyes and mind will be the mirrorworld version of that street. The robot’s success in navigating will depend on the previously mapped contours of the road—existing 3D scans of the light posts and fire hydrants on the sidewalk, of the precise municipal position of traffic signs, of the exquisite details on doorways and shop windows rendered by landlord scans.


In the mirrorworld too, virtual bots will become embodied; they’ll get a virtual, 3D, photorealistic shell, whether machine, animal, human, or alien. Inside the mirror­world, agents like Siri and Alexa will take on 3D forms that can see and be seen. Their eyes will be the embedded billion eyes of the matrix. They will be able not just to hear our voices but also, by watching our virtual avatars, to see our gestures and pick up on our microexpressions and moods. Their spatial forms—faces, limbs—will also increase the nuances of their interactions with us. The mirrorworld will be the badly needed interface where we meet AIs, which otherwise are abstract spirits in the cloud.

There is another way to look at objects in the mirrorworld. They can be dual use, performing different roles in different planes. “We can pick up a pencil and use it as a magic wand. We can turn our tables into touchscreens,” Matsuda writes.

We will be able to mess not only with the locations and roles of objects but with time as well. Say I’m walking along a path beside the Hudson River, the real Hudson River, and I notice a wren’s nest that my bird-­watching friend would be keen to know about, so I leave a virtual note along the path for her. It remains there until she passes by. We saw the same phenomenon of persistence with Pokémon Go: virtual creatures remaining in a real physical location, waiting to be encountered. Time is a dimension in the mirror­world that can be adjusted. Unlike the real world, but very much like the world of software apps, you will be able to scroll back.

History will be a verb. With a swipe of your hand, you will be able to go back in time, at any location, and see what came before. You will be able to lay a reconstructed 19th-century view right over the present reality. To visit an earlier time at a location, you simply revert to a previous version kept in the log. The entire mirror­world will be like a Word or Photo­shop file that you can keep “undoing.” Or you’ll scroll in the other direction: forward. Artists might create future versions of a place, in place. The verisimilitude of such crafty world-building will be revolutionary. These scroll-forward scenarios will have the heft of reality because they will be derived from a full-scale present world. In this way, the mirrorworld may be best referred to as a 4D world.

LIKE THE WEB and social media before it, the mirror­world will unfold and grow, producing unintended problems and unexpected benefits. Start with the business model. Will we try to jump-start the platform with the shortcut of advertising? Probably. I am old enough to remember the internet before it allowed commercial activity, and it was just too broke to grow. A commercial-free mirrorworld would be infeasible and undesirable. However, if the only business model is selling our attention, then we’ll have a nightmare—because, in this world, our attention can be tracked and directed with much greater resolution, which subjects it to easy exploitation.


The emergence of the mirrorworld will affect us all at a deeply personal level. We know there will be severe physiological and psychological effects of dwelling in dual worlds; we’ve already learned that from our experience living in cyberspace and virtual realities. But we don’t know what these effects will be, much less how to prepare for them or avoid them. We don’t even know the exact cognitive mechanism that makes the illusion of AR work in the first place.

The great paradox is that the only way to understand how AR works is to build AR and test ourselves in it. It’s weirdly recursive: The technology itself is the microscope needed to inspect the effects of the technology.


Without exception, every researcher in this field that I’ve spoken to has been acutely aware of these divergent paths and claims to be working toward a decentralized model—for many reasons, including the chief one that a decentralized and open platform will be richer and more robust. Clay Bavor, vice president of AR and VR at Google, says, “We want an open service that gets better each time someone uses it, like the web.”

The mirrorworld will raise major privacy concerns. It will, after all, contain a billion eyes glancing at every point, converging into one continuous view. The mirrorworld will create so much data, big data, from its legions of eyes and other sensors, that we can’t imagine its scale right now. To make this spatial realm work—to synchronize the virtual twins of all places and all things with the real places and things, while rendering it visible to millions—will require tracking people and things to a degree that can only be called a total surveillance state.

We reflexively recoil at the specter of such big data. We can imagine so many ways it might hurt us. But there are a few ways big data might benefit us, and the prime one is the mirrorworld. The route to civilizing big data so that we gain more than we lose is uncertain, complex, and not obvious.

But we already have some experience that can inform our approach to the mirrorworld. Good practices include mandatory transparency and accountability for any party that touches the data; symmetry in the flow of information, so that the watchers are themselves watched; and the insistence that data creators—you and me—receive clear benefits, including monetary ones, from the system. I am optimistic that a viable path can be found to handle this ubiquitous data, because the mirrorworld is not the only place it will accumulate. Big data will be everywhere. My hope is that with a fresh start, the mirrorworld is the place we can figure this out first.

FROM THE EARLIEST stirrings of the internet, the digital world was seen as a disembodied cyberspace—an intangible realm separated from the physical world, and so unlike material existence that this electronic space could claim its own rules. In many respects, the virtual and the physical worlds have indeed run in parallel, never meeting. In the virtual there was a sense of infinite liberty, unleashed by disconnecting from physical form: free of friction, gravity, momentum, and all the Newtonian constraints holding us back. Who wouldn’t want to escape into cyberspace to become the best (or worst) version of themself?

The mirrorworld bends that trajectory upon itself. Rather than continue two separate realms, this new platform melds the two so that digital bits are embedded into materials made of atoms. You interact in the virtual by interacting in the physical, moving your muscles, stubbing your toes. Information about that famous water fountain in a Roman plaza can be found at that fountain in Rome. To troubleshoot a 180-foot wind turbine, we troubleshoot its digital ghost. Pick up a towel in your bathroom and it becomes a magical cape. We will come to depend on the fact that every object contains its corresponding bits, almost as if every atom has its ghost, and every ghost its shell.

I imagine it will take at least a decade for the mirrorworld to develop enough to be used by millions, and several decades to mature. But we are close enough now to the birth of this great work that we can predict its character in rough detail.

Eventually this melded world will be the size of our planet. It will be humanity’s greatest achievement, creating new levels of wealth, new social problems, and uncountable opportunities for billions of people. There are no experts yet to make this world; you are not late.

Kevin Kelly was WIRED’s founding executive editor. He’s the author of The Inevitable: Understanding the 12 Technological Forces That Will Shape Our Future and many other books, including What Technology Wants; New Rules for the New Economy; and Out Of Control: The New Biology of Machines, Social Systems, and the Economic World

This entry was posted in Presence in the News. Bookmark the permalink. Trackbacks are closed, but you can post a comment.

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>


  • Find Researchers

    Use the links below to find researchers listed alphabetically by the first letter of their last name.

    A | B | C | D | E | F| G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z