[A new Facebook feature will create navigable 3D virtual reproductions of places where our memorable experiences occurred. Interestingly, the limitations of the reproductions (a “pointillism effect”) apparently create both the “sense of presence in a place… [and] a sense of memory.” This first-person report on Facebook’s demo at the F8 conference is from Engadget, where it includes more images. See CNET for a 2:14 minute video. –Matthew]
Facebook’s AI-created virtual memories are magically haunting
Wandering through pointillistic art felt like being in a dream.
Nicole Lee
May 1, 2018
Ever since Facebook bought Oculus, it’s been investing in virtual reality in a big way. Beyond just hardware like the Rift and the Go, it’s also putting a lot of money and research into software. It delved into the realm of social last year with Oculus Rooms and Facebook Spaces, but that’s not where Facebook’s VR ambitions stop. At F8, the company unveiled a new way for users to experience VR: by recreating their memories.
At the F8 keynote this week, head of social VR Rachel Franklin showed off an experimental feature where Facebook’s machine learning algorithms could convert a 2D photo or video into a virtual three-dimensional environment through point cloud reconstruction. According to Facebook, the tech uses “photogrammetry” to create 3D space from flat photos or videos. This means that you could conceivably transform a photo of your 10th birthday party into a virtual recreation of your childhood living room.
I went through a demo of this feature at F8, and it felt a lot like being in a dream. Thanks in part to the point cloud reconstruction mentioned earlier, the virtual room that I was in was not lifelike in the slightest. Instead, it was as if I stepped into a world of pointillistic art, which lent the environment an ethereal, magical quality. When I waved my cursor hand around, the dots that make up the room shifted and moved, emphasizing the unreal nature of the world around me.
Scattered throughout the place — which seems to be a recreation of a real Oakland home — were cut-out squares floating in the air. When I hovered over them with a cursor, I saw thumbnails of photos and videos, all of which were supposedly taken in the room that I was in. When I clicked on the thumbnails, I teleported over to them so that I could see the photos and videos up close. One was a photo of a family, while another was a short video clip of a young couple getting ready for prom.
The next environment I was in was yet another living room, and this time all of the photos and videos were taken during a birthday party. The thumbnail near the couch revealed a posed photo of several attendees, while the one by the dinner table showed the crowd around the cake. In the very last scenario, I was transported over to a pointillist version of the Palace of Fine Arts in San Francisco. The photos taken at this location were lined up in a path surrounding the lake, inviting you to follow along.
Facebook told me that this pointillism effect is used to help fill in the gaps, and to “create a sense of presence in the space.” But more than that, the fuzzy nature of the environment also evokes a sense of memory. After all, most people don’t dream in HD. Instead, dreams and memories are misty and hazy, and a pointillistic world where objects are blurry and ill-defined contributes to that feeling immensely. The fact that the environment was peppered with real-life photos was pretty surreal as well. Teleporting from one photo to another felt like I was hopping in and out of reality. I sometimes felt as if I was going through forgotten memories like in Eternal Sunshine of the Spotless Mind.
According to Facebook, the demo I experienced is just a prototype, and it has no plans to share specific implementations. In the future, the company hopes to help people share moments in “new and rich ways.” “We think in the future, people may be able to use technology like this to revisit moments and places that are important to them, with people they care about,” said a Facebook spokesperson in a statement.
In short, think of this as a Facebook album come to life. I can imagine a future where instead of forcing your relatives to sit through a boring vacation slideshow on the TV, you’ll be strapping them to headsets so that they can relive your trip to Italy with you. They still might not appreciate it, but at least it won’t be quite as boring.