[This story from The Guardian highlights one of, if not the, most important applications of presence-evoking technology, the preservation of individual and cultural history. The original version includes an additional image and a 0:51 minute video; also see “Holocaust Survivor Experiences Her Own Rescue in Virtual Reality“ (which includes a video) in TIME. –Matthew]

The virtual Holocaust survivor: How history gained new dimensions

Pinchas Gutter survived a Nazi death camp – and now his story will live on through a hologram that can answer your questions

Thomas McMullan
18 June 2016

Pinchas Gutter goes out of his way to find me biscuits. In a sun-baked living room in his north London home, he opens a packet of Rich Tea, sits down and tells me about the Holocaust.

Gutter was seven years old when the second world war broke out. He lived in the Warsaw ghetto for three and a half years, took part in its uprising, survived six Nazi concentration camps – including the Majdanek extermination camp – and lived through a death march across Germany to Theresienstadt in occupied Czechoslovakia.

“Remembrance is the secret of redemption, while forgetting leads to exile,” he says, quoting the Baal Shem Tov, the founder of Hasidic Judaism.

“For me this whole thing is about not forgetting. Because if you forget, you repeat it over and over again. To me, the great importance of testimony is not to forget what people are capable of.” Sitting opposite him here in Hampstead, meeting his eyes and hearing his voice, the idea of forgetting this extraordinary story seems impossible.

One week later I am staring into Gutter’s eyes again. But these eyes are on a screen in the Alternate Realities strand of Sheffield’s international documentary festival. The version of Gutter projected on the monitor is a prototype for a responsive hologram that will be wheeled into classrooms, lecture halls and museums. The idea is that the audience asks questions and pre-recorded memories from Gutter will respond – much as if you’re talking to the real person. This virtual Gutter meets my gaze and tilts his head when I speak into the microphone.

The project, called New Dimensions in Testimony, was thought up by concept developer Heather Maio, and was made in a collaboration between the University of Southern California’s Institute for Creative Technologies (ICT) and the Shoah Foundation – an organisation dedicated to making and archiving interviews with survivors and witnesses of genocide.

The prototype involved filming extensive interviews with Gutter using an array of cameras and a specialised light stage. The team at ICT then used natural language processing software to help create an interactive version of the video footage, with vocal cues triggering responses from the pool of recorded speech.

Complex algorithms dictate which responses are played back – faintly reminiscent of virtual assistants such as Apple’s Siri and Microsoft’s Cortana, except that while those pieces of software are primed to answer your questions about emails or the weather, New Dimensions in Testimony is based on communicating one man’s experience of genocide.

“We wanted the visitor to experience the discussion and what that means to them,” Maio tells me. “Not the technology that goes around it and makes it work. In fact, we didn’t want them to think there was technology around them at all.” Read more on Preserving experience: The virtual Holocaust survivor…

[Presence-evoking technology continues to advance toward the visions imagined in science fiction; note especially the possibilities for fully interactive videoconferencing and entertainment mentioned in this press release from the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute. –Matthew]

[Image: Right in the middle of the action with VR glasses: A new camera system virtually represents people naturally and realistically. © Photo Fraunhofer HHI]

Beam me up to the video conference

When science fiction heroes communicate, they don‘t use landlines or cell phones. The caller simply appears in virtual form in the middle of the room; full sized and three dimensional. For researchers at Fraunhofer, this vision is already within reach: At the trade fairs IFA in Berlin (Hall 11.1, Booth 3) and IBC in Amsterdam (Hall 8, Booth 8.B80) they will show a new technology called “3D Human Body Reconstruction”.

September 1, 2016

Scientists at the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute, HHI in Berlin have developed a method by which the realistic image of a person can be transmitted in a virtual world; and just like in science fiction movies, the image appears full sized and three dimensional. The image can be viewed from different directions and the viewer can even walk around it – just like in the movie. Until now, this was not possible; even virtual reality (VR) still has its limits. People can be represented by artificial three-dimensional models (so-called avatars) that can be seen when the viewer puts on VR data goggles. Nevertheless, these artificial figures do not have a lifelike appearance or natural movement. Another option is to play the video image of a person in frontal view in the VR data goggles. However, the viewer cannot walk around the image. As a result, the whole scene looks artificial as one moves through the virtual world. The person always turns his two-dimensional front to the viewer.

20 stereo cameras

In contrast, the HHI researchers have perfected the three-dimensional impression. To do so, they have developed a camera system that films the person. The core of this system is a stereo camera: Just as people do with their two eyes, the camera records the person with two lenses. This stereoscopic vision results in distances being estimated well, because both eyes look at an object from a slightly different angle. The result is a three-dimensional impression. Recording a person in detail from all directions takes more than one camera. “We are currently using more than 20 stereo cameras to map a human,” says Oliver Schreer, Head of the Research Group “Immersive Media & Communication” at HHI. Each camera only captures a part of the person. The challenge is to merge the individual camera images together so that a realistic overall picture is produced.

The system includes more than just the camera technology. The researchers have developed algorithms that can quickly extract depth information from the stereoscopic camera images. This is necessary in order to calculate the 3D form of a captured person. The computer calculates a virtual model of the human, which is then transferred into the virtual scene. The cameras perceive the surface shape with many details. In this way even small wrinkles, e.g. on the clothes of the person, can be shown. The model has a natural and realistic appearance.

“In developing these algorithms, special care has been taken to ensure they work efficiently and fast, so the movements of dialogue partners can very quickly be converted into a dynamic model,” Schreer says, since this is the only way that the movements will look natural. The images from a single camera pair can be processed in real time. Read more on New HHI system allows realistic, full-size 3D human body reconstruction in virtual environments…

[Now you can experience America’s controversial political campaign in virtual reality; this story is from Next Reality, where it includes more images and a 0:29 minute video. –Matthew]

[Image: Virtual avatars of Hillary and Trump nearly make physical contact in virtual space. Will this someday mirror reality? Via AltspaceVR]

AltspaceVR’s Democracy Plaza Puts You in the Middle of the 2016 Election

Adam Dachis
September 22, 2016

Most of us don’t get the opportunity to see election coverage first hand, but with virtual reality, we’re not as restricted. AltspaceVR has teamed up with NBC News to let you log in on your computer or VR headset and actually take part in the action—just not physically.

AltspaceVR created a virtual Rockefeller Center Plaza where we can take our virtual avatars to get a front row seat of NBC News’ election coverage this year. Anyone with a Mac or Windows PC can tune in and participate, but if you want the immersive experience, you’ll need a VR headset like the HTC Vive, Oculus Rift, or Samsung Gear VR. Read more on AltspaceVR’s Democracy Plaza puts you in the middle of the 2016 election…

[As this story from ScreenCrush notes, there are many factors that can ‘break’ presence in a movie theater, including the lack of proper screen masking. –Matthew]

A Small Issue at Movie Theaters Is Becoming a Major Problem

Matt Singer | September 19, 2016

People texting on their phones. Exorbitant ticket prices. People talking on their phones. Sound that’s either way too loud or way too soft. People taking pictures of the screen with their phones and then posting them to Facebook in the middle of the film. Going to the movies in 2016 can be a frustrating situation. An experience designed to take us away from our everyday troubles is now fraught with them.

For a lot of these problems, there’s no easy solution. But there’s one issue that I’ve noticed happening more and more that, if properly addressed, would make a significant improvement on the theatrical experience. It’s a problem that requires a small fix, one so small I’m surprised it’s become an issue in the first place: Theaters not properly masking their screens. Read more on Presence in movie theaters: The role of screen masking…

[As the author of this piece in TreeHugger notes, mixed reality (and the presence experiences it provides) has great potential beyond the important application area of elevator maintenance and repair described here. The original story includes two videos and several more images; the press release is available from thyssenkrupp. –Matthew]

[Image: © thyssenkrupp/ using hololens to look at elevator]

It’s starting with elevators, but 3D tech like the Hololens will change the way we look at buildings

Lloyd Alter
September 16, 2016

This site has long been preoccupied with the future of work: how new technology is changing our jobs, and can possibly change our ecological footprint. I saw a bit of that future in New York City, with the demonstration of the use of Microsoft’s Hololens by thyssenkrupp Elevator.

The Hololens is described by Microsoft as “the first fully self-contained, holographic computer, enabling you to interact with high‑definition holograms in your world.” It’s not virtual reality, but what they call “Mixed reality”, which blends 3D holographic content into your physical world, “giving your holograms real-world context and scale, allowing you to interact with both digital content and the world around you.” You can see it in action in this video:

Essentially, you look through the goggles and see the object, can walk around it, zoom in and out, rotate and if it was built to do so, explode it into its components. As an architect, I can say right now that this is going to revolutionize the design, construction and maintenance of buildings, perhaps as dramatically as CAD and BIM have.

I believe this because elevators are a big, complex and expensive part of buildings, and thyssenkrupp Elevator is revolutionizing the way they work on them right now. However the technologies that they are demonstrating have broad implications for everyone in the building and many other businesses. Read more on It’s starting with elevators, but 3D tech like the Hololens will change the way we look at buildings…