[This story from Nature describes the nascent use of virtual and augmented reality, and the presence experiences they produce, for scientific research. See the last paragraph for a succinct summary. –Matthew]
[Image: Credit: The Project Twins]
Virtual-reality applications give science a new dimension
Virtual- and augmented-reality tools allow researchers to view and share data as never before. But so far, they remain largely the tools of early adopters.
David Matthews
30 April 2018
As I put on a virtual-reality (VR) headset, the outside world disappears. A cell fills my visual field, and as I crane my neck, I can see it from several angles. I stick my head inside to explore its internal structure. Using hand controllers, I dissect the cell layer by layer, excavating with a flick of the wrist to uncover tiny, specialized structures buried beneath the surface.
Looking at a cell in VR is “as close as you can get to touching” such a minuscule structure, says Sebastian Konrad, product manager for VR at Arivis, a life-sciences software company in Munich, Germany, that developed this particular VR visualization tool, called InViewR, and who helped to arrange my demonstration of it.
VR isn’t new, but interest in the technology has boomed since 2016, when gamers and a handful of scientists introduced several high-quality, relatively inexpensive commercial headsets to the public. A similar surge has emerged in augmented reality (AR), a related technology that uses a see-through visor or smartphone screen to layer objects on top of real surroundings.
Some scientists see VR and AR as more intuitive to use than conventional flat screens for viewing complex 3D structures. Others have sought cheap, smartphone-based headsets, which use a smartphone screen as the goggles, to increase public understanding of their work. Their numbers are relatively small: VR and AR remain niche tools for scientific research. Yet some researchers say that the technology has provided new insights.
Adam Lacy-Hulbert is a principal investigator at the Benaroya Research Institute in Seattle, Washington. He is particularly interested in lysosomes — structures that help to clean up the insides of cells. But he was perplexed by some of the 2D images he was getting using conventional microscopy. “It looked as if part of the lysosomes of the cell had moved into the nucleus, which didn’t really make sense to us.”
But ConfocalVR, a tool developed at Benaroya that uses VR to visualize images from confocal microscopes, made what was really happening “jump out within seconds”, Lacy-Hulbert says. The nucleus was actually deforming and moving around the lysosomes.
Wilian Cortopassi, a postdoctoral researcher at the University of California, San Francisco, has also gained scientific insights from VR explorations. ChimeraX is a molecular-visualization tool for proteins and other structures, which added support for VR headsets in November 2016. ‘Walking’ in virtual space through a network of hydrogen bonds helped Cortopassi to understand how certain mutants of a protein could stymie drugs that target it. A computer monitor is “so messy when you turn on a lot of atoms for visualization”, Cortopassi says. But in VR, “you can just walk through the hydrogens at different angles and distances, and quickly detect some important interactions”.
Goggle-eyed
Although inexpensive options are available, most visualization tools work only with the priciest headsets — such as Facebook’s Oculus Rift, and the Vive from Taiwanese electronics company HTC — because they can track the user’s head and handheld-controller movements in 3D space. Researchers and gamers have their preferences, but the differences between Oculus Rift and Vive are small. “I don’t think there’s a clear winner at this point,” says Tom Ferrin, one of the developers of ChimeraX, whose lab at the University of California, San Francisco, specializes in molecular-visualization tools.
That said, not every tool is compatible with all headsets. InViewR works only with Oculus Rift, whereas ChimeraX and ConfocalVR work with both. Oculus Rift and Vive both run using the Windows operating system, although Vive is also compatible with MacOS X.
VR is computationally intensive, both because each eye must see a different image to produce a 3D effect, and because those images must refresh rapidly. In some cases, a new graphics card will add sufficient computing power, “but in general you’re probably going to buy a new computer”, says Tom Skillman, director of informatics and research technology at Benaroya and one of the creators of ConfocalVR. Oculus Rift suggests using VR-compatible computers ranging from US$850 to nearly $3,100; it recommends at least 8 gigabytes of memory and a high-end graphics card.
The VR software itself can also be expensive. Although ConfocalVR and ChimeraX are free for non-profit entities, that is not true for commercial firms. ConfocalVR declined to share pricing information, but ChimeraX can cost up to $20,000, depending on the number of users.
For researchers who like to work as a team, the developers of ConfocalVR added in April the option for up to four users to simultaneously view, point to and grab structures in the same VR space. This could mean that scientists do not have to meet face to face to work together, says Skillman, which would potentially reduce travel costs. The developers of both ChimeraX and InViewR are looking to add similar collaborative features in the future.
Augmenting reality
Compared with VR, visualization software for AR headsets is less advanced. Mark Hoffman, chief research information officer at Children’s Mercy Kansas City, a hospital in Missouri, has experimented with viewing proteins and computed tomography (CT) scans using Microsoft’s HoloLens — a kind of visor with a built-in computer that projects 3D objects over the real world.
He says that AR is more user-friendly than VR because users can see their surroundings and so are less prone to disorientation. Hoffman actually experiences motion sickness in VR — and this is not an uncommon complaint. “In all my work with the HoloLens, I’ve never been uncomfortable,” he says.
The downside is that, whereas a VR headset envelops your entire field of view, the HoloLens projects objects only onto a relatively narrow rectangle in the centre of your vision. “It’s part of the trade-off,” Hoffman says. AR is not completely immersive, but it is “an enabler to comprehension”, he says. “There may be things you can miss on a flat screen” that become clearer in AR — protein–protein interactions, for instance.
Surgeons at Children’s Mercy are exploring the use of AR to view CT scans of patients’ hearts before an operation, he says. Hoffman uses a step-by-step approach to make such data viewable using the HoloLens. The surgeon can explore the tissue by projecting it onto a fixed point in space — say, in the middle of the room. But if they turn their head, the image disappears and they see only what is actually there. “They walk into the ventricle or the atrium of the heart, and maybe they’ll see that, for a particular child, the entry point of a blood vessel is not where it normally would be.” The HoloLens costs $3,000, and must be ordered from Microsoft directly, because it is not available in the shops.
Low-cost options
Cheaper headsets that use smartphones as the screen in a pair of goggles, such as the Samsung Gear VR or Google’s $15, ultra-simple Cardboard, can help researchers to reach a broader audience.
Juicebox VR, an app designed for these simple devices, visualizes the connectivity of the human genome as a Mars-like landscape scarred with a colossal wall, says Erez Aiden, a geneticist at Baylor College of Medicine in Houston, Texas, whose lab developed the tool. The features of the landscape represent the topography of condensed DNA in animal cells, and the ridge represents intersections between different parts of the genome. “When people interact with this, they really get a sense of what the data look like,” he says.
Biologists have also adopted Augment, an app normally used to illustrate how furniture might look in a room, to allow colleagues, students and members of the public to inspect 3D models of proteins through their smartphone screens.
For researchers interested in creating their own visualization tools, Unity — software designed by Unity Technologies in San Francisco for building games — is one of the most commonly used development environments. It runs on relatively modest hardware, says Muhammad Saad Shamim, who used it to help to develop Juicebox VR on a Mac Pro. For the HoloLens, users needn’t be advanced developers to import 3D objects, Hoffman says. But they should be comfortable with Unity, as well as Microsoft’s Visual Studio programming environment. Other options include Unreal Engine, from Epic Games in Cary, North Carolina, which is free for academic users, and OpenGL, a no-cost 3D-graphics tool used in game development, computer-aided design and flight simulators. Ferrin, who used OpenGL to create ChimeraX, says OpenGL requires more initial work than Unity or Unreal because developers need to handle more programming details directly, but the pay-off is fewer constraints on functionality.
Despite the broad proliferation of VR and AR tools in consumer culture, only a small minority of labs currently uses the technology, and it remains to be seen how many others will follow suit. Yet many advocates predict that VR and AR could become standard lab tools over the next five years or so. The technology feeds information to our brains in three dimensions, the way “a million years of evolution” intended, says Skillman. It requires an enormous amount of intellectual work to construct a 3D mental model from a 2D screen, he says. “All that work goes away when you put on the goggles.”