[Pretty amazing… this is from Motherboard, where you can watch the 3:41 minute video; in addition to the link in the post, more images, including animated gifs, are available at Colossal]
MIT’s 3D Touch Display Brings the Virtual World into Real Life
By DJ Pangburn
January 5, 2013
It didn’t take long for 2014 to produce some rad videos of technology in action. MIT’s Tangible Media Group just released a video of its remote “touch” display, inFORM, allowing a user to manipulate the real, 3D world through some nifty digital wizardry.
Led by profession Hiroshi Ishii, the Tangible Media Group describes inFORM as a “Dynamic Shape Display that can render 3D content physically, so users can interact with digital information in a tangible way.” InFORM can also create interactions with the world around it. In the video, a remote participant uses inFORM to move various objects sitting atop a table’s surface.
In addition to the video, the Tangible Media Group released several hi-res images of inFORM in action. One of the more interesting of which is of a pair of pixelated hands holding a red flashlight. Surface level, it looks simple enough, but inFORM’s innards look like a futuristic loom. As it turns out, the project is an area where older science and engineering meets the new.
So what makes inFORM tick? Tangible Media Group’s Sean Follmer explained that the system uses a depth camera (in this case a Microsoft Kinect) that captures both the depth image and the color image of the remote participant’s hands. A second RGB camera captures the participant’s face. These images are then sent over the network to the inFORM system. From there, inFORM’s system functions much like the sliders on an audio or film mixing board.
“Basically, what’s going on with the inFORM system is that we have 900 actuators that are slide potentiometers used in mixing boards,” said Follmer. “These have small motors on them that move the controllers up and down the sliders, while simultaneously recording their positions.” Micro-computers read the pin’s current position, then change that position based on where it needs to go next. All of this is updated extremely quickly.
On the display side, the inFORM computer system then sends messages to the 900 actuators, which are spaced in a 30×30 grid, allowing it to render 3D shapes. Follmer describes the cables as similar to bicycle breaks. Follmer added that inFORM is also able to render 3D content on computers, such as 3D models or user interfaces.
“It’s not so different from a loom,” said Follmer. “Think of the loom as a physical manifestation of computation. Here it’s maybe digital computation turning it back into physical. InFORM transorms digital information into physical output.”
On the group’s website, researchers describe how they are exploring new applications for the inFORM shape display. One area where they see much promise is in mapping, especially urban landscapes. “Urban planners and architects can view 3D designs physically and better understand, share and discuss their designs,” they write. “We are collaborating with the urban planners in the Changing Places group at MIT on this. In addition, inFORM would allow 3D modelers and designers to prototype their 3D designs physically without 3D printing (at a low resolution).”
The Tangible Media Group also believes there could be applications in the medical field for what they are calling their Material User Interface (MUI). They want to be able to manipulate objects in hospitals, as well as explore the boundaries of surgical simulations. Imagine a surgeon appearing on inFORM from halfway across the world, their movements rendered in physical pixels. It would be mind-bending to behold, and maybe even a little freaky.
Follmer thinks that this technology could have wider applications in entertainment and gaming. Take table-top games, for example. InFORM would allow these games to change physical states. “It’s more of a platform for us to think about what the future could be when you can change the physical form of the computing device to match what you want it to,” added Follmer. “It’s like when people looked at the first Macintosh or Xerox computer with the bitmap screen and asked, ‘What is the application?’ Well, the application is everything with the computer. We think this is an open platform onto which you can output anything you want.”
Though inFORM exists only in prototype right now, researchers are interested in exploring other form factors. Currently, the Tangible Media Group is looking at two approaches for shape-changing user interfaces. The first approach, described above, allows for hi-resolution shape change, but this one-actuator-per-pixel system is limited by how many of those connections can be physically made.
The other approach is inspired by nature and soft robotics, such as the shape changes that can be observed in octopuses, different insects, and animals. Theoretically, a small actuator that is soft like an octopus could elongate and change a whole range of material and shape properties.
“We’ve developed a number of pneumatically-driven shape-changing user interfaces that then can sort of shrink down the size of these actuators,” said Follmer. “To get complex motion and shape change, you don’t need as many actuators, but then you don’t have the flexibility of being able to change the shape on demand. You’re limited to a set of different pre-defined shapes that it can change into.”
Follmer said that a number of other researchers are currently working on miniaturizing shape displays. A colleague in the mechanical engineering department, Benjamin Peters, is working on using shape displays for shape tooling, or changeable pin-tooling. Peters uses solder joints as mechanical clutches. A resistor then heats up the solder joint, allowing the pin to move freely. This technology has great potential for miniaturization, which could help with the Tangible Media Group’s own research.
Ultimately, the inFORM project is, according to the researchers, a step toward their vision of Radical Atoms, a “hypothetical generation of materials that can change form and appearance dynamically, becoming as reconfigurable as pixels on a screen.” The idea being that we could create changes in physical forms and digital states in real-time, and in both directions. In other words, man becomes robot, and robot becomes man, or something close to it anyway.