[This installation creates an interesting set of embodiment illusions; the creators explain the origins and purpose of the project in the second half of this post from the Leap Motion blog, which includes lots of pictures and videos. –Matthew ]
Reach into the Uncanny Valley with the Augmented Hand Series
Golan Levin / March 5, 2015
An earlier version of this post appeared on flong.com.
The Augmented Hand Series is a real-time interactive software system that presents playful, dreamlike, and uncanny transformations of its visitors’ hands. Originally conceived in 2004, the project was developed at the Frank-Ratchye STUDIO for Creative Inquiry in 2013-2014 through a commission from the Cinekid Festival of children’s media.
The installation consists of a box into which the visitor inserts their hand, and a touchscreen interface which displays their ‘reimagined’ hand, altered by various dynamic and structural transformations. In the version shown here, in its premiere at the 2014 Cinekid Festival in Amsterdam, the kiosk is accompanied by a large rear-projection. The touchscreen allows participants to select among the different transformations.
Critically, the project’s morphological transformations operate within the logical space of the hand itself. That is to say: the artwork performs “hand-aware” visualizations that alter the deep structure of how the hand appears—unlike, say, a funhouse mirror, which simply distorts the entire field of view.
The system uses the real-time posture of the participant’s real hand as the moment-to-moment baseline for its transformations. Participants are free to use either of their hands, and the system works properly even with visitors who wiggle their fingers, or who move and turn their hand—within certain limits. The software may produce unpredictable glitches if the visitor’s hand differs significantly from a flat palm-down or palm-up pose. Currently, the system’s behavior for postures like fists (in which many of the fingers are occluded) is undefined, as is multi-hand interaction.
Developing an interaction that can work with a very diverse public is always a challenge. The Augmented Hand Series accommodates a wide range of hand sizes, from children (of about 5 years old) through adults, as well as a very broad range of skin colors. The system also performs robustly with hands that have jewelry, nail polish, tattoos, birthmarks, wrinkles, arthritic swelling, and/or unusually long, short, wide or slender digits. Nevertheless, the system’s behavior for individuals with more or fewer than five fingers is presently undefined. There are many more kinds of hands to support, and doing so remains an area of active research for the project.
About twenty different transformations or scenes have been developed. Some of these perform structural edits to the hand’s archetypal form; others endow the hand with new dimensions of plasticity; and others imbue the hand with a kind of autonomy, whose resulting behavior is a dynamic negotiation between visitor and algorithm. The videos here present live demonstrations of ten of these scenes:
- Plus One: The hand obtains an additional finger.
- Minus One: The hand has one finger omitted.
- Variable Finger Length: The fingers’ length changes over time.
- Meandering Fingers: The fingers take on a life of their own.
- Procrustes: All fingers are made the same length.
- Lissajous: The palm is warped in a periodic way.
- Breathing Palm: The palm inflates and deflates.
- Vulcan Salute: The third and fourth fingers are cleaved.
- Angular Exaggeration: Finger adduction and abduction angles are amplified.
- Springers: Finger movements are exaggerated by bouncy simulated physics.
Many more transformations are planned. The sketches [see original post] depict some of the current scenes, as well as some as-yet-unfinished scenes conceived during the project’s proposal phase.
About
The hand is a critical interface to the world, allowing the use of tools, the intimate sense of touch, and a vast range of communicative gestures. Yet we frequently take our hands for granted, thinking with them, or through them, but hardly ever about them. Our investigation takes a position of exploration and wonder. Can real-time alterations of the hand’s appearance bring about a new perception of the body as a plastic, variable, unstable medium? Can such an interaction instill feelings of defamiliarization, prompt a heightened awareness of our own bodies, or incite a reexamination of our physical identities? Can we provoke simple wonder about the fact that we have any control at all over such a complex structure as the hand?
We know that the interrelations of hand, mind and identity are far from simple. Persons with alien hand syndrome, for example, have hands which move independently of their conscious will, as if they belonged to another person. By contrast, amputees suffering from phantom limb syndrome continue to feel their missing hand as if it were still there; their discomfort is sometimes relieved with a mirror box, which uses the virtual image of their intact hand to trick the mind and retrain the brain. Within this framework, the Augmented Hand Series can be understood as an instrument for probing or muddling embodied cognition, using a ‘direct manipulation’ interface and the suspension of disbelief to further problematize the mind-body problem. We see evidence of our instrument’s powers in the actions of young visitors who, uncertain whether to believe their eyes, peek into the box to double-check what is (not) really happening to their hand.
The Augmented Hand Series is positioned to prompt, like Dr. Seuss (“I wish I had eleven, too!“) an empathetic acceptance of difference, and a recognition that there are many ways to be. Or could have been. In his essay “Eight Little Piggies“, the eminent biologist Stephen Jay Gould considers why we are shaped the way we are, and concludes—based on fascinating fossil evidence from some of the first land animals, some 400 million years ago—that there is nothing special or inevitable, nor even necessarily evolutionarily optimal about having five fingers on each hand. As Walt Disney discovered nearly a century ago, five fingers may not even be ideal for expressive gestural communication. From this perspective, the Augmented Hand Series is a work of participatory, transhuman biology-fiction that allows for the first-person exploration of these concepts.
A wish to elicit and address wonder—our own, and others’—has been the deep common bond between Kyle, Chris and myself as we developed this project. Ultimately, as collaborators, our underlying personal motivations for developing the project are each quite different, but Kyle’s, which arise from his experiences as a lucid dreamer, are perhaps the most poetic [see original post for video]:
A longer discussion of some of the ideas motivating the Augmented Hands Project can be seen in Golan’s presentation at the 2013 Eyeo Festival, beginning at 22’18”.
Recent Progress
Since this post was originally published last year, the team has created several new scenes, including fractal hands (by student Zach Rispoli), swapped thumbs, double thumbs, fewer knuckles, and extra knuckles. [The images and videos that follow show] some of these more recent experiments, straight from the depths of the uncanny valley. –Ed.