[A story from the American Alliance of Museums describes efforts not only to create presence experiences but to develop an evaluation tool to measure the impacts of those and other experiences, including whether they actually lead to visitor learning. See the original story for two more images. –Matthew]
[Image: The dazzling effects of augmented and virtual reality are obvious, but how can museums know if they actually facilitate learning? Credit: Courtesy of Chicago History Museum]
Evaluating the Impact of Augmented and Virtual Reality
By Colleen McGaughey & John Russick [see bios at end below]
January 15, 2021
“I am perfectly astonished at the effect.”
“The thrilling scenes therein depicted are beyond description.”
“It is true and really wonderful.”
These are reactions from visitors to the Chicago Fire Cyclorama, who, in 1892, were captivated by the sensation of standing in the center of a rotunda, surrounded on all sides by a detailed painting of a burning city. Perched atop slowly spinning platforms, audiences got a 360-degree view and a narrated tour of the already historic 1871 Chicago Fire. Two tons of oil paint, six tons of canvas, and twenty person-years of labor resulted in scenes so massive and so realistically rendered that crowds delighted at the feeling of being transported back in time. Cycloramas were as close to virtual reality as you could get in 1892, and the novelty left an impression.
Apart from the quaint turns of phrase, the sentiments from the Chicago Fire Cyclorama visitors are strikingly similar to those offered by recent users of the Chicago History Museum’s virtual and augmented reality project, Chicago00 (“Chicago zero zero”), which we developed as a new kind of museum exhibition: mobile, immersive, and driven by the choices of the user. The project elevates our mission to share Chicago’s stories by activating them on the digital landscape.
“It felt like I could reach out and touch everything around me.”
“The experience took me back in time.”
“I loved it, magnificent view into what it was like back then.”
Like the creators of cycloramas, the team behind CHM’s experimental digital history initiative aims to astonish. Only instead of tons of paint and twenty thousand square feet of canvas, we leverage new technology, images from CHM’s extensive Prints and Photographs Collection, and other media archives to connect users with pivotal events in Chicago’s past. Through augmented reality apps and virtual reality videos, the project superimposes historical imagery onto the contemporary landscape. Stunning visuals accompany ambient sound, music, and narration to accomplish immersive storytelling that grounds users in the history all around them. At the edge of the Chicago River between Clark and LaSalle Streets, for example, smartphones become portals to the past, resurrecting ghostly scenes from when the S.S. Eastland capsized on that very spot over a century ago. The AR is a moving experience for users, inducing gasps and exclamations that satisfy both the historians and the creative team behind the experience.
But Chicago00 was always supposed to be more than a spectacle. It was meant to take the simulated reality that has been delighting gamers and amusement park goers for years and apply it to learning and connecting with history in new ways. As a history museum, our goal from the outset was to explore digital storytelling as a means to deliver on our mission. Our partnership with filmmaker and professor Geoffrey Alan Rhodes, which began in 2013, has pushed us to take the stories at our fingertips—the 1929 St. Valentine’s Day Massacre, the 1933 Century of Progress World’s Fair, the 1968 Democratic National Convention protests—and revitalize them with new interpretation and innovative presentation for an expanded audience. Designed as a series of experiments, each 00 initiative has been an evolution, allowing us to test, iterate, and refine. When the project was honored with a MUSE award from AAM in 2018, CHM and our partners were thrilled with how far we had come and how much we had learned.
The only thing was, even as we watched the people engaging with our VR and AR experiences gasp, we could not tell you for certain if they were learning anything about history. Museum AR and VR is relatively new, and very little has been developed and shared on evaluating the effectiveness of these formats in achieving experience goals for audiences. Analytics tell a nebulous story of views, clicks, and likes, but it’s difficult to draw meaningful conclusions from these numbers. According to museum evaluator Kate Haley Goldman, “Museums lack tools, scales, and even a vocabulary for measuring immersion, presence, and empathy, and how those elements might contribute to learning outcomes.” With the support of a National Endowment for the Humanities Digital Projects for the Public grant, CHM set out to change that. We used the development of our latest VR experience, Chicago00: 1893 World’s Columbian Exposition, to create and test an evaluation tool to measure the cognitive and affective impact of these emerging digital experiences.
Our Process
Working alongside Haley Goldman and her team at HG&Co, we embarked on a process to help us understand how users engage with digital history experiences, what impressions they had of our experience, and what was sinking in, with the goal of integrating this user testing into future design development.
As a test case, HG&Co evaluators observed about fifty participants trying out a previous Chicago00 VR project, Century of Progress, a vivid portrayal of the 1933 World’s Fair that culminates with a simulated trip up the 628-foot Sky Ride to get a bird’s eye view of the city. Afterwards, they interviewed the participants—who included students on a field trip, museum educators, and general museum visitors—about the experience.
While participants were inside the headset, the evaluators observed them pointing, exclaiming, and even grimacing, as they were brought high up in the Sky Ride, then cajoling friends and family members to try it out next. The wow factor was in full effect. The ensuing interviews confirmed the experience achieved powerful sensory immersion for the participants—so much so that, in some cases, they had trouble retaining or reflecting on the ideas being communicated. At times, they even actively tuned out the audio content in order to focus on being “there.” We learned that multiple viewings were warranted to fully grasp the historical information, which supported our approach to make the experiences free, easily accessible, and appealing to return to.
The disconnect between the sensory elements and the narration was a key finding for us. We also discovered that:
- Using place-based material works well. Historic images with extant landmarks alongside contemporary drone photography of the site grounded users in Chicago. They felt they were really “there.”
- Viewers noticed the everyday scenery. While they were not often able to retain narrated concepts about history with a capital “H,” they recalled small moments and attributes in the VR: the hats people wore, the scenic details of the buildings. They assumed the point of view of a 1933 World’s Fair visitor, observing the splendor of the structures and the bustle of faces and fashion in their path.
- Blending the past with the present prompted historical awareness. The VR’s historic images appear in black and white and are superimposed over present-day color visuals. Users gained a sense of the history that happened in places familiar to them today.
- Creating physical sensations works. The culminating experience of going up in the air impressed and at times spooked participants. The experience felt very real, very new and exciting, just as it did for those on the Sky Ride in the 1930s. The novelty of that moment and the intensity of “being there” led to a form of embodiment experienced by the visitors.
These findings—some surprising and some expected—became a formative study for our new World’s Columbian Exposition VR. We considered the complexity and pacing of the narration, the then-and-now images working together to say “history happened here,” and the drama of the moving Ferris wheel. Realizing that the experience would prompt curiosity about the 1893 fair, we developed a web portal with mapped imagery and interpretive content. Available now, Chicago 00: 1893 World’s Columbian Exposition is also the first of our digital history projects to be thoroughly evaluated with our new Experience Evaluation Rubric.
The Experience Evaluation Rubric (EER)
Based on the user testing for Century of Progress, HG&Co developed a framework for impact measurement for museum VR/AR called the Experience Evaluation Rubric (EER). The EER was designed to build a shared vocabulary and specificity around mixed reality experiences. At CHM, we use it to assess and inform the episodes we create of historical events in specific physical locations, but the tool is easily applied to other types of museum-based content and a range of experience goals.
The EER has two sections. The first is for the designer or evaluator to complete, and it captures details about the physical and digital environment of the experience and the content presented. The second is a participant-facing assessment that asks users to score twenty or so statements related to presence factors as well as cognitive and affective impacts. Using a scale of 1 (not at all) to 7 (completely), audiences rate qualities ranging from vividness and captivation to time-travel and physicality of place. These latter two elements were of particular interest to our history-driven, location-based project, but other museums might place more emphasis on traits related to agency or social presence, for example, depending on their intended effects.
Over time, data from the two sections of the EER can be analyzed for emergent patterns, such as what type and context of experience provides what type of impact. Museum digital producers can then make informed decisions about the kinds of content, technology, and settings that best support a desired outcome. For example, after viewing our World’s Columbian Exposition episode, 81 percent of test subjects indicated they wanted to visit the Museum’s World’s Fair exhibition content, which demonstrates VR’s potential to support our on-site audience development goal. We also discovered that 87 percent of test subjects were very interested in the details within the virtual world, which indicates that the design of the experience succeeds at piquing curiosity about historical visuals. Finally, over 80 percent reported learning new facts, affirming the educational effectiveness we sought as we designed the experience.
As the Chicago History Museum works toward its commitment to becoming a digital-first museum, projects like Chicago00 are at the top of our strategic agenda. With support from the National Endowment for the Humanities, which encourages new, technology-driven interpretive methods for museums and widespread sharing of resulting products and tools, we have been able to meaningfully advance this goal. We now have evidence of the efficacy and appeal of the mixed-reality work we are developing through the Chicago00 project and a pathway for designing future experiences with our desired outcomes in mind. Our Experience Evaluation Rubric offers clear guidance for how all museums experimenting with augmented and virtual reality can understand user engagement and deliver meaningful content.
Immersive storytelling has come a long way since the peak days of cyclorama exhibitions. Worlds displayed on building-size paintings have shrunk to fit inside screens and headsets, but the potential is bigger and the reach is farther than ever before. We see AR and VR as tools that will greatly influence the future of our museum—and probably yours, too.
—
About the authors
Colleen McGaughey is the senior manager of institutional advancement at the Chicago History Museum, where she oversees government, foundation, and corporate grants. During her more than 20-year career in nonprofit development, she has played a key role in diverse grant-funded initiatives ranging from HIV prevention and hunger relief to adult literacy instruction and museum exhibitions. She has authored hundreds of proposals grounded in research and driven by mission. Her practice involves program planning and evaluation strategies as essential phases of fundraising. She has a BA in philosophy from Loyola University Chicago and an MA in liberal studies from Northwestern University and insists both prove useful in her work.
John Russick is the senior vice president of the Chicago History Museum where he oversees the departments of collections and curatorial affairs, exhibitions, communications, and the Museum’s Research Center. His latest digital initiative, the Chicago 00 Project, is a collection of interactive augmented and virtual reality experiences that showcase the Museum’s film, photo, and sound archive, winning a MUSE award in 2018 and a Chicago Innovation award in 2019. His most recent publication, “The Museum Inside Out” was published in the January/February 2020 issue of the American Alliance of Museum’s magazine. Other publications include, “A Place for Everything: Museum Collections, Technology, and the Power of Place,”(Medium, 2014) and “Connecting Kids to History with Museum Exhibitions” (Left Coast Press, 2010). He served as a program advisor on the 2011 Florentine Films documentary, Prohibition, directed by Ken Burns and Lynn Novick. Since 2008, he has organized the American Alliance of Museum’s annual Excellence in Exhibition Label Writing Competition.