[This story from Tech Times explains the benefits, first to researchers and then for training new drivers, of a new mixed reality simulator being developed at Cornell University. The new simulator illustrates differences in and allows comparisons of presence experiences across different technologies. Follow the link at the end of the story to read coverage in the Cornell Chronicle; you can watch the 9:40 minute CHI 2024 presentation on YouTube; and an April 2022 story from Tech Xplore describes some of the team’s earlier work. –Matthew]

[Image: Screenshot from animated gif posted on X by co-author Fanjun (Frank) Bu with side-by-side images of in-lab and on-road (mixed reality) driving simulators.]

Cornell Tech Incorporates Virtual and Mixed Realities in New Driving Simulator

The simulator allows users to use mixed-reality headsets to overlay virtual objects onto real-world surroundings.

By Jace Dela Cruz
June 21, 2024

Cornell Tech has unveiled Portobello, a new driving simulator that integrates virtual and mixed realities. It aims to advance research capabilities in controlled lab settings and real-world driving scenarios.

Developed by researchers at the Jacobs Technion-Cornell Institute, led by associate professor Wendy Ju, Portobello was recently presented at the ACM Conference on Human Factors in Computing Systems (CHI), where it received recognition for its innovative approach.

Portobello: A New Driving Simulator Integrating Virtual and Mixed Realities

Portobello allows users, including drivers and passengers, to utilize mixed-reality headsets to overlay virtual objects onto their real-world surroundings.

This feature enables comprehensive user studies, bridging the gap between laboratory conditions and practical driving environments – a concept termed “platform portability” by the research team.

The project builds upon the team’s previous work with XR-OOM, an XR driving simulator that faced challenges in accurately placing virtual elements relative to physical objects.

According to doctoral student Fanjun Bu, XR-OOM struggled to precisely stage virtual events in real-world contexts, such as simulating a virtual pedestrian crossing at an actual crosswalk.

Portobello was designed to map and integrate virtual objects within the physical environment to address these challenges. Researchers meticulously map the study area and strategically position virtual elements based on real-world features.

This approach ensures that virtual scenarios, like virtual pedestrians near stop signs, align seamlessly with physical surroundings. Bu emphasized that Portobello facilitates consistent study conditions across diverse settings, from controlled lab environments to authentic road conditions.

Participants perceive lab simulations as visual approximations, whereas on-road simulations are experienced as functional approximations, eliciting a more genuine response from users.

‘Twinning of Studies’

The system’s ability to conduct identical studies in varied environments enhances the practical relevance of research findings. This capability, known as “twinning of studies,” allows researchers to observe how different environmental factors influence study outcomes, providing insights crucial for autonomous driving and human-machine interaction research.

According to Ju, Portobello’s flexibility in accommodating real-world variability enhances the practical relevance of research outcomes. By incorporating some environmental variability into studies, the system ensures that findings are better suited to real-world applications.

In addition to its application in academic research, Portobello addresses broader industry needs by advancing the capabilities of driving simulation technology. Integrating virtual and mixed realities is expected to enhance the precision of research methodologies and support innovations in autonomous vehicle development.

“We believe that by going beyond running pristine studies and allowing some variability from real-world to bleed through, research results will be more applicable to real-world settings,” Ju said in a statement.