[One may need to be a power user to fully recognize the benefits, presence-related and otherwise, of interacting with spreadsheets in the 3D space of virtual reality as described in this story from VentureBeat. The original story includes the 6:56 minute video referenced, along with the full research report in ArXiv, at the end. –Matthew]
[Image: The hierarchical pie menu is invoked in-place and navigated from bottom to top through in-air movements of the pen (a-c). An arc menu on the corner of the tablet controls visualization options (activated – solid color, non activated – semi-transparent) (d). Adding a trend-line to a chart through an in-air stroke gesture (e). Bottom row: Three functions with their respective links to their source cells highlighted (f). Abbreviations: O: overview window, D: dependency links, F: nested functions, C: cluster-nodes. S: additional sheets (tabs). From Figure 9 in “Pen-based Interaction with Spreadsheets in Mobile Virtual Reality” available from ArXiv.]
Researchers bring Google Sheets and Microsoft Excel into VR
By Jeremy Horwitz
August 13, 2020
If your dreams occasionally involve spreadsheets that extend endlessly in all directions, good news: Researchers have developed a virtual reality spreadsheet interface that could expand Google Sheets and Microsoft Excel files from flat screens into 3D spaces. Rather than being nightmare-inducing, it could actually make spreadsheet apps more usable than before.
While traditional spreadsheets have been limited by the boundaries of 2D windows and displays, the research team envisions VR opening up adjacent 2D workspaces for related content, then using 3D for everything from floating menus to cell selection and repositioning. In one example, a VR headset mirrors the view of one spreadsheet page displayed on a physical tablet, while two virtual sheets sit to the left and right, permitting drag-and-drop access to their cells, as an overview hovers above.
Alternatively, tablet-surrounding areas could display useful reference materials, expanded views of formulas, or the full collection of a spreadsheet’s pages displayed as floating previews. Another possibility is a single spreadsheet page that stretches far further than the 30-degree diagonal field of view occupied by a typical tablet on a desk, utilizing more of the ~110-degree fields of view supported by VR headsets.
Fans of the film Minority Report and portrayals of similarly holographic future 3D interfaces will appreciate the team’s use of a floating pie menu — complete with a drop shadow on the spreadsheet — for selecting features and functions, as well as spherical rather than flat buttons and other visual elements that appear to leap off the flat pages. The 3D spreadsheet workspace could also be extended with floating desktop objects, such as a virtual trash can, to make disposing of unwanted content more intuitive.
Interestingly, the project’s research team includes members of the Mixed Reality Lab at Germany’s Coburg University, as well as a professor from the University of Cambridge and two principal researchers from Microsoft — mixed reality expert Eyal Ofek and UX engineer Michel Pahud. But their work isn’t limited to potential Microsoft applications: Dr. Jens Grubert, one of the paper’s authors, tells VentureBeat that the cross-organizational team includes “long time collaborators” and actually used Google Sheets rather than Excel for the backend.
In addition to an HTC Vive Pro VR headset and Microsoft Surface Pro 4 tablet, the researchers employed a spatially tracked stylus, enabling precision direct input for spreadsheet interactions while adding the freedom of in-air movement within a 3D space. Unsurprisingly, the VR spreadsheets can be used on existing commodity PC hardware, and the virtual UI was created with the Unity engine. Sheets pages are rendered within Chromium browser windows to match the resolution and size of the Surface Pro 4’s screen.
Full project details are available in the “Pen-based Interaction with Spreadsheets in Mobile Virtual Reality” research paper. If you’re interested in deeper dives, you can see a video of the project here, as well as a broader exploration of the team’s VR-tablet research here, ahead of their presentation at the IEEE’s International Symposium on Mixed and Augmented Reality, being held online from November 9 to 13.