We propose a set of techniques leveraging mobile devices as lenses to explore, interact and annot... more We propose a set of techniques leveraging mobile devices as lenses to explore, interact and annotate n-dimensional data visualizations. The democratization of mobile devices, with their arrays of integrated sensors, opens up opportunities to create experiences for anyone to explore and interact with large information spaces anywhere. In this paper, we propose to revisit ideas behind the Chameleon prototype of Fitzmaurice et al. initially envisioned in the 90s for navigation, before spatially-aware devices became mainstream. We also take advantage of other input modalities such as pen and touch to not only navigate the space using the mobile as a lens, but interact and annotate it by adding toolglasses.
IEEE Transactions on Visualization and Computer Graphics
Fig. 1. Interaction techniques for authoring presentations in mobile VR. a) 3D object manipulatio... more Fig. 1. Interaction techniques for authoring presentations in mobile VR. a) 3D object manipulation technique, b) occlusion handling, c) animations with time represented through height, d) working across slides and displaying different content resources Abstract-Virtual Reality (VR) has the potential to support mobile knowledge workers by complementing traditional input devices with a large three-dimensional output space and spatial input. Previous research on supporting VR knowledge work explored domains such as text entry using physical keyboards and spreadsheet interaction using combined pen and touch input. Inspired by such work, this paper probes the VR design space for authoring presentations in mobile settings. We propose PoVRPoint-a set of tools coupling penand touch-based editing of presentations on mobile devices, such as tablets, with the interaction capabilities afforded by VR. We study the utility of extended display space to, for example, assist users in identifying target slides, supporting spatial manipulation of objects on a slide, creating animations, and facilitating arrangements of multiple, possibly occluded shapes or objects. Among other things, our results indicate that 1) the wide field of view afforded by VR results in significantly faster target slide identification times compared to a tablet-only interface for visually salient targets; and 2) the three-dimensional view in VR enables significantly faster object reordering in the presence of occlusion compared to two baseline interfaces. A user study further confirmed that the interaction techniques were found to be usable and enjoyable.
Virtual and Augmented Reality (VR and AR) have the potential to change information work. The abil... more Virtual and Augmented Reality (VR and AR) have the potential to change information work. The ability to modify the workers senses can transform everyday environments into a productive office, using portable head-mounted displays (HMDs) combined with conventional interaction devices, such as keyboards and tablets. While a stream of better, cheaper and lighter HMDs have been introduced for consumers in recent years, there are still many challenges to be addressed to allow this vision to become reality. This chapter summarizes the state of the art in the field of extended reality for knowledge work in everyday environments and proposes steps to address the open challenges.
Structure-from-motion and multi-view stereo can be used to create 3D models from image sequences,... more Structure-from-motion and multi-view stereo can be used to create 3D models from image sequences, but are not widely adopted by consumers. We study how to address two challenges of such systems for non-professional users: 1) their technical complexity, and, 2) the computational demand needed for processing. To this end, we embed an open-source pipeline in a scalable cloud environment and create a user interface aimed at non-professional users of photogrammetry systems. Finally, we evaluate both the cloud-based infrastructure and the user interface and demonstrate its usability.
Abstract. This poster and accompanying demo shows how Semantic Web and Linked Data technologies a... more Abstract. This poster and accompanying demo shows how Semantic Web and Linked Data technologies are incorporated into an Augmented Reality platform in the SmartReality project and form the basis for enhanced Augmented Reality mobile applications in which information and content in the user’s surroundings can be presented in a more meaningful and useful manner. We describe how things of interest are described semantically and linked into the Linked Open Data cloud. Relevant metadata about things is collected and processed in order to identify and retrieve related content and services on the Web. Finally, the user sees this information intuitively in their Smart Reality view of the reality around them.
The use of Augmented Reality for overlaying visual information on print media like street posters... more The use of Augmented Reality for overlaying visual information on print media like street posters has become widespread over the last few years. While this user interface metaphor represents an instance of cross-media information spaces the specific context of its use has not yet been carefully studied, resulting in productions generally relying on trial-and-error approaches. In this paper, we explicitly consider mobile contexts in the consumption of augmented print media. We explore the design space of hybrid user interfaces for augmented posters and describe different case studies to validate our approach. Outcomes of this work inform the design of future interfaces for publicly accessible augmented print media in mobile contexts. Author Keywords augmented reality; poster; print media; cross-media; hybrid user interface; mobile physical interaction; public displays
We introduce two new interaction techniques for creating and interacting with non-photorealistic ... more We introduce two new interaction techniques for creating and interacting with non-photorealistic images using stroke-based rendering. We provide bimanual control of a large interactive canvas through both remote pointing and direct touch. Remote pointing allows people to sit and interact at a distance with an overview of the entire display, while direct-touch interaction provides more precise control. We performed a user study to compare these two techniques in both a controlled setting with constrained tasks and an exploratory setting where participants created their own painting. We found that, although the direct-touch interaction outperformed remote pointing, participants had mixed preferences and did not consistently choose one or the other to create their own painting. Some participants also chose to switch between techniques to achieve different levels of precision and control for different tasks.
2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), 2017
Smartphones are a popular device class for mobile Augmented Reality but suffer from a limited inp... more Smartphones are a popular device class for mobile Augmented Reality but suffer from a limited input space. Around-device interaction techniques aim at extending this input space using various sensing modalities. In this paper we present our work towards extending the input area of mobile devices using front-facing device-centered cameras that capture reflections in the cornea. As current generation mobile devices lack high resolution front-facing cameras, we study the feasibility of around-device interaction using corneal reflective imaging based on a high resolution camera. We present a workflow, a technical prototype and a feasibility evaluation.
An increasing number of consumer-oriented head-mounted displays (HMD) for augmented and virtual r... more An increasing number of consumer-oriented head-mounted displays (HMD) for augmented and virtual reality (AR/VR) are capable of finger and hand tracking. We report on the accuracy of off-theshelf VR and AR HMDs when used for touch-based tasks such as
2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2019
Text entry is a challenge for Virtual Reality (VR) applications. In the context of immersive VR H... more Text entry is a challenge for Virtual Reality (VR) applications. In the context of immersive VR Head-Mounted Displays, text entry has been investigated for standard physical keyboards as well as for various hand representations. Specifically, prior work has indicated that minimalistic fingertip visualizations is an efficient hand representation. However, these representations typically require external tracking systems. Touch-sensitive physical keyboards allow for on-surface interaction, with sensing integrated into the keyboard itself. However, they have not been thoroughly investigated within VR. We close this gap by comparing text entry on a standard physical keyboard and a touch-sensitive physical keyboard in a controlled user study (n=26). Our results indicate that text entry using touch-sensitive physical keyboards can be as efficient as the fingertip visualization, but that results vary between experienced and inexperienced typists.
Creating and maintaining augmented scenes for mobile Augmented Reality browsers can be a challeng... more Creating and maintaining augmented scenes for mobile Augmented Reality browsers can be a challenging and time consuming task. The timeliness of digital information artifacts connected to changing urban environments require authors to constantly update the structural representations of augmented scenes or to accept that the information will soon be outdated. We present an approach for retrieving multimedia content and relevant web services for mobile Augmented Reality applications at runtime. Using semantic web technologies we are able to postpone the retrieval of actual media items to the moment a user actually perceives an augmented scene. This allows content creators to augment a scene only once and avoid continous manual updates. We also discuss the tradeoff between runtime content retrieval using Linked Data concepts and decreased control over the scene appearance at the time of authoring that comes along with this approach.
Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, 2015
Body proximate display environments can be formed by combining multiple hand-held, head-mounted, ... more Body proximate display environments can be formed by combining multiple hand-held, head-mounted, wrist-worn or other displays. Wearable displays such as smartwatches and smartglasses have the potential to extend the interaction capabilities of mobile users beyond a single display. However, the display ecosystem formed by multiple personal displays on and around users' bodies is not well understood, yet. Within this paper, we investigate the design and technology challenges that could inhibit the creation and the use of interactive display ecosystems.
Digitalization offers chances as well as risks for industrial companies. This article describes h... more Digitalization offers chances as well as risks for industrial companies. This article describes how the area of Mixed Reality, with its manifestations Augmented and Virtual Reality, can support industrial applications in the age of digitalization. Starting from a historical perspective on Augmented and Virtual Reality, this article surveys recent developments in the domain of Mixed Reality, relevant for industrial use cases. --- Die Digitalisierung bietet f\"ur Industrieunternehmen neue Chancen, stellt diese jedoch auch vor Herausforderungen. Dieser Artikel beleuchtet wie das Gebiet der vermischten Realit\"at mit seinen Auspr\"agungen der erweiterten Realit\"at und der virtuellen Realit\"at f\"ur industriellen Anwendungen im Zeitalter der Digitalisierung Vorteile schaffen kann. Ausgehend von einer historischen Betrachtung, werden aktuelle Entwicklungen auf dem Gebiet der erweiterten und virtuellen Realit\"at diskutiert.
The SmartReality project is a new Austrian national project dedicated to exploring how people can... more The SmartReality project is a new Austrian national project dedicated to exploring how people can access smarter information about the things around them through a combination of Augmented Reality, Web services and semantic technologies. This paper introduces the project and our initial thoughts concerning the current and necessary state of AR standards, interfaces and platforms.
This deliverable updates the scenarios and requirements described in D2.1.2 to include on-demand ... more This deliverable updates the scenarios and requirements described in D2.1.2 to include on-demand features identified during connectivity phase development, the usage by driving experiments and experiments funded through the open calls. The document provides a description of the functionality that needs to be developed within the expansion phase, to be included in the V2.0 release.
We present a demonstration of HeadPhones (Headtracking + smart-Phones), a novel approach for the ... more We present a demonstration of HeadPhones (Headtracking + smart-Phones), a novel approach for the spatial registration of multiple mobile devices into an ad hoc multi-display environment. We propose to employ the user's head as external reference frame for the registration of multiple mobile devices into a common coordinate system. Our approach allows for dynamic repositioning of devices during runtime without the need for external infrastructure such as separate cameras or fiducials. Specifically, our only requirements are local network connections and mobile devices with built-in front facing cameras. This way, HeadPhones enables spatially-aware multi-display applications in mobile contexts.
We propose a set of techniques leveraging mobile devices as lenses to explore, interact and annot... more We propose a set of techniques leveraging mobile devices as lenses to explore, interact and annotate n-dimensional data visualizations. The democratization of mobile devices, with their arrays of integrated sensors, opens up opportunities to create experiences for anyone to explore and interact with large information spaces anywhere. In this paper, we propose to revisit ideas behind the Chameleon prototype of Fitzmaurice et al. initially envisioned in the 90s for navigation, before spatially-aware devices became mainstream. We also take advantage of other input modalities such as pen and touch to not only navigate the space using the mobile as a lens, but interact and annotate it by adding toolglasses.
IEEE Transactions on Visualization and Computer Graphics
Fig. 1. Interaction techniques for authoring presentations in mobile VR. a) 3D object manipulatio... more Fig. 1. Interaction techniques for authoring presentations in mobile VR. a) 3D object manipulation technique, b) occlusion handling, c) animations with time represented through height, d) working across slides and displaying different content resources Abstract-Virtual Reality (VR) has the potential to support mobile knowledge workers by complementing traditional input devices with a large three-dimensional output space and spatial input. Previous research on supporting VR knowledge work explored domains such as text entry using physical keyboards and spreadsheet interaction using combined pen and touch input. Inspired by such work, this paper probes the VR design space for authoring presentations in mobile settings. We propose PoVRPoint-a set of tools coupling penand touch-based editing of presentations on mobile devices, such as tablets, with the interaction capabilities afforded by VR. We study the utility of extended display space to, for example, assist users in identifying target slides, supporting spatial manipulation of objects on a slide, creating animations, and facilitating arrangements of multiple, possibly occluded shapes or objects. Among other things, our results indicate that 1) the wide field of view afforded by VR results in significantly faster target slide identification times compared to a tablet-only interface for visually salient targets; and 2) the three-dimensional view in VR enables significantly faster object reordering in the presence of occlusion compared to two baseline interfaces. A user study further confirmed that the interaction techniques were found to be usable and enjoyable.
Virtual and Augmented Reality (VR and AR) have the potential to change information work. The abil... more Virtual and Augmented Reality (VR and AR) have the potential to change information work. The ability to modify the workers senses can transform everyday environments into a productive office, using portable head-mounted displays (HMDs) combined with conventional interaction devices, such as keyboards and tablets. While a stream of better, cheaper and lighter HMDs have been introduced for consumers in recent years, there are still many challenges to be addressed to allow this vision to become reality. This chapter summarizes the state of the art in the field of extended reality for knowledge work in everyday environments and proposes steps to address the open challenges.
Structure-from-motion and multi-view stereo can be used to create 3D models from image sequences,... more Structure-from-motion and multi-view stereo can be used to create 3D models from image sequences, but are not widely adopted by consumers. We study how to address two challenges of such systems for non-professional users: 1) their technical complexity, and, 2) the computational demand needed for processing. To this end, we embed an open-source pipeline in a scalable cloud environment and create a user interface aimed at non-professional users of photogrammetry systems. Finally, we evaluate both the cloud-based infrastructure and the user interface and demonstrate its usability.
Abstract. This poster and accompanying demo shows how Semantic Web and Linked Data technologies a... more Abstract. This poster and accompanying demo shows how Semantic Web and Linked Data technologies are incorporated into an Augmented Reality platform in the SmartReality project and form the basis for enhanced Augmented Reality mobile applications in which information and content in the user’s surroundings can be presented in a more meaningful and useful manner. We describe how things of interest are described semantically and linked into the Linked Open Data cloud. Relevant metadata about things is collected and processed in order to identify and retrieve related content and services on the Web. Finally, the user sees this information intuitively in their Smart Reality view of the reality around them.
The use of Augmented Reality for overlaying visual information on print media like street posters... more The use of Augmented Reality for overlaying visual information on print media like street posters has become widespread over the last few years. While this user interface metaphor represents an instance of cross-media information spaces the specific context of its use has not yet been carefully studied, resulting in productions generally relying on trial-and-error approaches. In this paper, we explicitly consider mobile contexts in the consumption of augmented print media. We explore the design space of hybrid user interfaces for augmented posters and describe different case studies to validate our approach. Outcomes of this work inform the design of future interfaces for publicly accessible augmented print media in mobile contexts. Author Keywords augmented reality; poster; print media; cross-media; hybrid user interface; mobile physical interaction; public displays
We introduce two new interaction techniques for creating and interacting with non-photorealistic ... more We introduce two new interaction techniques for creating and interacting with non-photorealistic images using stroke-based rendering. We provide bimanual control of a large interactive canvas through both remote pointing and direct touch. Remote pointing allows people to sit and interact at a distance with an overview of the entire display, while direct-touch interaction provides more precise control. We performed a user study to compare these two techniques in both a controlled setting with constrained tasks and an exploratory setting where participants created their own painting. We found that, although the direct-touch interaction outperformed remote pointing, participants had mixed preferences and did not consistently choose one or the other to create their own painting. Some participants also chose to switch between techniques to achieve different levels of precision and control for different tasks.
2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), 2017
Smartphones are a popular device class for mobile Augmented Reality but suffer from a limited inp... more Smartphones are a popular device class for mobile Augmented Reality but suffer from a limited input space. Around-device interaction techniques aim at extending this input space using various sensing modalities. In this paper we present our work towards extending the input area of mobile devices using front-facing device-centered cameras that capture reflections in the cornea. As current generation mobile devices lack high resolution front-facing cameras, we study the feasibility of around-device interaction using corneal reflective imaging based on a high resolution camera. We present a workflow, a technical prototype and a feasibility evaluation.
An increasing number of consumer-oriented head-mounted displays (HMD) for augmented and virtual r... more An increasing number of consumer-oriented head-mounted displays (HMD) for augmented and virtual reality (AR/VR) are capable of finger and hand tracking. We report on the accuracy of off-theshelf VR and AR HMDs when used for touch-based tasks such as
2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2019
Text entry is a challenge for Virtual Reality (VR) applications. In the context of immersive VR H... more Text entry is a challenge for Virtual Reality (VR) applications. In the context of immersive VR Head-Mounted Displays, text entry has been investigated for standard physical keyboards as well as for various hand representations. Specifically, prior work has indicated that minimalistic fingertip visualizations is an efficient hand representation. However, these representations typically require external tracking systems. Touch-sensitive physical keyboards allow for on-surface interaction, with sensing integrated into the keyboard itself. However, they have not been thoroughly investigated within VR. We close this gap by comparing text entry on a standard physical keyboard and a touch-sensitive physical keyboard in a controlled user study (n=26). Our results indicate that text entry using touch-sensitive physical keyboards can be as efficient as the fingertip visualization, but that results vary between experienced and inexperienced typists.
Creating and maintaining augmented scenes for mobile Augmented Reality browsers can be a challeng... more Creating and maintaining augmented scenes for mobile Augmented Reality browsers can be a challenging and time consuming task. The timeliness of digital information artifacts connected to changing urban environments require authors to constantly update the structural representations of augmented scenes or to accept that the information will soon be outdated. We present an approach for retrieving multimedia content and relevant web services for mobile Augmented Reality applications at runtime. Using semantic web technologies we are able to postpone the retrieval of actual media items to the moment a user actually perceives an augmented scene. This allows content creators to augment a scene only once and avoid continous manual updates. We also discuss the tradeoff between runtime content retrieval using Linked Data concepts and decreased control over the scene appearance at the time of authoring that comes along with this approach.
Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, 2015
Body proximate display environments can be formed by combining multiple hand-held, head-mounted, ... more Body proximate display environments can be formed by combining multiple hand-held, head-mounted, wrist-worn or other displays. Wearable displays such as smartwatches and smartglasses have the potential to extend the interaction capabilities of mobile users beyond a single display. However, the display ecosystem formed by multiple personal displays on and around users' bodies is not well understood, yet. Within this paper, we investigate the design and technology challenges that could inhibit the creation and the use of interactive display ecosystems.
Digitalization offers chances as well as risks for industrial companies. This article describes h... more Digitalization offers chances as well as risks for industrial companies. This article describes how the area of Mixed Reality, with its manifestations Augmented and Virtual Reality, can support industrial applications in the age of digitalization. Starting from a historical perspective on Augmented and Virtual Reality, this article surveys recent developments in the domain of Mixed Reality, relevant for industrial use cases. --- Die Digitalisierung bietet f\"ur Industrieunternehmen neue Chancen, stellt diese jedoch auch vor Herausforderungen. Dieser Artikel beleuchtet wie das Gebiet der vermischten Realit\"at mit seinen Auspr\"agungen der erweiterten Realit\"at und der virtuellen Realit\"at f\"ur industriellen Anwendungen im Zeitalter der Digitalisierung Vorteile schaffen kann. Ausgehend von einer historischen Betrachtung, werden aktuelle Entwicklungen auf dem Gebiet der erweiterten und virtuellen Realit\"at diskutiert.
The SmartReality project is a new Austrian national project dedicated to exploring how people can... more The SmartReality project is a new Austrian national project dedicated to exploring how people can access smarter information about the things around them through a combination of Augmented Reality, Web services and semantic technologies. This paper introduces the project and our initial thoughts concerning the current and necessary state of AR standards, interfaces and platforms.
This deliverable updates the scenarios and requirements described in D2.1.2 to include on-demand ... more This deliverable updates the scenarios and requirements described in D2.1.2 to include on-demand features identified during connectivity phase development, the usage by driving experiments and experiments funded through the open calls. The document provides a description of the functionality that needs to be developed within the expansion phase, to be included in the V2.0 release.
We present a demonstration of HeadPhones (Headtracking + smart-Phones), a novel approach for the ... more We present a demonstration of HeadPhones (Headtracking + smart-Phones), a novel approach for the spatial registration of multiple mobile devices into an ad hoc multi-display environment. We propose to employ the user's head as external reference frame for the registration of multiple mobile devices into a common coordinate system. Our approach allows for dynamic repositioning of devices during runtime without the need for external infrastructure such as separate cameras or fiducials. Specifically, our only requirements are local network connections and mobile devices with built-in front facing cameras. This way, HeadPhones enables spatially-aware multi-display applications in mobile contexts.
Uploads
Papers by Jens Grubert