Papers by Gyanendra Sharma
We describe interfaces and visualizations in the CRAIVE (Colaborative Research Augmented Immersiv... more We describe interfaces and visualizations in the CRAIVE (Colaborative Research Augmented Immersive Virtual Environment) Lab, an interactive human scale immersive environment at Rensselaer Polytechnic Institute. We describe the physical infrastructure and software architecture of the CRAIVE-Lab, and present two immersive scenarios within it. The first is “person folowing”, which alows a person walking inside the immersive space to be tracked by simple objects on the screen. This was implemented as a proof of concept of the overal system, which includes visual tracking from an overhead array of cameras, communication of the tracking results, and large-scale projection and visualization. The second “smart presentation” scenario features multimedia on the screen that reacts to the position of a person walking around the environment by playing or pausing automaticaly, and additionaly supports real-time speech-to-text transcription. Our goal is to continue research in natural human intera...
Large-scale immersive environments can convey a large amount of information to many participants ... more Large-scale immersive environments can convey a large amount of information to many participants simultaneously. When equipped with occupant-awareness, not only does the ability of the participants to interact with the environment increase dramatically, but also the environment's understanding of its audience. Using ceiling-mounted time-of-flight distance sensors, we discuss how person tracking, body orientation estimation, and pointing gesture recognition can augment an immersive environment to become simultaneously aware of all its participants. The testbed for our exploration is a large circular space 12 m in diameter enclosed by a 360 degree display 4.3 meters high, which can comfortably house more than 30 people at a time. This large-scale occupant-aware immersive environment is an ideal space to simulate different "living lab" scenarios or conduct group behavioral studies.
Permission to make digital or hard copies of all or part of this work for personal or classroom u... more Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, contact the Owner/Author. Request permissions from [email protected]. The research in this paper was partially supported by the US National Science Foundation under awards CNS-1229391 and IIP-1631674, and by IBM Research via Rensselaer’s Cognitive and Immersive Systems Laboratory. Living Labs: Measuring Human Experience in the Built Environment, in association with CHI’18., April 22, 2018, Montréal, Canada. Copyright © 2018 held by Owner/Author. Abstract Large display environments supporting fluid user inter...
As part of the final project for this course, I used CRAIVE (Collaborative-Research Augmented Imm... more As part of the final project for this course, I used CRAIVE (Collaborative-Research Augmented Immersive Virtual Environment Laboratory) space to create an interactive immersive visualizations based on human motion inside the space. As shown in Figure 1, CRAIVE lab has a 360 degree screen equipped with 8 projectors along with 6 overhead cameras used for tracking. The floor space enclosed within the screen is rectangular with curved corners and has a length of approximately 12 meters and width of 10 meters. The screen is approximately 5 meters in height.
We present a system that utilizes a mid-level score representation for aligning printed music to ... more We present a system that utilizes a mid-level score representation for aligning printed music to its audio rendition. The mid-level representation is designed to capture an approximation to the musical events present in the printed score. It consists of a template based note detection frontend that seeks to detect notes without regard to musical duration, accidentals or the key signature. The presented method is designed for the commonly used grand staff and the approach is extendable to other types of scores. The image processing consists of page segmentation into lines followed by multiple stages that optimally orient the lines and establish a reference grid to be used in the note identification stage. Both the audio and the printed score are converted into compatible frequency representations. Alignment is performed using dynamic time warping with a specially designed distance measure. The insufficient pitch resolution due to the reductive nature of the mid-level representation i...
In this paper, we present a multi-user interaction interface for a large immersive space that sup... more In this paper, we present a multi-user interaction interface for a large immersive space that supports simultaneous screen interactions by combining (1) user input via personal smartphones and Bluetooth microphones, (2) spatial tracking via an overhead array of Kinect sensors, and (3) WebSocket interfaces to a webpage running on the large screen. Users are automatically, dynamically assigned personal and shared screen sub-spaces based on their tracked location with respect to the screen, and use a webpage on their personal smartphone for touchpad-type input. We report user experiments using our interaction framework that involve image selection and placement tasks, with the ultimate goal of realizing display-wall environments as viable, interactive workspaces with natural multimodal interfaces.
Uploads
Papers by Gyanendra Sharma