Virtual reality system at RWTH Aachen University
Torsten Kuhlen
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
During the last decade, Virtual Reality (VR) systems have progressed from primary laboratory experiments into serious and valuable tools. Thereby, the amount of useful applications has grown in a large scale, covering conventional use, e.g., in science, design, medicine and engineering, as well as more visionary applications such as creating virtual spaces that aim to act real. However, the high capabilities of today's virtual reality systems are mostly limited to firstclass visual rendering, which directly disqualifies them for immersive applications. For general application, though, VR-systems should feature more than one modality in order to boost its range of applications. The CAVE-like immersive environment that is run at RWTH Aachen University comprises state-of-the-art visualization and auralization with almost no constraints on user interaction. In this article a summary of the concept, the features and the performance of our VR-system is given. The system features a 3D sketching interface that allows controlling the application in a very natural way by simple gestures. The sound rendering engine relies on present-day knowledge of Virtual Acoustics and enables a physically accurate simulation of sound propagation in complex environments, including important wave effects such as sound scattering, airborne sound insulation between rooms and sound diffraction. In spite of this realistic sound field rendering, not only spatially distributed and freely movable sound sources and receivers are supported, but also modifications and manipulations of the environment itself. The auralization concept is founded on pure FIR filtering which is realized by highly parallelized non-uniformly partitioned convolutions. A dynamic crosstalk cancellation system performs the sound reproduction that delivers binaural signals to the user without the need of headphones. The significant computational complexity is handled by distributed computation on PCclusters that drive the simulation in real-time even for huge audio-visual scenarios.
Related papers
Real-Time Virtual Audio Reality
1986
A real-time virtual audio reality model has been created. The system includes model-based soundsynthesizers, geometrical room acoustics modeling, binaural auralization for headphone or loudspeakerlistening, and hiqh-quality animation. This paper discusses the following subsystems of thedesigned environment: The implementation of the audio processing soft- and hardware, and the designof a dedicated multiprocessor DSP hardware platform. The design goal of the overall
Satie: A Live And Scalable 3D Audio Scene Rendering Environment For Large Multi-Channel Loudspeaker Configurations
2017
Recent advances in computing offer the possibility to scale real-time 3D virtual audio scenes to include hundreds of simultaneous sound sources, rendered in realtime, for large numbers of audio outputs. Our Spatial Audio Toolkit for Immersive Environments (SATIE), allows us to render these dense audio scenes to large multi-channel (e.g. 32 or more) loudspeaker systems, in realtime and controlled from external software such as 3D scenegraph software. As we describe here, SATIE is designed for improved scalability: minimum dependency between nodes in the audio DSP graph for parallel audio computation, controlling sound objects by groups and load balancing computation of geometry that allow to reduce the number of messages for controlling simultaneously a high number of sound sources. The paper presents SATIE along with example use case scenarios. Our initial work demonstrates SATIE's flexibility, and has provided us with novel sonic sensations such as ``audio depth of field'&#...
<title>Framework for integrating sound into virtual environment interfaces</title>
Stereoscopic Displays and Virtual Reality Systems III, 1996
Sound represents a largely untapped source of realism in Virtual Environments (VEs). In the real world, sound constantly surrounds us and pulls us into our world. In VEs, sound enhances the immersiveness of the simulation and provides valuable information about the environment. While there has been a growing interest in integrating sound into VE interfaces, current technology has not brought about its widespread use. This, we believe, can be attributed to the lack of proper tools for modeling and rendering the auditory world. We have been investigating some of the problems which we believe are pivotal to the widespread use of sound in VE interfaces. As a result of this work, we have developed the Virtual Audio Server (VAS). VAS is a distributed, real-time spatial sound generation server. It provides high level abstractions for modeling the auditory world and the events which occur in the world. VAS supports both sampled and synthetic sound sources and provides a device independent interface to spatialization hardware. Resource management schemes can easily be integrated into the server to manage the real-time generation of sound. We are currently investigating possible approaches to this important problem.
Virtual Audio Systems
Presence: Teleoperators and Virtual Environments, 2008
To be immersed in a virtual environment, the user must be presented with plausible sensory input including auditory cues. A virtual (three-dimensional) audio display aims to allow the user to perceive the position of a sound source at an arbitrary position in three-dimensional space despite the fact that the generated sound may be emanating from a fixed number of loudspeakers at fixed positions in space or a pair of headphones. The foundation of virtual audio rests on the development of technology to present auditory signals to the listener's ears so that these signals are perceptually equivalent to those the listener would receive in the environment being simulated. This paper reviews the human perceptual and technical literature relevant to the modeling and generation of accurate audio displays for virtual environments. Approaches to acoustical environment simulation are summarized and the advantages and disadvantages of the various approaches are presented.
Accurate and fast audio-realistic rendering of sounds in virtual environments
IEEE 6th Workshop on Multimedia Signal Processing, 2004., 2004
In this paper we propose a novel method for realtime auralization of sounds in complex environments using visibility diagrams. The method accounts for both specular and diffracted reflections with receivers and sources that are free to move. Our solution concentrates in a pre-processing phase all the operations that can be conducted without knowledge of either source or receiver locations. In fact we pre-compute a set of visibility diagrams and diffracted beam trees. Once the source location is specified, we can determine the reflective beam trees through a simple lookup process on the visibility diagrams. The additional knowledge of the receiver location allows us to immediately generate all reflective and diffractive paths that link source and receiver.
EVERTims: Open source framework for real-time auralization in architectural acoustics and virtual reality
2017
This paper presents recent developments of the EVERTims project, an auralization framework for virtual acoustics and real-time room acoustic simulation. The EVERTims framework relies on three independent components: a scene graph editor, a room acoustic modeler, and a spatial audio renderer for auralization. The framework was first published and detailed in previous publications. Recent developments presented here concern the complete redesign of the scene graph editor unit, and the C++ implementation of a new spatial renderer based on the JUCE framework. EVERTims now functions as a Blender add-on to support real-time auralization of any 3D room model, both for its creation in Blender and its exploration in the Blender Game Engine. The EVERTims framework is published as open source software: http://evertims.ircam.fr.
A multi-processor system for the production of virtual sound fields
The Journal of the Acoustical Society of America, 1998
A multi-processor system is described for the production of virtual sound fields using small arrays of up to four high-quality loudspeakers and up to four control microphones distributed around an artificial head and torso in a lightly damped listening room. The digital signal processing engine consists of four floating point processors interconnected for parallel computation, together with digital and analog converters. The performance of the system is analyzed in terms of the tradeoff between the maximum sampling rates that can be used and the complexity of the digital filter matrices in the different operating modes of the system such as system identification, single-input multiple-error adaptive filtering, and full multichannel filtering with fixed coefficients.
Human performance of computational sound models for immersive environments
This paper presents a method for incorporating the expressivity of human performance into real-time computational audio generation for games and other immersive environments. In film, Foley artistry is widely recognised to enrich the viewer's experience, but the creativity of the Foley artist cannot be easily transferred to interactive environments where sound cannot be recorded in advance. We present new methods for human performers to control computational audio models, using a model of a squeaky door as a case study. We focus on the process of selecting control parameters and on the mapping layer between gesture and sound, referring to results from a separate user evaluation study. By recording high-level control parameters rather than audio samples, performances can be later varied to suit the details of the interactive environment.
It sounds real when you see it. Realistic sound source simulation in multimodal virtual environments
Journal on Multimodal User Interfaces, 2015
Designing multimodal virtual environments promises revolutionary advances in interacting with computers in the near future. In this paper, we report the results of an experimental investigation on the possible use of surroundsound systems to support visualization, taking advantage of increased knowledge about how spatial perception and attention work in the human brain. We designed two auditoryvisual cross-modal experiments, where noise bursts and light-blobs were presented synchronously, but with spatial offsets. We presented sounds in two ways: using free field sounds and using a stereo speaker set. Participants were asked to localize the direction of sound sources. In the first experiment visual stimuli were displaced vertically relative to the sounds, in the second experiment we used horizontal offsets. We found that, in both experiments, sounds were mislocalized in the direction of the visual stimuli in each condition (ventriloquism effect), but this effect was stronger when visual stimuli were displaced vertically, as compared to horizontally. Moreover we found that the ventriloquism Electronic supplementary material The online version of this article (
Related papers
Creating interactive virtual auditory environments
IEEE Computer Graphics and Applications, 2002
SAE Institute London SAE 620: Major Project Efficient Real-Time Virtual Auditory Display for Virtual Reality Experiences Student details
Virtual Reality (VR) is on the edge of becoming the next major delivery plat- form for digital content. Technological progress has made accessible a variety of new tools, allowing development of the necessary hardware and software for VR interaction. The importance of audio in this process is increasing. Audio can intensify the three dimensional experience and immersion of game players and users in other applications. This research focuses on determining and implementing the necessary ele- ments for a system able to deliver Virtual Auditory Display (VAD). The system structure is adapted to fit a new emerging gaming platform: smart- phones. Developing for mobile platforms requires consideration of many con- straints such as memory, processor load and development components. The result is a real-time dynamic VAD, manageable by mobile devices, and able to trigger spatial auditory perception across the azimuthal plane. In this study it is also shown how the VAD developed following custom imple- mentation of selected Head Related Transfer Function (HRTF), is able to generate azimuthal localization of sound events in VR scenarios.
Cocktail – a demonstration of massively multi-component audio environments for illustration and analysis
We present MMAE – Massively Multi-component Audio Environments – a new concept in auditory presentation, and Cocktail – a demonstrator built on this technology. MMAE creates a dynamic audio environment by playing a large number of sound clips simultaneously at different locations in a virtual 3D space. The technique utilizes standard soundboards and is based in the Snack Sound Toolkit. The result is an efficient 3D audio environment that can be modified dynamically, in real time. Applications range from the creation of canned as well as online audio environments for games and entertainment to the browsing, analyzing and comparing of large quantities of audio data. We also demonstrate the Cocktail implementation of MMAE using several test cases as examples.
A beam tracing approach to acoustic modeling for interactive virtual environments
Proceedings of the 25th annual conference on Computer graphics and interactive techniques - SIGGRAPH '98, 1998
Virtual environment research has focused on interactive image generation and has largely ignored acoustic modeling for spatialization of sound. Yet, realistic auditory cues can complement and enhance visual cues to aid navigation, comprehension, and sense of presence in virtual environments. A primary challenge in acoustic modeling is computation of reverberation paths from sound sources fast enough for real-time auralization. We have developed a system that uses precomputed spatial subdivision and "beam tree" data structures to enable real-time acoustic modeling and auralization in interactive virtual environments. The spatial subdivision is a partition of 3D space into convex polyhedral regions (cells) represented as a cell adjacency graph. A beam tracing algorithm recursively traces pyramidal beams through the spatial subdivision to construct a beam tree data structure representing the regions of space reachable by each potential sequence of transmission and specular reflection events at cell boundaries. From these precomputed data structures, we can generate high-order specular reflection and transmission paths at interactive rates to spatialize fixed sound sources in real-time as the user moves through a virtual environment. Unlike previous acoustic modeling work, our beam tracing method: 1) supports evaluation of reverberation paths at interactive rates, 2) scales to compute highorder reflections and large environments, and 3) extends naturally to compute paths of diffraction and diffuse reflection efficiently. We are using this system to develop interactive applications in which a user experiences a virtual environment immersively via simultaneous auralization and visualization.
The Immersive Audio Environment Implementation, Subjective Tests and Results
2011
The effectiveness of virtual environments depends largely on how efficiently they recreate the real world. In the case of auditory virtual environments, the importance of accurate recreation is enhanced since there are no visual cues to assist perception, as in the case of audiovisual virtual environments. In this paper, we present the Immersive Audio Environment (IAE), an easily constructible and portable structure, which is capable of 3-D sound auralization with very high spatial resolution. A novel method for acoustically positioning loudspeakers in space, which is required by the IAE for simulation of sound sources, is presented in this paper. We also present a method to calibrate loudspeakers in an audio system in the case when the loudspeakers are not of the same type. Our contribution is the creation of a system that uses existing and modifications of existing techniques; Vector Based Amplitude Panning (VBAP), in order to recreate an audio battle environment. The IAE recreates the audio effects of battle scenes and can be used as a virtual battle environment for training soldiers. The results of subjective tests show a very low error standard deviation for azimuth and elevation angles and a high correlation between user responses and true angles.
Perceptual audio rendering of complex virtual environments
ACM Transactions on Graphics, 2004
: Left, an overview of a test virtual environment, containing 174 sound sources. All vehicles are moving. Mid-left, the magenta dots indicate the locations of the sound sources while the red sphere represents the listener. Notice that the train and the river are extended sources modeled by collections of point sources. Mid-right, ray-paths from the sources to the listener. Paths in red correspond to the perceptually masked sound sources. Right, the blue boxes are clusters of sound sources with the representatives of each cluster in grey. Combination of auditory culling and spatial clustering allows us to render such complex audio-visual scenes in real-time.
Virtual Reality System with Integrated Sound Field Simulation and Reproduction
EURASIP Journal on Advances in Signal Processing, 2007
A real-time audio rendering system is introduced which combines a full room-specific simulation, dynamic crosstalk cancellation, and multitrack binaural synthesis for virtual acoustical imaging. The system is applicable for any room shape (normal, long, flat, coupled), independent of the a priori assumption of a diffuse sound field. This provides the possibility of simulating indoor or outdoor spatially distributed, freely movable sources and a moving listener in virtual environments. In addition to that, near-tohead sources can be simulated by using measured near-field HRTFs. The reproduction component consists of a headphone-free reproduction by dynamic crosstalk cancellation. The focus of the project is mainly on the integration and interaction of all involved subsystems. It is demonstrated that the system is capable of real-time room simulation and reproduction and, thus, can be used as a reliable platform for further research on VR applications.
A framework for immersive spatial audio performance
Proceedings of the 2006 …, 2006
Traditional uses of virtual audio environments tend to focus on perceptually accurate acoustic representations. Though spatialization of sound sources is important, it is necessary to leverage control of the sonic representation when considering musical applications. The proposed framework allows for the creation of perceptually immersive scenes that function as musical instruments. Loudspeakers and microphones are modeled within the scene along with the listener/performer, creating a navigable 3D sonic space where sound sources and sinks process audio according to user-defined spatial mappings.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.