Proceedings of the 28th International Conference on Intelligent User Interfaces
Novel AI-generated audio samples are evaluated for descriptive qualities such as the smoothness o... more Novel AI-generated audio samples are evaluated for descriptive qualities such as the smoothness of a morph using crowdsourced human listening tests. However, the methods to design interfaces for such experiments and to effectively articulate the descriptive audio quality under test receive very little attention in the evaluation metrics literature. In this paper, we explore the use of visual metaphors of image-schema to design interfaces to evaluate AI-generated audio. Furthermore, we highlight the importance of framing and contextualizing a descriptive audio quality under measurement using such constructs. Using both pitched sounds and textures, we conduct two sets of experiments to investigate how the quality of responses vary with audio and task complexities. Our results show that, in both cases, by using image-schemas we can improve the quality and consensus of AI-generated audio evaluations. Our findings reinforce the importance of interface design for listening tests and stationary visual constructs to communicate temporal qualities of AI-generated audio samples, especially to naïve listeners on crowdsourced platforms. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI; User studies; • Computing methodologies → Artificial intelligence; • Applied computing → Sound and music computing.
Zenodo (CERN European Organization for Nuclear Research), Jun 23, 2019
This paper concerns the development of a music cocreation system that engenders creative partners... more This paper concerns the development of a music cocreation system that engenders creative partnerships in musical interaction. Towards this goal, specific design guidelines are identified for improving the musicians' sense of creative partnership with the system through behaviors that communicate musical intent, alternate musical roles based on musical stability, and negotiate transitions in musical characteristics based on togetherness in interaction. The design guidelines motivate the development of an agent architecture for interactive improvisation. The architecture enables an agent to select music response behaviors based on the stability and togetherness sensed during interaction. An agent improvisation system is developed that uses the architecture to make decisions during improvised percussion duets. The agent negotiates transitions between musical roles and musical characteristics by maintaining different degrees of metrical coherence and similarity with the rhythms played by the musician. By negotiating transitions with a musician, the system presumably has an impact on the musicians sense of creative musical partnership.
Standard evaluation metrics such as the Inception score and Fréchet Audio Distance provide a gene... more Standard evaluation metrics such as the Inception score and Fréchet Audio Distance provide a general audio quality distance metric between the synthesized audio and reference clean audio. However, the sensitivity of these metrics to variations in the statistical parameters that define an audio texture is not well studied. In this work, we provide a systematic study of the sensitivity of some of the existing audio quality evaluation metrics to parameter variations in audio textures. Furthermore, we also study three more potentially parameter-sensitive metrics for audio texture synthesis, (a) a Gram matrix based distance, (b) an Accumulated Gram metric using a summarized version of the Gram matrices, and (c) a cochlear-model based statistical features metric. These metrics use deep features that summarize the statistics of any given audio texture, thus being inherently sensitive to variations in the statistical parameters that define an audio texture. We study and evaluate the sensitivity of existing standard metrics as well as Gram matrix and cochlear-model based metrics to controlparameter variations in audio textures across a wide range of texture and parameter types, and validate with subjective evaluation. We find that each of the metrics is sensitive to different sets of texture-parameter types. This is the first step towards investigating objective metrics for assessing parameter sensitivity in audio textures.
Multiple sound sources often contain harmonics that overlap and may be degraded by environmental ... more Multiple sound sources often contain harmonics that overlap and may be degraded by environmental noise. The auditory system is capable of teasing apart these sources into distinct mental objects, or streams. Such an "auditory scene analysis" enables the brain to solve the cocktail party problem. A neural network model of auditory scene analysis, called the AIRSTREAM model, is presented to propose how the brain accomplishes this feat. The model clarifies how the frequency components that correspond to a give acoustic source may be coherently grouped together into distinct streams based on pitch and spatial cues. The model also clarifies how multiple streams may be distinguishes and seperated by the brain. Streams are formed as spectral-pitch resonances that emerge through feedback interactions between frequency-specific spectral representaion of a sound source and its pitch. First, the model transforms a sound into a spatial pattern of frequency-specific activation across a...
Audio textures are a superset of standard musical instrument timbres that include more complex so... more Audio textures are a superset of standard musical instrument timbres that include more complex sounds such as rain, wind, rolling, or scraping. With appropriate modeling strategies, textures can be synthesized under parametric control analogous to the way musical instruments are, and can then become a powerful creativity tools for music making. However, audio textures, with complex structure spanning multiple time scales, are a challenge to model and generate synthetically. They are even challenging to define. Deep learning approaches offer new ways to develop generative audio texture models, and they create different demands on training data than traditional modeling approaches, In this paper we briefly review previous modeling approaches, and attempt to rationalize and converge on a definition of textures using modeling concepts. We introduce a new and growing data set along with a system for managing metadata specifically designed for audio textures. Finally, we report on some re...
This paper describes innovative aspects of the Sonoport Stu-dio, a dynamic audio experience creat... more This paper describes innovative aspects of the Sonoport Stu-dio, a dynamic audio experience creation platform which will be presented in a demonstration at the 1st Web Audio Con-ference in Paris. Two key components of the platform are 1) Sound Model templates that represent classes of dynamic sonic behaviors, and 2) Interaction Model templates that represent classes of interactive behavior that are commonly mapped to interac-tive sound elements. These components, together with a database of sound files and other authoring tools, comprise an interactive audio ex-perience creation platform which is aimed at a typical cre-ative web designer/developer who wants to take advantage of the new capabilities offered by the Web Audio platform, but may not have a deep synthesis or audio processing back-ground.
[Proceedings 1992] IJCNN International Joint Conference on Neural Networks
This article describes a neural network model capable of generating a spatial representation of t... more This article describes a neural network model capable of generating a spatial representation of the pitch of an acoustic source. Pitch is one of several auditory percepts used by humans to separate multiple sound sources in the environment from each other. The model provides a neural instantiation of a type of ''harmonic sieve''. It is capable of quantitatively simulating a large body of psychoacoustical datal including new data on octave shift perception.
The goal of our research is to find ways of supporting and encouraging musical behavior by non-mu... more The goal of our research is to find ways of supporting and encouraging musical behavior by non-musicians in shared public performance environments. Previous studies indicated simultaneous music listening and performance is difficult for non-musicians, and that visual support for the task might be helpful. This paper presents results from a preliminary user study conducted to evaluate the effect of visual feedback on a musical tracking task. Participants generated a musical signal by manipulating a hand-held device with two dimensions of control over two parameters, pitch and density of note events, and were given the task of following a target pattern as closely as possible. The target pattern was a machine-generated musical signal comprising of variation over the same two parameters. Visual feedback provided participants with information about the control parameters of the musical signal generated by the machine. We measured the task performance under different visual feedback stra...
Mapping gestures to digital musical instrument parameters is not trivial when the dimensionality ... more Mapping gestures to digital musical instrument parameters is not trivial when the dimensionality of the sensor-captured data is high and the model relating the gesture to sensor data is unknown. In these cases, a front-end processing system for extracting gestural information embedded in the sensor data is essential. In this paper we propose an unsupervised offline method that learns how to reduce and map the gestural data to a generic instrument parameter control space. We make an unconventional use of the Self-Organizing Maps to obtain only a geometrical transformation of the gestural data, while dimensionality reduction is handled separately. We introduce a novel training procedure to overcome two main SelfOrganizing Maps limitations which otherwise corrupt the interface usability. As evaluation, we apply this method to our existing Voice-Controlled Interface for musical instruments, obtaining sensible usability improvements.
Bridges is a software system designed to support composition and performance for networked musica... more Bridges is a software system designed to support composition and performance for networked musical ensembles. Bridges facilitates dynamic connections between a variety of devices, applications, and individual musical objects within multiple applications. The focus is on colocated performers, and on control data and communication structures rather than on audio delivery. It manages addressing and mapping between components making instrument design and networked compositions more object oriented, reconfigurable, and portable. Bridges is designed to support research in to new kinds of musical activity, particularly communication between performing musicians, mappings between interfaces and sound synthesis, and most importantly, dynamic network and control structures.
Continuous pitch movements called “gamakas” are key to Carnatic music, but are not included in it... more Continuous pitch movements called “gamakas” are key to Carnatic music, but are not included in its traditional notation system. Modeling the melodic and shape aspects of gamakas and how they are selected for phrases given as notation are interesting interrelated problems that have analogs in text to speech and expressive singing synthesis systems. Descriptive textual and graphical notation techniques have also been proposed as tools for studying gamakas. However, a concatenative model of phrase-level gamaka selection leads to a combinatorial explosion of possibilities to be selected from. We propose a “longest path” optimization algorithm for phrase-level selection of gamakas that solves this combinatorial explosion using a scoring function that expresses local continuity preferences; and a two-component “stage-dance” representation of gamakas which simplifies the scoring function by separating local melodic and shape continuity preferences. We analyzed a performance of a compositio...
sonicBard is a system providing real-time sound interaction for traditional oral storytelling per... more sonicBard is a system providing real-time sound interaction for traditional oral storytelling performers. The system is designed to give storytellers an extended capability to accompany themselves with music, sound effects, and vocal voice transformations while minimally disrupting their established performative techniques. Since a given story may potentially incorporate dozens of sounds, each with their own interaction demands, a serious challenge is posed. The constraints stemming from the “hands and eyes busy” nature of storytelling leaves little room for the attentional demands of typical computer-controlled instruments. This paper presents the interface design and implementation, and describes newly emerging browser platform capabilities upon which we have chosen to develop sonicBard.
Large multi-channel spatial audio systems have historically been play-grounds for universities an... more Large multi-channel spatial audio systems have historically been play-grounds for universities and well-funded studios, but only a dream for independent composers. Similarly, "parallel computers" were locked in research facilities, where only a few musicians ever gained access to, for example, the compute power to convolve hundreds of separate audio streams with spatiallyspecific room impulse responses. Mobile devices in the hands of audiences can quickly configure themselves into such systems at very affordable (and distributed) cost and little effort, making powerful and expressive spatially distributed musical platforms accessible to anyone today. We describe some software systems and artistic works that have been developed recently to explore some of the spatial audio capabilities of the mobile device browser platform.
In this article we describe a user-driven adaptive method to control the sonic response of digita... more In this article we describe a user-driven adaptive method to control the sonic response of digital musical instruments using information extracted from the timbre of the human voice. The mapping between heterogeneous attributes of the input and output timbres is determined from data collected through machine-listening techniques and then processed by unsupervised machine-learning algorithms. This approach is based on a minimum-loss mapping that hides any synthesizer-specific parameters and that maps the vocal interaction directly to perceptual characteristics of the generated sound. The mapping adapts to the dynamics detected in the voice and maximizes the timbral space covered by the sound synthesizer. The strategies for mapping vocal control to perceptual timbral features and for automating the customization of vocal interfaces for different users and synthesizers, in general, are evaluated through a variety of qualitative and quantitative methods.
Conceptors are a recent development in the field of reservoir computing; they can be used to infl... more Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with generation of long sequences (of more than 20 samples) and multi-timbre pattern generation. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined. INTRODUCTION 33 Machine Learning and Sound Synthesis 34 Current intersections between sound synthesis and machine learning are evolving quickly. We have seen 35 significant progress in symbolic note generation (e.g. RL Tuner (Jaques et al., 2016), Flow Machines 36 (Ghedini et al., 2016)), parametric control of sound synthesis models (e.g Wekinator (Fiebrink, 2011), 37 automatic VST programming (Yee-King et al., 2018)) and also with current state of the art raw audio 38 generation techniques. These recent advances in raw audio synthesis principally use deep architectures, 39
Style transfer is a technique for combining two images based on the activations and feature stati... more Style transfer is a technique for combining two images based on the activations and feature statistics in a deep learning neural network architecture. This paper studies the analogous task in the audio domain and takes a critical look at the problems that arise when adapting the original vision-based framework to handle spectrogram representations. We conclude that CNN architectures with features based on 2D representations and convolutions are better suited for visual images than for time-frequency representations of audio. Despite the awkward fit, experiments show that the Gram matrix determined style for audio is more closely aligned with timbral signatures without temporal structure whereas network layer activity determining audio content seems to capture more of the pitch and rhythmic structures. We shed insight on several reasons for the domain differences with illustrative examples. We motivate the use of several types of one-dimensional CNNs that generate results that are better aligned with intuitive notions of audio texture than those based on existing architectures built for images. These ideas also prompt an exploration of audio texture synthesis with architectural variants for extensions to infinite textures, multi-textures, parametric control of re-This research was supported by an NVIDIA Corporation Academic Programs GPU grant.
Proceedings of the 22nd ACM international conference on Multimedia, 2014
New low-level sound synthesis capabilities have recently become available in Web browsers. Howeve... more New low-level sound synthesis capabilities have recently become available in Web browsers. However, there is a considerable gap between the enabling technology for interactive audio and its widespread adoption in Web media content. We identify several areas where technologies are necessary to support the various stages of development and deployment, describe systems we have developed to address those needs, and show how they work together within a specific Web content development scenario.
Proceedings of the 22nd ACM international conference on Multimedia, 2014
States of Diffusion is a participative audio installation artwork that explores patterns of sound... more States of Diffusion is a participative audio installation artwork that explores patterns of sound transformation that move through space as they evolve spectrally. Six one-minute segments exploring themes of intonation and timing of events across possibly many different audio sources are coordinated via a web server. The spatial diffusion flows through mobile devices held by visitors to the gallery. Visitors also exert a subtle influence over sound through the movement of their device and their movement through the gallery.
This article explores listening and communications strategies that arise with a collaborative sco... more This article explores listening and communications strategies that arise with a collaborative scoring system we are developing for use within improvisational contexts. Performers generate notation on a scrolling score a short time before it is played or rendered into sound. Working a short time in the future allows performers to respond to sound as they would in any improvisatory situation, and yet coordinate their activity through notation in a way typically associated with pre-composed music. The ‘Anticipatory Score’ platform supports the exploration of different kinds of relationships between performers, composers and audience members, and different listening and engagement strategies that affect the musical experience for all participants.
In this paper we develop adaptive techniques for mapping generic user interfaces to synthesis eng... more In this paper we develop adaptive techniques for mapping generic user interfaces to synthesis engines. Upon selecting a subset of synthesis parameters, the system automatically finds the parameters-to-sound deterministic relationship in a multidimensional space. We analyze this sonic space using two different unsupervised dimensionality reduction techniques and we build the mapping using statistical information on a lower, but maximally representative, number of dimensions. The result is an adaptation of any generalpurpose interface to a specific synthesis engine, providing control directly over the perceptual features with greatest variance. This approach guarantees a linear relationship between control signals and perceptual features, and at the same time, reduces the control space dimensionality maintaining the maximum explorability of the sonic space.
Proceedings of the 28th International Conference on Intelligent User Interfaces
Novel AI-generated audio samples are evaluated for descriptive qualities such as the smoothness o... more Novel AI-generated audio samples are evaluated for descriptive qualities such as the smoothness of a morph using crowdsourced human listening tests. However, the methods to design interfaces for such experiments and to effectively articulate the descriptive audio quality under test receive very little attention in the evaluation metrics literature. In this paper, we explore the use of visual metaphors of image-schema to design interfaces to evaluate AI-generated audio. Furthermore, we highlight the importance of framing and contextualizing a descriptive audio quality under measurement using such constructs. Using both pitched sounds and textures, we conduct two sets of experiments to investigate how the quality of responses vary with audio and task complexities. Our results show that, in both cases, by using image-schemas we can improve the quality and consensus of AI-generated audio evaluations. Our findings reinforce the importance of interface design for listening tests and stationary visual constructs to communicate temporal qualities of AI-generated audio samples, especially to naïve listeners on crowdsourced platforms. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI; User studies; • Computing methodologies → Artificial intelligence; • Applied computing → Sound and music computing.
Zenodo (CERN European Organization for Nuclear Research), Jun 23, 2019
This paper concerns the development of a music cocreation system that engenders creative partners... more This paper concerns the development of a music cocreation system that engenders creative partnerships in musical interaction. Towards this goal, specific design guidelines are identified for improving the musicians' sense of creative partnership with the system through behaviors that communicate musical intent, alternate musical roles based on musical stability, and negotiate transitions in musical characteristics based on togetherness in interaction. The design guidelines motivate the development of an agent architecture for interactive improvisation. The architecture enables an agent to select music response behaviors based on the stability and togetherness sensed during interaction. An agent improvisation system is developed that uses the architecture to make decisions during improvised percussion duets. The agent negotiates transitions between musical roles and musical characteristics by maintaining different degrees of metrical coherence and similarity with the rhythms played by the musician. By negotiating transitions with a musician, the system presumably has an impact on the musicians sense of creative musical partnership.
Standard evaluation metrics such as the Inception score and Fréchet Audio Distance provide a gene... more Standard evaluation metrics such as the Inception score and Fréchet Audio Distance provide a general audio quality distance metric between the synthesized audio and reference clean audio. However, the sensitivity of these metrics to variations in the statistical parameters that define an audio texture is not well studied. In this work, we provide a systematic study of the sensitivity of some of the existing audio quality evaluation metrics to parameter variations in audio textures. Furthermore, we also study three more potentially parameter-sensitive metrics for audio texture synthesis, (a) a Gram matrix based distance, (b) an Accumulated Gram metric using a summarized version of the Gram matrices, and (c) a cochlear-model based statistical features metric. These metrics use deep features that summarize the statistics of any given audio texture, thus being inherently sensitive to variations in the statistical parameters that define an audio texture. We study and evaluate the sensitivity of existing standard metrics as well as Gram matrix and cochlear-model based metrics to controlparameter variations in audio textures across a wide range of texture and parameter types, and validate with subjective evaluation. We find that each of the metrics is sensitive to different sets of texture-parameter types. This is the first step towards investigating objective metrics for assessing parameter sensitivity in audio textures.
Multiple sound sources often contain harmonics that overlap and may be degraded by environmental ... more Multiple sound sources often contain harmonics that overlap and may be degraded by environmental noise. The auditory system is capable of teasing apart these sources into distinct mental objects, or streams. Such an "auditory scene analysis" enables the brain to solve the cocktail party problem. A neural network model of auditory scene analysis, called the AIRSTREAM model, is presented to propose how the brain accomplishes this feat. The model clarifies how the frequency components that correspond to a give acoustic source may be coherently grouped together into distinct streams based on pitch and spatial cues. The model also clarifies how multiple streams may be distinguishes and seperated by the brain. Streams are formed as spectral-pitch resonances that emerge through feedback interactions between frequency-specific spectral representaion of a sound source and its pitch. First, the model transforms a sound into a spatial pattern of frequency-specific activation across a...
Audio textures are a superset of standard musical instrument timbres that include more complex so... more Audio textures are a superset of standard musical instrument timbres that include more complex sounds such as rain, wind, rolling, or scraping. With appropriate modeling strategies, textures can be synthesized under parametric control analogous to the way musical instruments are, and can then become a powerful creativity tools for music making. However, audio textures, with complex structure spanning multiple time scales, are a challenge to model and generate synthetically. They are even challenging to define. Deep learning approaches offer new ways to develop generative audio texture models, and they create different demands on training data than traditional modeling approaches, In this paper we briefly review previous modeling approaches, and attempt to rationalize and converge on a definition of textures using modeling concepts. We introduce a new and growing data set along with a system for managing metadata specifically designed for audio textures. Finally, we report on some re...
This paper describes innovative aspects of the Sonoport Stu-dio, a dynamic audio experience creat... more This paper describes innovative aspects of the Sonoport Stu-dio, a dynamic audio experience creation platform which will be presented in a demonstration at the 1st Web Audio Con-ference in Paris. Two key components of the platform are 1) Sound Model templates that represent classes of dynamic sonic behaviors, and 2) Interaction Model templates that represent classes of interactive behavior that are commonly mapped to interac-tive sound elements. These components, together with a database of sound files and other authoring tools, comprise an interactive audio ex-perience creation platform which is aimed at a typical cre-ative web designer/developer who wants to take advantage of the new capabilities offered by the Web Audio platform, but may not have a deep synthesis or audio processing back-ground.
[Proceedings 1992] IJCNN International Joint Conference on Neural Networks
This article describes a neural network model capable of generating a spatial representation of t... more This article describes a neural network model capable of generating a spatial representation of the pitch of an acoustic source. Pitch is one of several auditory percepts used by humans to separate multiple sound sources in the environment from each other. The model provides a neural instantiation of a type of ''harmonic sieve''. It is capable of quantitatively simulating a large body of psychoacoustical datal including new data on octave shift perception.
The goal of our research is to find ways of supporting and encouraging musical behavior by non-mu... more The goal of our research is to find ways of supporting and encouraging musical behavior by non-musicians in shared public performance environments. Previous studies indicated simultaneous music listening and performance is difficult for non-musicians, and that visual support for the task might be helpful. This paper presents results from a preliminary user study conducted to evaluate the effect of visual feedback on a musical tracking task. Participants generated a musical signal by manipulating a hand-held device with two dimensions of control over two parameters, pitch and density of note events, and were given the task of following a target pattern as closely as possible. The target pattern was a machine-generated musical signal comprising of variation over the same two parameters. Visual feedback provided participants with information about the control parameters of the musical signal generated by the machine. We measured the task performance under different visual feedback stra...
Mapping gestures to digital musical instrument parameters is not trivial when the dimensionality ... more Mapping gestures to digital musical instrument parameters is not trivial when the dimensionality of the sensor-captured data is high and the model relating the gesture to sensor data is unknown. In these cases, a front-end processing system for extracting gestural information embedded in the sensor data is essential. In this paper we propose an unsupervised offline method that learns how to reduce and map the gestural data to a generic instrument parameter control space. We make an unconventional use of the Self-Organizing Maps to obtain only a geometrical transformation of the gestural data, while dimensionality reduction is handled separately. We introduce a novel training procedure to overcome two main SelfOrganizing Maps limitations which otherwise corrupt the interface usability. As evaluation, we apply this method to our existing Voice-Controlled Interface for musical instruments, obtaining sensible usability improvements.
Bridges is a software system designed to support composition and performance for networked musica... more Bridges is a software system designed to support composition and performance for networked musical ensembles. Bridges facilitates dynamic connections between a variety of devices, applications, and individual musical objects within multiple applications. The focus is on colocated performers, and on control data and communication structures rather than on audio delivery. It manages addressing and mapping between components making instrument design and networked compositions more object oriented, reconfigurable, and portable. Bridges is designed to support research in to new kinds of musical activity, particularly communication between performing musicians, mappings between interfaces and sound synthesis, and most importantly, dynamic network and control structures.
Continuous pitch movements called “gamakas” are key to Carnatic music, but are not included in it... more Continuous pitch movements called “gamakas” are key to Carnatic music, but are not included in its traditional notation system. Modeling the melodic and shape aspects of gamakas and how they are selected for phrases given as notation are interesting interrelated problems that have analogs in text to speech and expressive singing synthesis systems. Descriptive textual and graphical notation techniques have also been proposed as tools for studying gamakas. However, a concatenative model of phrase-level gamaka selection leads to a combinatorial explosion of possibilities to be selected from. We propose a “longest path” optimization algorithm for phrase-level selection of gamakas that solves this combinatorial explosion using a scoring function that expresses local continuity preferences; and a two-component “stage-dance” representation of gamakas which simplifies the scoring function by separating local melodic and shape continuity preferences. We analyzed a performance of a compositio...
sonicBard is a system providing real-time sound interaction for traditional oral storytelling per... more sonicBard is a system providing real-time sound interaction for traditional oral storytelling performers. The system is designed to give storytellers an extended capability to accompany themselves with music, sound effects, and vocal voice transformations while minimally disrupting their established performative techniques. Since a given story may potentially incorporate dozens of sounds, each with their own interaction demands, a serious challenge is posed. The constraints stemming from the “hands and eyes busy” nature of storytelling leaves little room for the attentional demands of typical computer-controlled instruments. This paper presents the interface design and implementation, and describes newly emerging browser platform capabilities upon which we have chosen to develop sonicBard.
Large multi-channel spatial audio systems have historically been play-grounds for universities an... more Large multi-channel spatial audio systems have historically been play-grounds for universities and well-funded studios, but only a dream for independent composers. Similarly, "parallel computers" were locked in research facilities, where only a few musicians ever gained access to, for example, the compute power to convolve hundreds of separate audio streams with spatiallyspecific room impulse responses. Mobile devices in the hands of audiences can quickly configure themselves into such systems at very affordable (and distributed) cost and little effort, making powerful and expressive spatially distributed musical platforms accessible to anyone today. We describe some software systems and artistic works that have been developed recently to explore some of the spatial audio capabilities of the mobile device browser platform.
In this article we describe a user-driven adaptive method to control the sonic response of digita... more In this article we describe a user-driven adaptive method to control the sonic response of digital musical instruments using information extracted from the timbre of the human voice. The mapping between heterogeneous attributes of the input and output timbres is determined from data collected through machine-listening techniques and then processed by unsupervised machine-learning algorithms. This approach is based on a minimum-loss mapping that hides any synthesizer-specific parameters and that maps the vocal interaction directly to perceptual characteristics of the generated sound. The mapping adapts to the dynamics detected in the voice and maximizes the timbral space covered by the sound synthesizer. The strategies for mapping vocal control to perceptual timbral features and for automating the customization of vocal interfaces for different users and synthesizers, in general, are evaluated through a variety of qualitative and quantitative methods.
Conceptors are a recent development in the field of reservoir computing; they can be used to infl... more Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with generation of long sequences (of more than 20 samples) and multi-timbre pattern generation. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined. INTRODUCTION 33 Machine Learning and Sound Synthesis 34 Current intersections between sound synthesis and machine learning are evolving quickly. We have seen 35 significant progress in symbolic note generation (e.g. RL Tuner (Jaques et al., 2016), Flow Machines 36 (Ghedini et al., 2016)), parametric control of sound synthesis models (e.g Wekinator (Fiebrink, 2011), 37 automatic VST programming (Yee-King et al., 2018)) and also with current state of the art raw audio 38 generation techniques. These recent advances in raw audio synthesis principally use deep architectures, 39
Style transfer is a technique for combining two images based on the activations and feature stati... more Style transfer is a technique for combining two images based on the activations and feature statistics in a deep learning neural network architecture. This paper studies the analogous task in the audio domain and takes a critical look at the problems that arise when adapting the original vision-based framework to handle spectrogram representations. We conclude that CNN architectures with features based on 2D representations and convolutions are better suited for visual images than for time-frequency representations of audio. Despite the awkward fit, experiments show that the Gram matrix determined style for audio is more closely aligned with timbral signatures without temporal structure whereas network layer activity determining audio content seems to capture more of the pitch and rhythmic structures. We shed insight on several reasons for the domain differences with illustrative examples. We motivate the use of several types of one-dimensional CNNs that generate results that are better aligned with intuitive notions of audio texture than those based on existing architectures built for images. These ideas also prompt an exploration of audio texture synthesis with architectural variants for extensions to infinite textures, multi-textures, parametric control of re-This research was supported by an NVIDIA Corporation Academic Programs GPU grant.
Proceedings of the 22nd ACM international conference on Multimedia, 2014
New low-level sound synthesis capabilities have recently become available in Web browsers. Howeve... more New low-level sound synthesis capabilities have recently become available in Web browsers. However, there is a considerable gap between the enabling technology for interactive audio and its widespread adoption in Web media content. We identify several areas where technologies are necessary to support the various stages of development and deployment, describe systems we have developed to address those needs, and show how they work together within a specific Web content development scenario.
Proceedings of the 22nd ACM international conference on Multimedia, 2014
States of Diffusion is a participative audio installation artwork that explores patterns of sound... more States of Diffusion is a participative audio installation artwork that explores patterns of sound transformation that move through space as they evolve spectrally. Six one-minute segments exploring themes of intonation and timing of events across possibly many different audio sources are coordinated via a web server. The spatial diffusion flows through mobile devices held by visitors to the gallery. Visitors also exert a subtle influence over sound through the movement of their device and their movement through the gallery.
This article explores listening and communications strategies that arise with a collaborative sco... more This article explores listening and communications strategies that arise with a collaborative scoring system we are developing for use within improvisational contexts. Performers generate notation on a scrolling score a short time before it is played or rendered into sound. Working a short time in the future allows performers to respond to sound as they would in any improvisatory situation, and yet coordinate their activity through notation in a way typically associated with pre-composed music. The ‘Anticipatory Score’ platform supports the exploration of different kinds of relationships between performers, composers and audience members, and different listening and engagement strategies that affect the musical experience for all participants.
In this paper we develop adaptive techniques for mapping generic user interfaces to synthesis eng... more In this paper we develop adaptive techniques for mapping generic user interfaces to synthesis engines. Upon selecting a subset of synthesis parameters, the system automatically finds the parameters-to-sound deterministic relationship in a multidimensional space. We analyze this sonic space using two different unsupervised dimensionality reduction techniques and we build the mapping using statistical information on a lower, but maximally representative, number of dimensions. The result is an adaptation of any generalpurpose interface to a specific synthesis engine, providing control directly over the perceptual features with greatest variance. This approach guarantees a linear relationship between control signals and perceptual features, and at the same time, reduces the control space dimensionality maintaining the maximum explorability of the sonic space.
Uploads
Papers by lonce wyse