Papers by Christian Frisson
Proc. of the 3rd …, Jan 1, 2007
While an extensive palette of sound and visual generation techniques have been developed during t... more While an extensive palette of sound and visual generation techniques have been developed during the era of digital signal processing, the design of innovative virtual instruments has come to dramatic fruition over the last decade. The use of measured biological signals to drive these instruments proposes some new and powerful tools for clinical, scientific and artistic applications. Over the period of one month -during the eNTERFACE'07 summer workshop in Istanbul, Turkeyresearchers from the fields of human-computer interfaces, sound synthesis and new media art worked together towards this common goal. A framework for auditory display and bio-musical applications was established upon which were based different experimental prototypes. Diverse methods for the analysis of measured physiological signals and of mapping the extracted parameters to sound and visual synthesis processes were explored. Biologically-driven musical instruments and data displays for clinical and medical purposes were built. From this have emerged some worthwhile perspectives on future research. This report summarises the results of that project.
While an extensive palette of sound and visual generation techniques have been developed during t... more While an extensive palette of sound and visual generation techniques have been developed during the era of digital signal processing, the design of innovative virtual instruments has come to dramatic fruition over the last decade. The use of measured biological signals to drive these instruments proposes some new and powerful tools for clinical, scientific and artistic applications. Over the period of one month -during the eNTERFACE'07 summer workshop in Istanbul, Turkeyresearchers from the fields of human-computer interfaces, sound synthesis and new media art worked together towards this common goal. A framework for auditory display and bio-musical applications was established upon which were based different experimental prototypes. Diverse methods for the analysis of measured physiological signals and of mapping the extracted parameters to sound and visual synthesis processes were explored. Biologically-driven musical instruments and data displays for clinical and medical purposes were built. From this have emerged some worthwhile perspectives on future research. This report summarises the results of that project.
This project proposes to combine efforts gathered in fields such as rapid prototyping, informatio... more This project proposes to combine efforts gathered in fields such as rapid prototyping, information visualization, gestural interaction; towards a framework for the design and implementation of multimodal annotation tools dedicated to specific tasks elicited by different needs and use cases. More precisely, this project consists in adding all the necessary and remaining components to a rapid prototyping tool so as to allow the development of multimodal annotation tools by visually programming the application workflow.
This paper focuses on the robust indexing of sperm whale hydrophone recordings based on a set of ... more This paper focuses on the robust indexing of sperm whale hydrophone recordings based on a set of features extracted from a realtime passive underwater acoustic tracking algorithm for multiple whales using four hydrophones. Acoustic localization permits the study of whale behavior in deep water without interfering with the environment. Given the position coordinates, we are able to generate different features such as the speed, energy of the clicks, Inter-Click-Interval (ICI), and so on. These features allow to construct different markers which allow us to index and structure the audio files. Thus, the behavior study is facilitated by choosing and accessing the corresponding index in the audio file. The complete indexing algorithm is processed on real data from the NUWC (Naval Undersea Warfare Center of the US Navy) and the AUTEC (Atlantic Undersea Test & Evaluation Center-Bahamas). Our model is validated by similar results from the US Navy (NUWC) and SOEST (School of Ocean and Earth Science and Technology) Hawaii university labs in a single whale case. Finally, as an illustration, we index a single whale sound file using the extracted whale's features provided by the tracking, and we present an example of an XML script structuring it.
Multimodal Guitar : A Toolbox For Augmented Guitar Performances
ABSTRACT This project aims at studying how recent interactive and interactions technologies would... more ABSTRACT This project aims at studying how recent interactive and interactions technologies would help extend how we play the guitar, thus defining the multimodal guitar. Our contributions target three main axes: audio analysis, gestural control and ...
This paper presents a prototype tool for browsing through multimedia libraries using content-base... more This paper presents a prototype tool for browsing through multimedia libraries using content-based multimedia information retrieval techniques. It is composed of several groups of components for multimedia analysis, data mining, interactive visualization, as well as connection with external hardware controllers. The musical application of this tool uses descriptors of timbre, harmony, as well as rhythm and two different approaches for exploring/browsing content. First, a dynamic data mining allows the user to group sounds into clusters according to those different criteria, whose importance can be weighted interactively. In a second mode, sounds that are similar to a query are returned to the user, and can be used to further proceed with the search. This approach also borrows from multi-criteria optimization concept to return a relevant list of similar sounds.
This project focused on the management of the various visualisation methods and on the implementa... more This project focused on the management of the various visualisation methods and on the implementation of audio features in the MediaCycle framework. Two different kinds of visualisation methods has been defined, either to represent the collection of media as a two-dimensional map or as a tree.
This paper presents AudioCycle, a prototype application for browsing through music loop libraries... more This paper presents AudioCycle, a prototype application for browsing through music loop libraries. AudioCycle provides the user with a graphical view where the audio extracts are visualized and organized according to their similarity in terms of musical properties, such as timbre, harmony, and rhythm. The user is able to navigate in this visual representation, and listen to individual audio extracts, searching for those of interest. AudioCycle draws from a range of technologies, including audio analysis from music information retrieval research, 3D visualization, spatial auditory rendering, audio time-scaling and pitch modification. The proposed approach extends on previously described music and audio browsers. Concepts developed here will be of interest to DJs, remixers, musicians, soundtrack composers, but also sound designers and foley artists. Possible extension to multimedia libraries are also suggested.
The purpose of the LoopJam installation is to gather people to collaboratively build a live music... more The purpose of the LoopJam installation is to gather people to collaboratively build a live musical atmosphere. A two-dimensional "sound map" is created by our software which analyzes sounds, as well as musical loops, and groups them by similarity. Musical loops are chunks of music content (typically a few musical measures long) that create rhythmic music when played repeatedly. Wandering through the installation, each participant explores this sound map and activates sounds by simple gestures such as raising the hand, clapping or stepping. The playback of each sound is synchronized by our software, allowing a collaborative audio composition shared between all participants. Networking technologies are used for the communication between the 3D cameras tracking the participant motions and the music and visual rendering components. Network communication can hence potentially be used to create interactions between distant installations.
The aim of this project is to explore new user interfaces and data organization for database navi... more The aim of this project is to explore new user interfaces and data organization for database navigation within the MediaCycle software. Both issues are deeply interconnected since the database display depends on the internal data organization and vice-versa, the data organization has to be chosen to allow useful and meaningful displays as well as intuitive navigation. Because each data type or application might require a specific visualization, we expanded the plugin architecture of MediaCycle to facilitate the testing of various prototypes. Two types of practical applications have been considered: query by similarity and database exploration. We present here a new browsing interface that combines both types, as well as some tests of external controllers (3D mouse, trackpad, . . . ) that can potentially enhance the navigation process.
This paper presents the LoopJam installation which allows participants to interact with a sound m... more This paper presents the LoopJam installation which allows participants to interact with a sound map using a 3D computer vision tracking system. The sound map results from similarity-based clustering of sounds. The playback of these sounds is controlled by the positions or gestures of participants tracked with a Kinect depth-sensing camera. The beat-inclined bodily movements of participants in the installation are mapped to the tempo of played sounds, while the playback speed is synchronized by default among all sounds. We presented and tested an early version of the installation to three exhibitions in Belgium, Italy and France. The reactions among participants ranged between curiosity and amusement.
This project aims at providing segmentation tools for assisting the browsing of multimedia conten... more This project aims at providing segmentation tools for assisting the browsing of multimedia content. We implemented several segmentation algorithms: two based on the Bayesian Information Criterion (BIC), the first frame-by-frame and the second using a "divide-and-conquer" approach; a third based on the self-similarity matrix. We integrated the Torch3 library to benefit from classification algorithms. We also introduced optimization methods, notably parallelized image/video feature extraction distributed over GPUs and CPUs of one computer, using StarPU. All the components developed as part of this project have been incorporated in the MediaCycle framework.
Recently, various dimensionality reduction approaches have been proposed as alternatives to PCA o... more Recently, various dimensionality reduction approaches have been proposed as alternatives to PCA or LDA. These improved approaches do not rely on a linearity assumption, and are hence capable of discovering more complex embeddings within different regions of the data sets. Despite their success on artificial datasets, it is not straightforward to predict which technique is the most appropriate for a given real dataset. In this paper, we empirically evaluate recent techniques on two real audio use cases: musical instrument loops used in music production and sound effects used in sound editing. ISOMAP and t-SNE are being compared to PCA in a visualization problem, where we end up with a two-dimensional view. Various evaluation measures are used: classification performance, as well as trustworthiness/continuity assessing the preservation of neighborhoods. Although PCA and ISOMAP can yield good continuity performance even locally (samples in the original space remain close-by in the low-dimensional one), they fail to preserve the structure of the data well enough to ensure that distinct subgroups remain separate in the visualization. We show that t-SNE presents the best performance, and can even be beneficial as a pre-processing stage for improving classification when the amount of labeled data is low.
This paper presents AudioCycle, a prototype application for browsing through music loop libraries... more This paper presents AudioCycle, a prototype application for browsing through music loop libraries. AudioCycle provides the user with a graphical view where the audio extracts are visualized and organized according to their similarity in terms of musical properties, such as timbre, harmony, and rhythm. The user is able to navigate in this visual representation and listen to individual audio extracts, as well as query the database by providing audio examples. AudioCycle draws from a range of technologies, including audio analysis from music information retrieval research, 3D visualization, spatial auditory rendering, audio time-scaling and pitch modification. The proposed approach extends on previously described music and audio browsers. Possible extension to multimedia libraries is also suggested.
This paper presents AudioCycle, a prototype application for browsing through music loop libraries... more This paper presents AudioCycle, a prototype application for browsing through music loop libraries. AudioCycle provides the user with a graphical view where the audio extracts are visualized and organized according to their similarity in terms of musical properties, such as timbre, harmony, and rhythm. The user is able to navigate in this visual representation, and listen to individual audio extracts, searching for those of interest. AudioCycle draws from a range of technologies, including audio analysis from music information retrieval research, 3D visualization, spatial auditory rendering, audio time-scaling and pitch modification. The proposed approach extends on previously described music and audio browsers. Concepts developped here will be of interest to DJs, remixers, musicians, soundtrack composers, but also sound designers and foley artists. Possible extension to multimedia libraries are also suggested.
Similarity in media content: digital art perspectives
DI-fusion, le Dépôt institutionnel numérique de l'ULB, est l... more DI-fusion, le Dépôt institutionnel numérique de l'ULB, est l'outil de référencementde la production scientifique de l'ULB.L'interface de recherche DI-fusion permet de consulter les publications des chercheurs de l'ULB et les thèses qui y ont été défendues.
This paper presents a numediart project in collaboration with two artistic projects: Musichorégra... more This paper presents a numediart project in collaboration with two artistic projects: Musichorégraphie d'appartement by André and BioDiva by Laurence Moletta (Nios Karma) [34]. The scope of this project was to offer technological forecasting and development consultancy to these two artists that both share a common goal of using gestures to control on audiovisual rendering on stage (see section 2).
This project aims at studying how recent interactive and interaction technologies would help exte... more This project aims at studying how recent interactive and interaction technologies would help extend how we play the guitar, thus defining the "multimodal guitar". We investigate two axes, 1) "A gestural/polyphonic sensing/processing toolbox to augment guitar performances", and 2) "An interactive guitar score following environment for adaptive learning". These approaches share quite similar technological challenges (sensing, analysis, processing, synthesis and interaction methods) and dissemination intentions (community-based, low-cost, open-source whenever possible), while leading to different applications (respectively artistic and educational), still targeted towards experienced players and beginners.
This paper presents the development of rapid and reusable gestural interface prototypes for navig... more This paper presents the development of rapid and reusable gestural interface prototypes for navigation by similarity in an audio database and for sound manipulation, using the AudioCycle application. For this purpose, we propose and follow guidelines for rapid prototyping that we apply using the PureData visual programming environment. We have mainly developed three prototypes of manual control: one combining a 3D mouse and a jog wheel, a second featuring a force-feedback 3D mouse, and a third taking advantage of the multitouch trackpad. We discuss benefits and shortcomings we experienced while prototyping using this approach.
Uploads
Papers by Christian Frisson