AV-RIR: Audio-Visual Room Impulse Response Estimation

Anton Ratnarajah

2023, arXiv (Cornell University)

visibility

…

description

15 pages

link

1 file

Abstract

Accurate estimation of Room Impulse Response (RIR), which captures an environment's acoustic properties, is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and the visual cues of its corresponding environment. AV-RIR builds on a novel neural codec-based architecture that effectively captures environment geometry and materials properties and solves speech dereverberation as an auxiliary task by using multi-task learning. We also propose Geo-Mat features that augment material information into visual cues and CRIP that improves late reverberation components in the estimated RIR via image-to-RIR retrieval by 86%. Empirical results show that AV-RIR quantitatively outperforms previous audio-only and visual-only approaches by achieving 36% -63% improvement across various acoustic metrics in RIR estimation. Additionally, it also achieves higher preference scores in human evaluation. As an auxiliary benefit, dereverbed speech from AV-RIR shows competitive performance with the state-of-the-art in various spoken language processing tasks and outperforms reverberation time error score in the real-world AVSpeech dataset. Qualitative examples of both synthesized reverberant speech and enhanced speech are available online 1 .

Figures (9)

Figure 1. Overview of AV-RIR: Given a source reverberant speech in any environment, AV-RIR estimates the RIR from the reverber- ant speech using additional visual cues. The estimated RIR can be used to transform any target clean speech as if it is spoken in that environment. {jeran, sreyang, sonalkum, pchiniya, dmanocha}@umd. edu

Figure 2. Overview of our AV-RIR learning method: Given the input reverberant speech Sr from any source environment &s, the primary task of AV-RIR is to estimate the room impulse response RZR by separating it from the clean speech Sc (see Eq. 1). The input Sp is first encoded using a Reverberant Speech Encoder Ex. The latent output Sz is then passed to two different encoders in two different branches. While one of these branches solves the RIR estimation task, the other solves the speech dereverberation task by estimating Sc. Outputs from both the Speech Dereverberation Encoder Szc and RIR Encoder Rg are fused with ResNet-18 encodings from the panoramic image Tp and Geo-Mat feature Zg respectively. The output latent multi-modal encodings Zzp and Zzq are then passed to a trainable Residual Vector Quantization module (RVQ), which quantizes Fs to latent codes Os, and Fr to latent codes Or. Finally, the HiFi-GAN vocoder decodes the enhanced speech Sc from Qs and the RIR decoder decodes estimated early components of RIR Res from Op which are used to calculate losses for training. At inference time, our CRIP retrieves an RIR from a database DS and is used to improve late reverberation in the estimated RIR. Finally, post addition, the final estimated RIR is convolved with any Sc to make it sound like it was uttered in Eg.

Figure 3. The computation pipeline of Geo-Mat feature map. The first two channels of the Geo-Mat feature (Zg) comprise the absorption coefficients (AC) of each acoustic material. The third channel comprises the depth map. We illustrate objects in the en- vironment having similar AC with similar colors: chairs and furni- ture with similar materials are represented in light blue, painting, and wall pictures with similar materials are represented in yellow, and the rest in grey. More details on the method to obtain AC is described in Section 3.3.

Figure 4. Illustration of CRIP training. Like CLIP [64], we pro- pose two networks, one to encode a panoramic image and the other to encode the RIR to learn a joint embedding space between both. We use our CRIP-based image-to-RIR retrieval during inference to improve late reverberation in the estimated RIR from AV-RIR.

Table 2. Performance comparison of AV-RIR with audio-only baselines (marked with +) and audio-visual baselines (marked with *) on the SLP tasks on the SoundSpaces dataset. “Reverberant” refers to clean speech convolved with ground-truth RIR. We also report RTE for real-world audio from the AVSpeech dataset.

Table 1. We compare the RIR estimated using our AV-RIR with prior visual-only method (Image2Reverb [81]) and audio-only methods (FiNS [83] and S2IR-GAN [71]). We perform an ablation study to show the benefit of each component of our network.

Table 3. Perceptual Evaluation. Participants find that the rever- berant speech generated using our AV-RIR is closer to GT rever- berant speech when compared to visual-only baselines. onds. We can see that, irrespective of the T¢ 9 and the en- vironment complexity, 56% to 79% of the participants said that the generated speech from AV-RIR closely matched the GT speech.

Figure 5. Qualitative Results. (Left) We show the Geo-Mat feature generated using our approach. The cushion chairs with a similar material absorption property are represented in green. The table and window with similar material are represented in red. (Right) We plot the time-domain representation of the RIRs estimated using prior methods and our approach with the ground truth (GT) RIR (GT: Red, Estimated: Blue). We also report the MSE (Eq. 4), Téo error (RTE), and EDT error (EDT). It can be seen that the RIR estimated using our AV-RIR matches closely with GT RIR when compared with the baseline. Also, we can see that the RIR retrieved from our CRIP has similar late components as the GT RIR. However, the early component of the retrieved RIR (shown in zoom) significantly differs from the GT. Our full AV-RIR pipeline estimates the early components of the RIR using audio-visual features and adds the late component of the RIR from our CRIP to accurately predict the full RIR. The energy decay curve (EDC) depicts the energy remaining in the RIR over time [80]. We can see that the EDC of the late component of RIR estimated from AV-RIR (yellow) matches closely with the GT RIR (purple).

Q2) Listen to the ground truth speech and synthetic speech created using 3 different algorithms (A, B, C). Choose which synthetic speech is close to ground truth speech. Figure 6. User study interface. We created 3 synthetic speech using our AV-RIR, Image2Reverb [81] and Visual Acoustic Matching [13] and asked the participants, which synthetic reverberant speech matches closely with ground-truth reverberant speech. For each question, we randomly shuffle the order of the synthetic reverberant speech from different approaches.