Affiliation:
1. City University of Hong Kong, Kowloon, Hong Kong SAR, Hong Kong, China
2. Michigan State University, East Lansing, Michigan, USA
Abstract
Speech enhancement is regarded as the key to the quality of digital communication and is gaining increasing attention in the research field of audio processing. In this paper, we present EarSE, the first robust, hands-free, multi-modal speech enhancement solution using commercial off-the-shelf headphones. The key idea of EarSE is a novel hardware setting---leveraging the form factor of headphones equipped with a boom microphone to establish a stable acoustic sensing field across the user's face. Furthermore, we designed a sensing methodology based on Frequency-Modulated Continuous-Wave, which is an ultrasonic modality sensitive to capture subtle facial articulatory gestures of users when speaking. Moreover, we design a fully attention-based deep neural network to self-adaptively solve the user diversity problem by introducing the Vision Transformer network. We enhance the collaboration between the speech and ultrasonic modalities using a multi-head attention mechanism and a Factorized Bilinear Pooling gate. Extensive experiments demonstrate that EarSE achieves remarkable performance as increasing SiSDR by 14.61 dB and reducing the word error rate of user speech recognition by 22.45--66.41% in real-world application. EarSE not only outperforms seven baselines by 38.0% in SiSNR, 12.4% in STOI, and 20.5% in PESQ on average but also maintains practicality.
Funder
CityU SIRG grant
CityU APRC grant
CityU SRG-Fd grant
Research Grants Council of Hong Kong
CityU MFPRC grant
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Networks and Communications,Hardware and Architecture,Human-Computer Interaction
Reference106 articles.
1. 2023. Krisp. https://krisp.ai/.
2. SPEECH ENHANCEMENT USING AN ADAPTIVE WIENER FILTERING APPROACH
3. Antlion Audio. 2023. Antilion Mod Mic. Retrieved April 6, 2023 from https://antlionaudio.com/collections/microphones/products/modmic-usb
4. Audio-Technica. 2023. ATH-G1WL. Retrieved April 6 2023 from https://www.audio-technica.com/en-us/ath-g1wl
5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).