Affiliation:
1. School of Computer and Electronic Information, Guangxi University, Nanning 530004, China
Abstract
In open environments, multimodal sentiment analysis (MSA) often suffers from low-quality data and can be disrupted by noise, inherent defects, and outliers. In some cases, unreasonable multimodal fusion methods can perform worse than unimodal methods. Another challenge of MSA is effectively enabling the model to provide accurate prediction when it is confident and to indicate high uncertainty when its prediction is likely to be inaccurate. In this paper, we propose an uncertain-aware late fusion based on hybrid uncertainty calibration (ULF-HUC). Firstly, we conduct in-depth research on the issue of sentiment polarity distribution in MSA datasets, establishing a foundation for an uncertain-aware late fusion method, which facilitates organic fusion of modalities. Then, we propose a hybrid uncertainty calibration method based on evidential deep learning (EDL) that balances accuracy and uncertainty, supporting the reduction of uncertainty in each modality of the model. Finally, we add two common types of noise to validate the effectiveness of our proposed method. We evaluate our model on three publicly available MSA datasets (MVSA-Single, MVSA-Multiple, and MVSA-Single-Small). Our method outperforms state-of-the-art approaches in terms of accuracy, weighted F1 score, and expected uncertainty calibration error (UCE) metrics, proving the effectiveness of the proposed method.
Funder
National Natural Science Foundation of China
Subject
Electrical and Electronic Engineering,Computer Networks and Communications,Hardware and Architecture,Signal Processing,Control and Systems Engineering
Reference55 articles.
1. Machine learning and deep learning for sentiment analysis across languages: A survey;Mercha;Neurocomputing,2023
2. Zad, S., Heidari, M., Jones, J.H., and Uzuner, O. (2021, January 10–13). A survey on concept-level sentiment analysis techniques of textual data. Proceedings of the 2021 IEEE World AI IoT Congress (AIIoT), Seattle, WA, USA.
3. Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges;Das;ACM Comput. Surv.,2023
4. Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions;Gandhi;Inform. Fusion,2023
5. Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning;Amrani;Proc. AAAI Conf. Artif. Intell.,2021