Supervised fusion content-based framework for breakdown detection in task-oriented conversational systems

Abstract

Conversational agents (CAs) have been widely used for many domains, such as healthcare, education, and business. One main category of CAs is task-oriented CAs, which aim to help users to complete a set of specific tasks. However, task-oriented CAs can fail to answer the user’s question, which can lead to a breakdown in the dialogue (when it is not possible to complete a conversation with a CA). Breakdown detection is an essential task for developing better CAs. Several related studies have focused on breakdown detection using different sets of features, for example, topic transition, word-based similarity and clustering; but, the existing studies develop features mainly from the system’s outputs or user’s inputs, whereas the features can be extracted from both sides, as well as from the interaction between them. Therefore, in this work, we developed a new supervised fusion machine learning (ML) model that combines the prediction from two machine learning algorithms for breakdown detection CAs services system. We developed features from different groups focusing on both the user input and the system response. Then we select the optimal combined features. The features are based on sentence similarity, sentiment features, and count-based features. The developed fusion model is mainly based on the two best performances of the single classifiers (SVM and RF). We explore several single ML algorithms using different sets of features and the combined features. To verify the effectiveness of the proposed fusion model, we compared the proposed models against baseline methods using four sets of data. We conclude that the proposed fusion model with the combined features outperforms the baselines and all other models in terms of prediction accuracy and f-score measures.