DO CHATBOTS PROVIDE RELIABLE INFORMATION ABOUT MOBILE APPS IN AUDIOLOGY?

Abstract

IntroductionIn light of the growing interest in utilizing AI for information retrieval, assessing the accuracy and reliability of tools such as chatbots is essential. This study aims to evaluate the efficacy of chatbots in providing accurate information about mobile applications (apps) in the field of audiology.Material and methodsThe responses of the Gemini and ChatGPT chatbots to eight open-ended questions posed in Polish and English were compared. Each answer was assessed for correctness.ResultsGemini_ENG achieved the highest correctness with a score of 5 points (62.5%), while ChatGPT_PL scored 2 points (25%), and both Gemini_PL and ChatGPT_ENG scored 1 point (12.5%). Chatbots were most effective in recommending apps for older adults, with three of the four chatbots providing accurate recommendations. However, they struggled when asked to recommend apps for non-English speakers, to describe apps, or to provide direct links, with none of them scoring points in these areas.ConclusionsChatbots are currently unreliable sources of information about audiology apps. Depending on the language, there is significant variability in response accuracy. A good example is that Gemini_ENG performed far better than Gemini_PL. A major issue for all of them was the frequent fabrication of data, including the creation of nonexistent app names and incorrect links.