Affiliation:
1. The University of Jordan
2. Applied Science Private University
3. University Medicine Essen (AöR)
Abstract
Abstract
Background
Assessment of artificial intelligence (AI)-based models across languages is crucial to ensure equitable access and accuracy of information in multilingual contexts. This study aimed to compare AI model efficiency in English and Arabic for infectious disease queries.
Methods
The study employed the METRICS checklist for the design and reporting of AI-based studies in healthcare. The AI models tested included ChatGPT-3.5, ChatGPT-4, Bing, and Bard. The queries comprised 15 questions on HIV/AIDS, tuberculosis, malaria, COVID-19, and influenza. The AI-generated content was assessed by two bilingual experts using the validated CLEAR tool.
Results
In comparing AI models' performance in English and Arabic for infectious disease queries, variability was noted. English queries showed consistently superior performance, with Bard leading, followed by Bing, ChatGPT-4, and ChatGPT-3.5 (P = .012). The same trend was observed in Arabic, albeit without statistical significance (P = .082). Stratified analysis revealed higher scores for English in most CLEAR components, notably in completeness, accuracy, appropriateness, and relevance, especially with ChatGPT-3.5 and Bard. Across the five infectious disease topics, English outperformed Arabic, except for flu queries in Bing and Bard. The four AI models' performance in English was rated as “excellent”, significantly outperforming their “above-average” Arabic counterparts (P = .002).
Conclusions
Disparity in AI model performance was noticed between English and Arabic in response to infectious disease queries. This language variation can negatively impact the quality of health content delivered by AI models among native speakers of Arabic. This issue is recommended to be addressed by AI developers, with the ultimate goal of enhancing health outcomes.
Publisher
Research Square Platform LLC
Reference51 articles.
1. UNESCO. World Arabic Language Day. 25. December 2023, 2023. Updated 18 December 2023. Accessed 25 December 2023, 2023. https://www.unesco.org/en/world-arabic-language-day.
2. Alfakhry GM, Dashash M, Jamous I. Native Arabic Language Use Acceptability and Adequacy in Health Professional Instruction: Students and Faculty’s Perspectives. Health Professions Education. 2020/12/01/ 2020;6(4):454–464. doi:10.1016/j.hpe.2020.06.004.
3. Implications of Language Barriers for Healthcare: A Systematic Review;Al Shamsi H;Oman Med J Mar,2020
4. Language Barriers in Understanding Healthcare Information: Arabic-Speaking Students' Comprehension of Diabetic Questionnaires in Arabic and English Languages;Gazzaz ZJ;Cureus Oct,2023
5. Big Three Infectious Diseases: Tuberculosis, Malaria and HIV/AIDS;Makam P;Curr Top Med Chem,2021