Performance of GPT-4v in answering the Japanese otolaryngology board certification examination questions: An evaluation study (Preprint)

Abstract

Artificial intelligence (AI) models can learn from medical literature and clinical cases and generate answers that rival human experts. However, challenges remain in the analysis of complex data containing images and diagrams.

We aimed to assess the answering capabilities and accuracy of ChatGPT-4 Vision (GPT-4v) for a set of 100 questions, including image-based questions, from the 2023 otolaryngology board certification examination.

Answers to 100 questions from the 2023 otolaryngology board certification examination, including image-based questions, were generated using GPT-4v. The accuracy rate was evaluated using different prompts, and the presence of images, clinical area of the questions, and variations in the answer content were examined.

The accuracy rate for text-only input was, on average, 24.7% but improved to 47.3% with the addition of English translation and prompts. (p < 0.001) The average non-response rate for text-only input was 46.3%; this decreased to 2.7% with the addition of English translation and prompts. (p < 0.001) The accuracy rate was lower for image-based questions than for text-only questions across all types of input, with a relatively high non-response rate. General questions and questions from the fields of head and neck allergies and nasal allergies had relatively high accuracy rates, which increased with the addition of translation and prompts. In terms of content, questions related to anatomy had the highest accuracy rate. For all content types, the addition of translation and prompts increased the accuracy rate. As for the performance based on Image-Based question, the average of correct answer rate with text-only input was 30.4%, with text-plus-image input was 41.3% (p = 0.016).

Examination of AI’s answering capabilities for the otolaryngology board certification exam improves our understanding of its potential and limitations in this field. Although the improvement was noted with the addition of translation and prompts, the accuracy rate for image-based questions was lower than that for text-based questions, suggesting room for improvement in GPT-4v at this stage. Furthermore, text-plus-image input answer higher rate in Image-Based question. Our findings imply the usefulness and potential of GPT-4v in medicine; however, future consideration of safe usage methods is needed.