An Exploratory Analysis of ChatGPT Compared to Human Performance With the Anesthesiology Oral Board Examination: Initial Insights and Implications

Abstract

BACKGROUND: Chat Generative Pre-Trained Transformer (ChatGPT) has been tested and has passed various high-level examinations. However, it has not been tested on an examination such as the American Board of Anesthesiology (ABA) Standardized Oral Examination (SOE). The SOE is designed to assess higher-level competencies, such as judgment, organization, adaptability to unexpected clinical changes, and presentation of information. METHODS: Four anesthesiology fellows were examined on 2 sample ABA SOEs. Their answers were compared to those produced by the same questions asked to ChatGPT. All human and ChatGPT responses were transcribed, randomized by module, and then reproduced as complete examinations, using a commercially available software-based human voice replicator. Eight ABA applied examiners listened to and scored the topic and modules from 1 of the 4 versions of each of the 2 sample examinations. The ABA did not provide any support or collaboration with any authors. RESULTS: The anesthesiology fellow’s answers were found to have a better median score than ChatGPT, for the module topics scores (P = .03). However, there was no significant difference in the median overall global module scores between the human and ChatGPT responses (P = .17). The examiners were able to identify the ChatGPT-generated answers for 23 of 24 modules (95.83%), with only 1 ChatGPT response perceived as from a human. In contrast, the examiners thought the human (fellow) responses were artificial intelligence (AI)-generated in 10 of 24 modules (41.67%). Examiner comments explained that ChatGPT generated relevant content, but were lengthy answers, which at times did not focus on the specific scenario priorities. There were no comments from the examiners regarding Chat GPT fact “hallucinations.” CONCLUSIONS: ChatGPT generated SOE answers with comparable module ratings to anesthesiology fellows, as graded by 8 ABA oral board examiners. However, the ChatGPT answers were deemed subjectively inferior due to the length of responses and lack of focus. Future curation and training of an AI database, like ChatGPT, could produce answers more in line with ideal ABA SOE answers. This could lead to higher performance and an anesthesiology-specific trained AI useful for training and examination preparation.