BACKGROUND
We queried ChatGPT and Google Assistant about amblyopia and compared its answers to the keywords found in the website of American Association for Pediatric Ophthalmology and Strabismus, section for amblyopia. Of the 26 keywords chosen from website: ChatGPT included 11/26 in its responses while Google included 8/26.
OBJECTIVE
Our study investigates ChatGPT-3.5TM and Google assistantTM’s adherence to the American Association for Pediatric Ophthalmology and Strabismus’ (AAPOS) guidelines for patient education of amblyopia.
METHODS
ChatGPT-3.5 was used, available from November 30, 2022. The 4 questions were taken from American Association for Pediatric Ophthalmology and Strabismus’s website (AAPOS.org), specifically its glossary section for amblyopia: 1) What is amblyopia? 2) What causes amblyopia? 3) How is amblyopia treated? 4) What happens if amblyopia is untreated? Approved and selected by Dr. Gloria Wu and Dr. David Lee, the keywords from AAPOS.org were words or phrases that are deemed significant to the education of patients with amblyopia. The “Flesch-Kincaid Grade Level” formula, approved by the US Department of Education, was used to evaluate the reading comprehension level for the ChatGPT, Google Assistant, and AAPOS responses.
RESULTS
In their responses, ChatGPT did not mention the term “ophthalmologist” whereas Google Assistant and AAPOS.org both mentioned the term once, and twice respectively. ChatGPT did, however, use the term “eye doctors” once. From the Flesch- Kincaid test, AAPOS.org reading level was 11.4 ± 2.1, the lowest level. Google displayed 13.1 ± 4.8, the highest required reading level while also having the greatest variation in grade level in its responses. ChatGPT answers scored 12.4 ± 1.1. They were all similar in terms of level of difficulty in reading. For the key words, of the 4 responses, ChatGPT used the keywords 11/26 times, and Google Assistant 8/26 times.
CONCLUSIONS
ChatGPT trains on texts and phrases and generates new sentences vs. Google Assistant automatically copies website links. As ophthalmologists, through our websites and journals, we should consider encoding “see an ophthalmologist” in our websites. While ChatGPT is here to stay, we, as physicians, need to monitor its answers.