A recent study, published in JAMA Ophthalmology, has suggested that advanced artificial intelligence (AI) tools could play an important role in diagnosing and managing glaucoma and retina disorders.
Researchers from New York Eye and Ear Infirmary of Mount Sinai (NYEE) matched the knowledge of ophthalmic specialists against the capabilities of the latest generation AI system, GPT-4 (Generative Pre-Training–Model 4) from OpenAI designed to replicate human-level performance.
Sophisticated AI tools could revolutionise diagnosis and treatment tools through the accuracy and comprehensiveness of their large language model (LLM)-generated responses. Ophthalmology, with its high volume of often complex patients, is said to be an ideal field for AI, giving specialists more time to practice evidence-based medicine.
“The performance of GPT-4 in our study was quite eye-opening,” Dr Andy Huang, ophthalmology resident at NYEE and lead author of the study, said.
“We recognised the enormous potential of this AI system from the moment we started testing it and were fascinated to observe that GPT-4 could not only assist but in some cases match or exceed, the expertise of seasoned ophthalmic specialists.”
As part of the study, the NYEE team recruited 12 attending specialists and three senior trainees from the Department of Ophthalmology at the Icahn School of Medicine at Mount Sinai.
A basic set of 20 questions (10 each for glaucoma and retina) from the American Academy of Ophthalmology’s list of commonly asked questions by patients was randomly selected, along with 20 deidentified patient cases from Mount Sinai-affiliated eye clinics.
Responses from both the GPT-4/AI system and human specialists were then statistically analysed and rated for accuracy and thoroughness using a Likert scale, which is commonly used in clinical research to score responses.
The results showed that AI matched or outperformed human specialists in both accuracy and completeness of its medical advice and assessments. More specifically, AI demonstrated superior performance in response to glaucoma questions and case-management advice, while reflecting a more balanced outcome in retina questions, where AI matched humans in accuracy but exceeded them in completeness.
“AI was particularly surprising in its proficiency in handling both glaucoma and retina patient cases, matching the accuracy and completeness of diagnoses and treatment suggestions made by human doctors in a clinical note format,” Dr Louis R. Pasquale, Deputy Chair for ophthalmology research for the Department of Ophthalmology, and senior author of the study, said.
“Just as the AI application Grammarly can teach us how to be better writers, GPT-4 can give us valuable guidance on how to be better clinicians, especially in terms of how we document findings of patient exams.”
Huang said while additional testing is required, he believes this work points to a promising future for AI in ophthalmology.
“It could serve as a reliable assistant to eye specialists by providing diagnostic support and potentially easing their workload, especially in complex cases or areas of high patient volume,” he said.
“For patients, the integration of AI into mainstream ophthalmic practice could result in quicker access to expert advice, coupled with more informed decision-making to guide their treatment.”
AI as an adjunct
Despite this, ChatGPT version 3.5 has been shown to be a potential adjunct to patient education but is not sufficient without human medical supervision.
In a recent Nature study, the chatbot demonstrated positive scores when asked about the description, diagnosis and treatment of five diseases from eight subspecialties of ophthalmology.
The responses were graded by comparing them to the American Academy of Ophthalmology (AAO) guidelines for patients, with the chatbot providing incomplete, incorrect, and potentially harmful information about common ophthalmic conditions.
This was defined as the recommendation of invasive procedures or other interventions with potential for adverse sequelae which are not supported by the AAO for the disease in question.
More reading
AI models predict long-term visual acuity for high myopia
Bringing telehealth and artificial intelligence into real-world ophthalmology practice
AI in eyecare: myths and attitudes
