Kidney Week

Abstract: SA-PO004

Self-Captured Images Recognition by Artificial Intelligence (AI) in Common Nephrology Medications: A Comparative Analysis of ChatGPT-4 and Claude 3 Opus

Session Information

• Augmented Intelligence, Large Language Models, and Digital Health
  October 26, 2024 | Location: Exhibit Hall, Convention Center
  Abstract Time: 10:00 AM - 12:00 PM

Category: Augmented Intelligence, Digital Health, and Data Science

• 300 Augmented Intelligence, Digital Health, and Data Science

Authors

• Sheikh, M. Salman, Mayo Clinic Minnesota, Rochester, Minnesota, United States

M. Salman Sheikh, MD

• Dreesman, Benjamin, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Benjamin Dreesman, PharmD

• Barreto, Erin F., Mayo Clinic Minnesota, Rochester, Minnesota, United States

Erin F. Barreto, PharmD, MS, FASN

• Miao, Jing, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Jing Miao, MD, PhD, FASN

• Thongprayoon, Charat, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Charat Thongprayoon, MD, FASN

• Qureshi, Fawad, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Fawad Qureshi, MD, FASN

• Craici, Iasmina, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Iasmina Craici, MD

• Kashani, Kianoush, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Kianoush Kashani, MD, MS, FASN

• Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Wisit Cheungpasitporn, MD, FASN

Background

Accurate medication identification is crucial in nephrology, where patients with kidney diseases often require complex regimens. Artificial intelligence (AI) shows the potential to assist medication management, particularly when patients cannot remember medication names or labels are unavailable. Self-captured image recognition by AI could enhance medication safety and reduce errors for these patients by streamlining medication reconciliation, improving adherence, and supporting clinical decision-making. This study evaluates ChatGPT-4 and Claude 3 Opus in identifying common medications in nephrology from self-captured images.

Methods

Twenty-five medications commonly encountered in nephrology were randomly selected and systematically photographed using an iPhone 13 Pro Max, capturing both front and back views. The images were analyzed by ChatGPT-4 and Claude Opus with the inquiry, "What is this medication?" The percentage of correct identifications determined the accuracy of each model, and a two-tailed Fisher's exact test was used to compare the accuracy rates between the two models, with a P-value < 0.05 considered statistically significant.

Results

ChatGPT-4 showed a robust performance, correctly identifying twenty-two out of twenty-five medications, with an accuracy rate of 88%. Errors occurred in identifying hydrochlorothiazide, nifedipine, and spironolactone due to challenging imprints. Conversely, Claude 3 Opus only accurately identified one medication – trazodone. It misidentified the remaining twenty-four medications, achieving a notably lower accuracy rate of 4%, even though it correctly read the imprint of eighteen of these. ChatGPT-4 significantly outperformed Claude 3 Opus (P<0.001).

Conclusion

The study highlights the superiority of ChatGPT-4 over Claude 3 Opus in the task of identifying common medications from self-captured pictures. While ChatGPT-4 demonstrated high accuracy, Claude 3 Opus showed significant limitations, underscoring its inadequacy for clinical application in this context. These findings advocate for further development and integration of robust image recognition technology to support accurate medication management in nephrology.