Kidney Week

Abstract: SA-PO006

Integrating Artificial Intelligence (AI) in Nephrology Workflows: Evaluating ChatGPT's Performance in Triaging Patients to Nephrology Subspecialty Clinics

Session Information

• Augmented Intelligence, Large Language Models, and Digital Health
  October 26, 2024 | Location: Exhibit Hall, Convention Center
  Abstract Time: 10:00 AM - 12:00 PM

Category: Augmented Intelligence, Digital Health, and Data Science

• 300 Augmented Intelligence, Digital Health, and Data Science

Authors

• Koirala, Priscilla, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Priscilla Koirala, MD

• Thongprayoon, Charat, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Charat Thongprayoon, MD, FASN

• Miao, Jing, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Jing Miao, MD, PhD, FASN

• Garcia Valencia, Oscar Alejandro, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Oscar Alejandro Garcia Valencia, MD

• Sheikh, M. Salman, Mayo Clinic Minnesota, Rochester, Minnesota, United States

M. Salman Sheikh, MD

• Craici, Iasmina, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Iasmina Craici, MD

• Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Wisit Cheungpasitporn, MD, FASN

Background

The rising prevalence of kidney-related conditions and aging global populations pose challenges for the healthcare sector, particularly nephrology. Efficient patient referral and treatment processes are crucial, and artificial intelligence offers promise in revolutionizing medical triage. This study evaluates ChatGPT's utility in triaging nephrology cases through simulated real-world scenarios, aiming to enhance decision-making efficiency and accuracy in clinical settings.

Methods

Two nephrologists created 100 simulated cases, encompassing a variety of aspects within nephrology and its intersecting subspecialties, to test ChatGPT 4.0. We assessed ChatGPT's performance in two separate evaluation attempts in March and April 2024. The evaluation focused on two main areas: the appropriateness of nephrology consultations and the accuracy in identifying suitable nephrology subspecialties.

Results

ChatGPT demonstrated high overall accuracy (98.5%) in identifying the need for nephrology consultations across two rounds, achieving 99% on the first attempt and 98% on the second. However, some misreferrals occurred to nephrology subspecialty clinics, such as recommending an Electrolyte Disorders clinic instead of Onconephrology for a cancer patient with worsening renal failure. While ChatGPT's suggestions were reasonable and safe, they sometimes lacked the depth needed to fully prioritize the intertwined oncological impacts. The AI's triage suggestions showed a reasonable level of clinical appropriateness by identifying related nephrological issues, even when the optimal subspecialty choice was missed. The output was grounded in logical clinical reasoning but could better integrate multidisciplinary care approaches for complex, intersecting medical conditions.

Conclusion

ChatGPT demonstrated high accuracy in triaging nephrology cases, providing safe and clinically appropriate recommendations. However, the AI could be enhanced to better integrate multidisciplinary care approaches for patients with complex, intersecting medical conditions. This study highlights the potential of AI in improving medical triage efficiency and accuracy while identifying areas for refinement.