Abstract: TH-PO009
Artificial Intelligence Language Processing Models in Literature Reviews for Nephrology
Session Information
- AI, Digital Health, Data Science - I
November 02, 2023 | Location: Exhibit Hall, Pennsylvania Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Augmented Intelligence, Digital Health, and Data Science
- 300 Augmented Intelligence, Digital Health, and Data Science
Authors
- Suppadungsuk, Supawadee, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Krisanapan, Pajaree, Mayo Clinic Minnesota, Rochester, United States
- Tangpanithandee, Supawit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Garcia Valencia, Oscar Alejandro, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Thongprayoon, Charat, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Kashani, Kianoush, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
Background
Literature reviews are a valuable tool for summarizing and evaluating the available evidence in various medical fields, including Nephrology. However, identifying and selecting relevant literature can be time-consuming for clinicians and researchers. ChatGPT is a novel artificial intelligence (AI) language model renowned for its exceptional ability to generate human-like responses across various tasks. However, whether ChatGPT can effectively assist medical professionals in identifying relevant literature is unclear. Therefore, this study aimed to assess the effectiveness of ChatGPT in identifying references to literature reviews in Nephrology.
Methods
We keyed the prompt "What are the references to literature reviews in Nephrology, with subgroup analysis in specific areas?" into ChatGPT (03/23 Version). We selected all provided results by ChatGPT and assessed them for existence, relevance, and author/link correctness. We recorded each resource's citations, references, authors' names, and links. The relevance of each resource was assessed by its citation index on Google Scholar.
Results
Of the total 511 references in Nephrology literature, only 319 (62.43%) of the references provided by ChatGPT existed, while 30.3% did not exist, and in 7.3% of recommendations, they were incomplete. Additionally, 208 (40.7%) of the listed authors were found to be incorrect, and the digital object identifier (DOI) was inaccurate in 357 (69.86%) of the references. Moreover, among those with a link provided, the link was correct in only 108 (21.14%) cases, and 7.4% of the references were irrelevant. Notably, analysis of specific topics in electrolyte, hemodialysis, and kidney stones found that >60% of the references were inaccurate or misleading, with less reliable authors and links provided by ChatGPT.
Conclusion
Based on our findings, the use of ChatGPT as a sole resource for identifying references to literature reviews in Nephrology is not recommended. Future studies could explore ways to improve AI language models' performance in identifying relevant Nephrology literature.