Kidney Week

Abstract: TH-OR23

Automated Extraction of Kidney Failure Concepts from Clinical Notes Using Artificial Intelligence

Session Information

• Augmented Intelligence and Digital Health Advances
  October 24, 2024 | Location: Room 2, Convention Center
  Abstract Time: 05:10 PM - 05:20 PM

Category: Augmented Intelligence, Digital Health, and Data Science

• 300 Augmented Intelligence, Digital Health, and Data Science

Authors

• Harris, Luke, Galway University Hospitals, Galway, Galway, Ireland

Luke Harris, MBChB

• Buosi, Samuele, Insight Data Analytics, University of Galway, Galway, Ireland

Samuele Buosi

• Monahan, Sarah, Galway University Hospitals, Galway, Galway, Ireland

Sarah Monahan, MBChB

• Krewer, Finn, Galway University Hospitals, Galway, Ireland

Finn Krewer, PhD

• Timilsina, Mohan, Insight Data Analytics, University of Galway, Galway, Ireland

Mohan Timilsina, PhD

• Corcoran, Niamh, Galway University Hospitals, Galway, Galway, Ireland

Niamh Corcoran, MD, DrMed

• Deacon, Nina, Galway University Hospitals, Galway, Galway, Ireland

Nina Deacon, MBChB

• Wrynn, Michael, Galway University Hospitals, Galway, Galway, Ireland

Michael Wrynn, MBChB

• Farnan, Richard, Galway University Hospitals, Galway, Galway, Ireland

Richard Farnan, PharmD, MBBS, MSc

• Reddan, Donal N., Galway University Hospitals, Galway, Galway, Ireland

Donal N. Reddan, MBChB

• Mellotte, George S., HSE National Renal Office, Dublin, Ireland

George S. Mellotte, MD

• Conlon, Peter J., Beaumont Hospital, Dublin, Dublin, Ireland

Peter J. Conlon, MD, MBBS, MBChB

• Judge, Conor S., Galway University Hospitals, Galway, Ireland

Conor S. Judge, MBBS

Background

The Kidney Disease Clinical Patient Management System (KDCPMS) contains structured (e.g. date of birth) and unstructured or free-text data fields (clinic notes). Unstructured data is challenging to use for audit, QI and research.

Methods

Bidirectional Encoder Representations from Transformers (BERT) are a transformer-based machine learning architecture useful for Natural Language Processing (NLP). Galway University Hospital data were extracted from KDCPMS, anonymised, pre-processed, annotated as Diabetes-Y/N and IgA Nephropathy-Y/N by physicians using Prodigy annotation software, and NLP model trained using a modified BERT algorithm running in HSE Integrated Information Services. We compared the NLP classification with an algorithmic classification using ICD-10/EDTA codes, medication and HbA1c and a retrospective chart review by physicians (gold standard).

Results

43314 patient notes were extracted. 10000 notes were annotated by 5 physicians. Classification was performed on 33314 notes. 955 patients were classified as Diabetes-Y, 4190 as Diabetes-N, 269 as IgA Neph-Y and 2609 as IgA Neph-N. 1744 patients were then classified algorithmically as Diabetes-Y and 214 patients were classified algorithmically as IgA Neph-Y. For Diabetes, the NLP method achieved an accuracy of 79.0% (95% CI [77.4%,80.6%]), a sensitivity of 73.3% and specificity of 81.3%. Precision was 61.1%, F1 score 66.7% and macro F1 score 0.757. The algorithmic method outperformed the NLP method with an accuracy of 98.1% (95% CI [97.5%,98.6%]), sensitivity 96.7% and specificity 98.7%. Precision was 96.7%, F1 score 96.7% and macro F1 score 0.977. For IgA Nephropathy, the NLP method demonstrated high accuracy of 96.2% (95% CI [95.1%,97.2%]), with sensitivity 83.3% and specificity 97.8%. Precision was 83.3%, F1 score 83.3% and macro F1 score 0.906. The algorithmic method achieved perfect scores across all metrics, with accuracy, sensitivity, specificity, precision, F1 score, and macro F1 score all at 100.0% (95% CI [99.9%,100%]).

Conclusion

These results highlight the superior performance of the algorithmic method over the NLP method, particularly in terms of accuracy and sensitivity. However the NLP method demonstrated substantial potential, especially for IgA Nephropathy.

Funding

• Government Support – Non-U.S.