External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients

Abstract

Objective To assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6–12 hours, >12–18 hours, >18–24 hours, and >24–30 hours in the future. Design Prognostic study comparing multi-institutional CI-D models’ performance in a single-site electronic health record dataset to an institution-specific CI-D model developed using identical variables and modelling methods. The institution did not participate in the multi-institutional dataset. Participants All pediatric inpatients admitted from January 1st 2018 –February 29th 2020 through the emergency department. Main outcome(s) and measure(s) The main outcome was inpatient care in routine or ICU care locations. Results A total of 29,037 pediatric hospital admissions were included, with 5,563 (19.2%) admitted directly to the ICU, 869 (3.0%) transferred from routine to ICU care, and 5,023 (17.3%) transferred from ICU to routine care. Patients had a median [IQR] age 68 months (15–157), 47.5% were female and 43.4% were black. The area under the receiver operating characteristic curve (AUROC) for the multi-institutional CI-D models applied to a single-site test dataset was 0.493–0.545 and area under the precision-recall curve (AUPRC) was 0.262–0.299. The single-site CI-D models applied to an independent single-site test dataset had an AUROC 0.906–0.944 and AUPRC range from 0.754–0.824. Accuracy at 0.95 sensitivity for those transferred from routine to ICU care was 72.6%-81.0%. Accuracy at 0.95 specificity was 58.2%-76.4% for patients who transferred from ICU to routine care. Conclusion and relevance Models developed from multi-institutional datasets and intended for application to individual institutions should be assessed locally and may benefit from re-development with site-specific data prior to deployment.