External validation of deep learning-based automated detection algorithm for chest radiograph: practical issues in outpatient clinic

Abstract

Background There have been no reports on diagnostic performance of deep learning-based automated detection (DLAD) for thoracic diseases in real-world outpatient clinic. Purpose To validate DLAD for use at an outpatient clinic and analyze the interpretation time for chest radiographs. Material and Methods This is a retrospective single-center study. From 18 January 2021 to 18 February 2021, 205 chest radiographs with DLAD and paired chest CT from 205 individuals (107 men and 98 women; mean ± SD age: 63 ± 8 years) from an outpatient clinic were analyzed for external validation and observer performance. Two radiologists independently reviewed the chest radiographs by referring to the paired chest CT and made reference standards. Two pulmonologists and two thoracic radiologists participated in observer performance tests, and the total amount of time taken during the test was measured. Results The performance of DLAD (area under the receiver operating characteristic curve [AUC] = 0.920) was significantly higher than that of pulmonologists (AUC = 0.756) and radiologists (AUC = 0.782) without assistance of DLAD. With help of DLAD, the AUCs were significantly higher for both groups (pulmonologists AUC = 0.853; radiologists AUC = 0.854). A greater than 50% decrease in mean interpretation time was observed in the pulmonologist group with assistance of DLAD compared to mean reading time without aid of DLAD (from 67 s per case to 30 s per case). No significant difference was observed in the radiologist group (from 61 s per case to 61 s per case). Conclusion DLAD demonstrated good performance in interpreting chest radiographs of patients at an outpatient clinic, and was especially helpful for pulmonologists in improving performance.