Impact of ROIs Delineation Strategies on the Performance of Artificial Intelligence-Aided COVID-19 Screening Algorithms

Abstract

Abstract Purpose To compare the effects of different annotation strategies on the performance of Radiomics models in identifying COVID-19. Methods A total of 775 CT scans were retrospectively collected from 5 hospitals in China between Jan 19 and Mar 26, 2020, including 310 COVID-19 scans and 465 other community-acquired pneumonia (CAP) scans. Coarse annotation which labels the major lesions on certain CT slides and fine annotation which delineates the contour of lesions on each slide was performed on CT images. Four feature selection methods and four machine learning algorithms were then applied in combinations to develop Radiomics models on different sizes of datasets, including small (56 CT scans) and large (56 + 489 CT scans). Model performance was evaluated by ROC curve, PR curve, and other diagnostic metrics on an external test set. Statistical analyses were performed with Chi-square tests and DeLong Test; P < 0.05 was considered statistically significant. Results Differences between coarse and fine annotated data were quantitatively analyzed by a Dice index of 0.689, an average Hausdorff distance of 3.7%, and an average volume difference of 5.8%. Inaccurate segmentations were observed in coarse annotated images, including relatively smaller ROI and missed delineation of ground-glass opacity. In addition, more abundant features were extracted from fine annotated images in categories of FirstOrder, GLSZM, and GLCM features. With regard to model performance, fine annotation enabled an over better performance of Radiomics models while enlarged dataset size could remedy the influence of coarse annotation. Meanwhile, models trained on large datasets displayed more stable performance on all selection methods and algorithm combinations. Among them, L1-LR-MLP was selected as the optimal combination for modeling. In particular, SDFine, SDRough, LDFine, and LDRough datasets developed L1-LR-MLP models achieved the AUROC of 0.864,0.707, 0.904, and 0.899, and the AUPR of 0.888, 0.714, 0.934 and 0.896, respectively, on the external dataset. Conclusions Fine annotation generally enables a better model performance in the identification of COVID-19 while the efficient coarse annotation strategy could also be applied to achieve the equivalent diagnostic performance by expanding the training dataset, especially in urgent scenarios. L1-LR-MLP displayed great potential to be applied for establishing COVID-19 identification models.