MF-AV-Net: an open-source deep learning network with multimodal fusion options for artery-vein segmentation in OCT angiography

Abstract

This study is to demonstrate the effect of multimodal fusion on the performance of deep learning artery-vein (AV) segmentation in optical coherence tomography (OCT) and OCT angiography (OCTA); and to explore OCT/OCTA characteristics used in the deep learning AV segmentation. We quantitatively evaluated multimodal architectures with early and late OCT-OCTA fusions, compared to the unimodal architectures with OCT-only and OCTA-only inputs. The OCTA-only architecture, early OCT-OCTA fusion architecture, and late OCT-OCTA fusion architecture yielded competitive performances. For the 6 mm×6 mm and 3 mm×3 mm datasets, the late fusion architecture achieved an overall accuracy of 96.02% and 94.00%, slightly better than the OCTA-only architecture which achieved an overall accuracy of 95.76% and 93.79%. 6 mm×6 mm OCTA images show AV information at pre-capillary level structure, while 3 mm×3 mm OCTA images reveal AV information at capillary level detail. In order to interpret the deep learning performance, saliency maps were produced to identify OCT/OCTA image characteristics for AV segmentation. Comparative OCT and OCTA saliency maps support the capillary-free zone as one of the possible features for AV segmentation in OCTA. The deep learning network MF-AV-Net used in this study is available on GitHub for open access.