Abstract
AbstractMultimodal machine learning models are being developed to analyze pathology images and other modalities, such as gene expression, to gain clinical and biological in-sights. However, most frameworks for multimodal data fusion do not fully account for the interactions between different modalities. Here, we present an attention-based fusion architecture that integrates a graph representation of pathology images with gene expression data and concomitantly learns from the fused information to predict patient-specific survival. In our approach, pathology images are represented as undirected graphs, and their embeddings are combined with embeddings of gene expression signatures using an attention mechanism to stratify tumors by patient survival. We show that our framework improves the survival prediction of human non-small cell lung cancers, out-performing existing state-of-the-art approaches that lever-age multimodal data. Our framework can facilitate spatial molecular profiling to identify tumor heterogeneity using pathology images and gene expression data, complementing results obtained from more expensive spatial transcriptomic and proteomic technologies.
Publisher
Cold Spring Harbor Laboratory
Reference37 articles.
1. F. M. Bianchi , D. Grattarola , and C. Alippi . Spectral Clustering with Graph Neural Networks for Graph Pooling. In Proc. ICML, Virtual Event, pages 874–883, 2020.
2. S. Brody , U. Alon , and E. Yahav . How Attentive are Graph Attention Networks?. In International Conference on Learning Representations (ICLR), 2022.
3. Pan-cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning;Cancer Cell,2022
4. R. J. Chen , M. Y. Lu , M. Shaban , C. Chen , T. Y. Chen , D. F. Williamson , and F. Mahmood . Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, pages 339–349. Springer International Publishing, 2021.
5. Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis;IEEE Transactions on Medical Imaging,2022