An Interpretable Vision Model Integrating Radiomics for Precision Oncology Diagnostics Using Multi-Modal Medical Imaging

Abstract

The use of deep learning models in clinical oncology remains underutilized, despite their potential in cancer diagnosis. The current methods rely on either traditional radiomics features, whose representational power is limited, or opaque deep neural networks that cannot provide explanations useful to clinicians. This study addresses the interpretability-performance trade-off by introducing a novel hybrid architecture that synergistically combines convolutional neural networks with radiomics biomarkers through attention-based fusion mechanisms. Our framework takes multi-modal imaging (CT, MRI, and PET) data (2,847 patients with 5 different cancer types). It operates through a two-stream architecture, specifically enforcing a correlation-based constrained relationship and sparsity-based regularization between the deep learning and radiomics pathways. The model employs trained gating decisions, automatic feature selection, and cross-modal attention as an ad hoc weighting mechanism to produce an accurate forecast and a human-comprehensible explanation. The results of the experiments show improved performance, with an area under the ROC curve of 0.947, representing 8.4% and 2.6% improvements over pure radiomics methods and standard deep learning models, respectively. In older people, as validated by five expert radiologists, the generated explanations received a high relevance rating (78.4% rated 4-5 on a 5-point scale) and demonstrated high inter-rater agreement (α = 0.68). The study made contributions, including a learnable architecture with interpretability constraints built into its objective, direct measurement of individual features through quantified attention weights consistent with radiological intuitions, detection consistency across a variety of cancer types, and providing generalizability. It demonstrated that improvements in interpretability do not affect predictive accuracy. Therefore, this study developed a reliable AI in oncology by offering an empirical roadmap for engineering high-performance diagnostic environments that meet clinical accountability and transparency standards.