Table of Links Abstract Abstract Introduction Methods Results Discussion Conclusions, Acknowledgments, and References Introduction Introduction Methods Methods Results Results Discussion Discussion Conclusions, Acknowledgments, and References Conclusions, Acknowledgments, and References 2. Methods This retrospective study was approved by the institutional review board (IRB) of INSTITUTION1, INSTITUTION2, and INSTITUTION3 with waiver of informed consent by each study participant. The proposed clinical risk assessment algorithm is a DL-based framework incorporating multimodal neural networks embedded within an image analysis backbone model to predict survival outcomes (Figure 1). (Figure 1). We aimed to develop four deep learning-based prediction models capable of PE survival prediction for performance comparison: Model 1: Uses only CTPA imaging data (deep imaging) Model 2: Uses the 11 clinical variables considered within the PESI framework (deep clinical); notably, no manual weighting or scoring are applied to the data (no PESI score calculation)- the model is allowed to interpret these variables independently. Model 3: Incorporates both CTPA imaging data and aforementioned clinical variables (deep multimodal) Model 4: Combines the deep multimodal model with PESI score (deep PESI-fused) Model 1: Uses only CTPA imaging data (deep imaging) Model 1: Uses only CTPA imaging data (deep imaging) Model 2: Uses the 11 clinical variables considered within the PESI framework (deep clinical); notably, no manual weighting or scoring are applied to the data (no PESI score calculation)- the model is allowed to interpret these variables independently. Model 2: Uses the 11 clinical variables considered within the PESI framework (deep clinical); notably, no manual weighting or scoring are applied to the data (no PESI score calculation)- the model is allowed to interpret these variables independently. Model 3: Incorporates both CTPA imaging data and aforementioned clinical variables (deep multimodal) Model 3: Incorporates both CTPA imaging data and aforementioned clinical variables (deep multimodal) Model 4: Combines the deep multimodal model with PESI score (deep PESI-fused) Model 4: Combines the deep multimodal model with PESI score (deep PESI-fused) 2.1. Clinical and Imaging Data Acquisition 2.1. Clinical and Imaging Data Acquisition Retrospective chart review identified patients between March 2015 and February 2019 meeting the following inclusion criteria: confirmed PE on CTPA, transthoracic/transesophageal echocardiography (TTE, TEE) within two months of diagnosis. 918 patients were identified with corresponding clinical reports and CTPA series (3,978 CTPA acquisitions total). CTPA acquisitions for each patient were from the same date, consisting of different axial resolutions- the larger number of 3,978 represents the total sum of individual CTPA image acquisitions from all patients in our dataset. From electronic medical records, the 11 clinical variables considered in PESI were collected: age, sex, heart rate, systolic blood pressure, respiratory rate, temperature, mental status, previous pulmonary embolism or deep vein thrombosis, cancer, congestive heart failure, and chronic lung disease. Age was normalized within the overall dataset, and the remaining variables were binarized. Within the dataset, 94 patients (10.2%) had missing variables that required imputation using median values for binary variables and normalization for decimal variables. Mortality and hemodynamic collapse (as defined in PEITHO trial[17]) were recorded for applicable patients. PESI score was calculated for each patient using the aforementioned clinical variables. Regarding ground truth for performance evaluation, clinical patient outcomes such as mortality and recorded time in electronic medical records were used to evaluate model performance with concordance index (c-index). For censored patients, the last recorded time point in the system was used as the cutoff time. C-index was employed to evaluate model performance by quantifying the concordance between predicted risk scores and observed survival times, taking censoring into consideration. This measures the likelihood of correctly ranking the survival times of pairs of individuals. 2.2. Image Preprocessing and PE Detection 2.2. Image Preprocessing and PE Detection A U-Net model was used for lung segmentation on CT to obtain lung region masks.[18] The corresponding lung regions of the respective CTPA images were extracted with a slice thickness of 1.25mm and scaled to 512x512 pixels. The entire image volume with N slices was saved as a Nx512x512 array. Hounsfield units of all slices were clipped to the range [−1000, 900] and zero-centered. A robust trained PE detection model, PENet, was employed as the backbone for our image-based survival prediction model.[12] PENet automatically analyzes and identifies the most indicative features of PE within CTPAs. Across window-level predictions, the highest PE probability was used to determine a patient-level classification score. For each patient, the window-level prediction with the highest PE probability was selected, with the 2048 output features from the last convolutional layer designated as imaging features. In cases where a single patient had multiple CTPA acquisitions, the acquisitions were analyzed together to output a single set of optimal imaging features, resulting in each patient having only one corresponding set of imaging features. 2.3. Learning-based Survival Analysis Framework 2.3. Learning-based Survival Analysis Framework Cross-modal fusion CoxPH models employed the two modal risk predictions to train a fused survival prediction model, incorporating time-to-event evaluation. This semi-parametric fusing enhances predictive capabilities by combining information from multiple modalities, offering a robust and nuanced understanding of survival outcomes. To assess performance of the multimodal learning-based model over PESI, a PESI-fused CoxPH model was evaluated, combining multimodal features and PESI. To compare relative performance between DL survival prediction and RSF, two single-modal RSFs were trained to model hazard functions, then fused with the same CoxPH models used in the DL survival framework.7 Detailed methods and CLAIM checklist are included in Supplemental Materials. Supplemental Materials. 2.4. Statistical Analysis 2.4. Statistical Analysis The INSTITUTION1 data were divided randomly into training, validation, and internal test sets (7:1:2). The INSTITUTION2/INSTITUTION3 data were consolidated as an external test set, due to relative similarity and smaller size. The survival prediction models underwent training and validation using the INSTITUTION1 training set and validation dataset. The frameworks were then applied to the internal and external test set. Model performance was evaluated by c-index and compared to PESI predictions using the Wilcoxon signed-rank test.[19] Net Reclassification Improvement (NRI) was used to assess performance improvement from each modality in mortality classification prediction.[20] Kaplan-Meier analysis was performed to stratify patients into high- and low-risk groups.[21] CoxPH models were used to analyze risk scores from the multimodal models.[8] Significance level was set to p<0.05. 2.5. RV Dysfunction Factor-risk Analysis 2.5. RV Dysfunction Factor-risk Analysis As right ventricular (RV) dysfunction is an important prognostic factor in PE, we took it into consideration when evaluating the survival framework.[4,22,23] First, a binarized label for RV dysfunction was collected for each patient. This was then incorporated into a factor-risk analysis with multimodal survival predictions. Patients were sorted by the median of multimodal predicted risk. RV dysfunction was then designated as a risk factor. Authors: (1) Zhusi Zhong, BS, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA, Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA, and School of Electronic Engineering, Xidian University, Xi’an 710071, China; (2) Helen Zhang, BS, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (3) Fayez H. Fayad, BA, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (4) Andrew C. Lancaster, BS, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA and Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA; (5) John Sollee, BS, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (6) Shreyas Kulkarni, BS, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (7) Cheng Ting Lin, MD, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA; (8) Jie Li, PhD, School of Electronic Engineering, Xidian University, Xi’an 710071, China; (9) Xinbo Gao, PhD, School of Electronic Engineering, Xidian University, Xi’an 710071, China; (10) Scott Collins, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (11) Colin Greineder, MD, Department of Pharmacology, Medical School, University of Michigan, Ann Arbor, MI, 48109, USA; (12) Sun H. Ahn, MD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (13) Harrison X. Bai, MD, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA; (14) Zhicheng Jiao, PhD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (15) Michael K. Atalay, MD, PhD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA. Authors: Authors (1) Zhusi Zhong, BS, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA, Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA, and School of Electronic Engineering, Xidian University, Xi’an 710071, China; (2) Helen Zhang, BS, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (3) Fayez H. Fayad, BA, a Co-first authors from Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (4) Andrew C. Lancaster, BS, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA and Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA; (5) John Sollee, BS, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (6) Shreyas Kulkarni, BS, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (7) Cheng Ting Lin, MD, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA; (8) Jie Li, PhD, School of Electronic Engineering, Xidian University, Xi’an 710071, China; (9) Xinbo Gao, PhD, School of Electronic Engineering, Xidian University, Xi’an 710071, China; (10) Scott Collins, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (11) Colin Greineder, MD, Department of Pharmacology, Medical School, University of Michigan, Ann Arbor, MI, 48109, USA; (12) Sun H. Ahn, MD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (13) Harrison X. Bai, MD, Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA; (14) Zhicheng Jiao, PhD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA; (15) Michael K. Atalay, MD, PhD, Department of Diagnostic Radiology, Rhode Island Hospital, Providence, RI, 02903, USA and Warren Alpert Medical School of Brown University, Providence, RI, 02903, USA. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv
