Multimodal Machine Learning for 30-Days Post-Operative Mortality Prediction of Elderly Hip Fracture Patients

Hip fractures in the elderly are a major health care problem in the society. In the clinic, it is important for medical specialists to identify high-risk patients to assist them in the decision-making process in choosing the right treatment. In this paper, we propose a multimodal machine learning model for prediction of 30-days mortality of elderly hip fracture patients. The paper addresses both the clinical problem of identifying high-risk patients and the specific risks involved, as well as the technical problem of how to fuse information from different modalities as input to one predictive model. Our model uses features from three modalities: hip X-ray images, chest X-ray images and structured data and fuses them based on early fusion and late fusion techniques for the prediction task. Our model uses a convolutional neural network to extract features from the chest and hip images before combining them with the structured data. The fused features are further passed through a fusion classifier for the final prediction. The proposed model outperforms a replicated version of Almelo Hip Fracture Score (AHFS-a) with an AUC score of 0.786 vs 0.717. Finally, by the analysis of feature importance, we found that chest X-ray images contain important signs to predict the 30-days mortality of elderly hip fracture patients. We also found that structured and chest X-ray modalities were more important in predicting high-risk patients as compared to hip X-ray modality, though the final results on the test set show that structured, hip and chest X-ray modalities together are needed to get the best performance for predicting 30-days mortality. Further, we achieved the best performance using early fusion with random forest technique, though late fusion achieved a competitive performance