Advancements in machine learning algorithms have had a beneficial impact on
representation learning, classification, and prediction models built using
electronic health record (EHR) data. Effort has been put both on increasing
models' overall performance as well as improving their interpretability,
particularly regarding the decision-making process. In this study, we present a
temporal deep learning model to perform bidirectional representation learning
on EHR sequences with a transformer architecture to predict future diagnosis of
depression. This model is able to aggregate five heterogenous and
high-dimensional data sources from the EHR and process them in a temporal
manner for chronic disease prediction at various prediction windows. We applied
the current trend of pretraining and fine-tuning on EHR data to outperform the
current state-of-the-art in chronic disease prediction, and to demonstrate the
underlying relation between EHR codes in the sequence. The model generated the
highest increases of precision-recall area under the curve (PRAUC) from 0.70 to
0.76 in depression prediction compared to the best baseline model. Furthermore,
the self-attention weights in each sequence quantitatively demonstrated the
inner relationship between various codes, which improved the model's
interpretability. These results demonstrate the model's ability to utilize
heterogeneous EHR data to predict depression while achieving high accuracy and
interpretability, which may facilitate constructing clinical decision support
systems in the future for chronic disease screening and early detection.Comment: in IEEE Journal of Biomedical and Health Informatics (2021
