Project based learning in Biomedical Data Science using the MIMIC III open dataset

[EN] The subjects Health Information Systems and Telemedicine and Data Quality and Interoperability of the Degree and Master in Biomedical Engineering of the Universitat Politècnica de València, Spain, address learning outcomes related to managing and processing biomedical databases, using health information standards for data capture and exchange, data quality assessment, and developing machine-learning models from these data. These learning outcomes cover a large range of distinct activities in the biomedical data life-cycle, what may hinder the learning process in the limited time assigned for the subject. We propose a project based learning approach addressing the full life-cycle of biomedical data on the MIMIC-III (Medical Information Mart for Intensive Care III) Open Dataset, a freely accessible database comprising information relating to patients admitted to critical care units. By means of this active learning approach, students can achieve all the learning outcomes of the subject in an integrated manner: understanding the MIMIC-III data model, using health information standards such as International Classification of Diseases 9th Edition (ICD-9), mapping to interoperability standards, querying data, creating data tables and addressing data quality towards applying reliable statistical and machine learning analysis and, developing predictive models for several tasks such as predicting in-hospital mortality. MIMIC-III is widely used in the academia and science, with a large amount of publicly available resources and scientific articles to support the students learning. Additionally, the students will gain new competences in the use of Open Data and Research Ethics and Compliance Training.Alcalá, L.; García Gómez, JM.; Sáez Silvestre, C. (2021). Project based learning in Biomedical Data Science using the MIMIC III open dataset. En Proceedings INNODOCT/20. International Conference on Innovation, Documentation and Education. Editorial Universitat Politècnica de València. 203-212. https://doi.org/10.4995/INN2020.2020.11890OCS20321

Comparative study of probability distribution distances to define a metric for the stability of multi-source biomedical research data

Research biobanks are often composed by data from multiple sources. In some cases, these different subsets of data may present dissimilarities among their probability density functions (PDF) due to spatial shifts. This, may lead to wrong hypothesis when treating the data as a whole. Also, the overall quality of the data is diminished. With the purpose of developing a generic and comparable metric to assess the stability of multi-source datasets, we have studied the applicability and behaviour of several PDF distances over shifts on different conditions (such as uni- and multivariate, different types of variable, and multi-modality) which may appear in real biomedical data. From the studied distances, we found information-theoretic based and Earth Mover’s Distance to be the most practical distances for most conditions. We discuss the properties and usefulness of each distance according to the possible requirements of a general stability metric.Sáez Silvestre, C.; Robles Viejo, M.; García Gómez, JM. (2013). Comparative study of probability distribution distances to define a metric for the stability of multi-source biomedical research data. Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE. 3226-3229. doi:10.1109/EMBC.2013.6610228S3226322

Potential limitations in COVID-19 machine learning due to data source variability: A case study in the nCov2019 dataset

[EN] Objective: The lack of representative coronavirus disease 2019 (COVID-19) data is a bottleneck for reliable and generalizable machine learning. Data sharing is insufficient without data quality, in which source variability plays an important role. We showcase and discuss potential biases from data source variability for COVID-19 machine learning. Materials and Methods: We used the publicly available nCov2019 dataset, including patient-level data from several countries. We aimed to the discovery and classification of severity subgroups using symptoms and comorbidities. Results: Cases from the 2 countries with the highest prevalence were divided into separate subgroups with distinct severity manifestations. This variability can reduce the representativeness of training data with respect the model target populations and increase model complexity at risk of overfitting. Conclusions: Data source variability is a potential contributor to bias in distributed research networks. We call for systematic assessment and reporting of data source variability and data quality in COVID-19 data sharing, as key information for reliable and generalizable machine learning.This work was supported by Universitat Politecnica de Valencia contract no. UPV-SUB.2-1302 and FONDO SUPERA COVID-19 by CRUE-Santander Bank grant "Severity Subgroup Discovery and Classification on COVID-19 Real World Data through Machine Learning and Data Quality assessment (SUBCOVERWD-19)."Sáez Silvestre, C.; Romero, N.; Conejero, JA.; Garcia-Gomez, JM. (2021). Potential limitations in COVID-19 machine learning due to data source variability: A case study in the nCov2019 dataset. Journal of the American Medical Informatics Association. 28(2):360-364. https://doi.org/10.1093/jamia/ocaa25836036428

EHRtemporalVariability: delineating temporal data-set shifts in electronic health records

[EN] Background: Temporal variability in health-care processes or protocols is intrinsic to medicine. Such variability can potentially introduce dataset shifts, a data quality issue when reusing electronic health records (EHRs) for secondary purposes. Temporal data-set shifts can present as trends, as well as abrupt or seasonal changes in the statistical distributions of data over time. The latter are particularly complicated to address in multimodal and highly coded data. These changes, if not delineated, can harm population and data-driven research, such as machine learning. Given that biomedical research repositories are increasingly being populated with large sets of historical data from EHRs, there is a need for specific software methods to help delineate temporal data-set shifts to ensure reliable data reuse. Results: EHRtemporalVariability is an open-source R package and Shiny app designed to explore and identify temporal data-set shifts. EHRtemporalVariability estimates the statistical distributions of coded and numerical data over time; projects their temporal evolution through non-parametric information geometric temporal plots; and enables the exploration of changes in variables through data temporal heat maps. We demonstrate the capability of EHRtemporalVariability to delineate data-set shifts in three impact case studies, one of which is available for reproducibility. Conclusions: EHRtemporalVariability enables the exploration and identification of data-set shifts, contributing to the broad examination and repurposing of large, longitudinal data sets. Our goal is to help ensure reliable data reuse for a wide range of biomedical data users. EHRtemporalVariability is designed for technical users who are programmatically utilizing the R package, as well as users who are not familiar with programming via the Shiny user interface.This work was supported by Universitat Politecnica de Valencia grant PAID-00-17, Generalitat Valenciana grant BEST/2018, and projects H2020-SC1-2016-CNECT No. 727560 and H2020-SC1-BHC-2018-2020 No. 825750Sáez Silvestre, C.; Gutiérrez-Sacristán, A.; Kohane, I.; Garcia-Gomez, JM.; Avillach, P. (2020). EHRtemporalVariability: delineating temporal data-set shifts in electronic health records. GigaScience. 9(8):1-7. https://doi.org/10.1093/gigascience/giaa079S1798Gewin, V. (2016). Data sharing: An open mind on open data. Nature, 529(7584), 117-119. doi:10.1038/nj7584-117aKatzan, I. L., & Rudick, R. A. (2012). Time to Integrate Clinical and Research Informatics. Science Translational Medicine, 4(162). doi:10.1126/scitranslmed.3004583Zhu, L., & Zheng, W. J. (2018). Informatics, Data Science, and Artificial Intelligence. JAMA, 320(11), 1103. doi:10.1001/jama.2018.8211Rajkomar, A., Dean, J., & Kohane, I. (2019). Machine Learning in Medicine. New England Journal of Medicine, 380(14), 1347-1358. doi:10.1056/nejmra1814259Andreu-Perez, J., Poon, C. C. Y., Merrifield, R. D., Wong, S. T. C., & Yang, G.-Z. (2015). Big Data for Health. IEEE Journal of Biomedical and Health Informatics, 19(4), 1193-1208. doi:10.1109/jbhi.2015.2450362Sáez, C., Rodrigues, P. P., Gama, J., Robles, M., & García-Gómez, J. M. (2014). Probabilistic change detection and visualization methods for the assessment of temporal stability in biomedical data quality. Data Mining and Knowledge Discovery, 29(4), 950-975. doi:10.1007/s10618-014-0378-6Schlegel, D. R., & Ficheur, G. (2017). Secondary Use of Patient Data: Review of the Literature Published in 2016. Yearbook of Medical Informatics, 26(01), 68-71. doi:10.15265/iy-2017-032Agniel, D., Kohane, I. S., & Weber, G. M. (2018). Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ, k1479. doi:10.1136/bmj.k1479Sáez, C., & García-Gómez, J. M. (2018). Kinematics of Big Biomedical Data to characterize temporal variability and seasonality of data repositories: Functional Data Analysis of data temporal evolution over non-parametric statistical manifolds. International Journal of Medical Informatics, 119, 109-124. doi:10.1016/j.ijmedinf.2018.09.015Leek, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Langmead, B., Johnson, W. E., … Irizarry, R. A. (2010). Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics, 11(10), 733-739. doi:10.1038/nrg2825Goh, W. W. B., Wang, W., & Wong, L. (2017). Why Batch Effects Matter in Omics Data, and How to Avoid Them. Trends in Biotechnology, 35(6), 498-507. doi:10.1016/j.tibtech.2017.02.012Sáez, C., Zurriaga, O., Pérez-Panadés, J., Melchor, I., Robles, M., & García-Gómez, J. M. (2016). Applying probabilistic temporal and multisite data quality control methods to a public health mortality registry in Spain: a systematic approach to quality control of repositories. Journal of the American Medical Informatics Association, 23(6), 1085-1095. doi:10.1093/jamia/ocw010Wright, A., Ash, J. S., Aaron, S., Ai, A., Hickman, T.-T. T., Wiesen, J. F., … Sittig, D. F. (2018). Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study. International Journal of Medical Informatics, 118, 78-85. doi:10.1016/j.ijmedinf.2018.08.001Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012). A unifying view on dataset shift in classification. Pattern Recognition, 45(1), 521-530. doi:10.1016/j.patcog.2011.06.019Svolba, G., & Bauer, P. (1999). Statistical Quality Control in Clinical Trials. Controlled Clinical Trials, 20(6), 519-530. doi:10.1016/s0197-2456(99)00029-xBray, F., & Parkin, D. M. (2009). Evaluation of data quality in the cancer registry: Principles and methods. Part I: Comparability, validity and timeliness. European Journal of Cancer, 45(5), 747-755. doi:10.1016/j.ejca.2008.11.032Springate, D. A., Parisi, R., Olier, I., Reeves, D., & Kontopantelis, E. (2017). rEHR: An R package for manipulating and analysing Electronic Health Record data. PLOS ONE, 12(2), e0171784. doi:10.1371/journal.pone.0171784Choi, L., Carroll, R. J., Beck, C., Mosley, J. D., Roden, D. M., Denny, J. C., & Van Driest, S. L. (2018). Evaluating statistical approaches to leverage large clinical datasets for uncovering therapeutic and adverse medication effects. Bioinformatics, 34(17), 2988-2996. doi:10.1093/bioinformatics/bty306Gutiérrez-Sacristán, A., Bravo, À., Giannoula, A., Mayer, M. A., Sanz, F., & Furlong, L. I. (2018). comoRbidity: an R package for the systematic analysis of disease comorbidities. Bioinformatics, 34(18), 3228-3230. doi:10.1093/bioinformatics/bty315Denny, J. C., Bastarache, L., Ritchie, M. D., Carroll, R. J., Zink, R., Mosley, J. D., … Roden, D. M. (2013). Systematic comparison of phenome-wide association study of electronic medical record data and genome-wide association study data. Nature Biotechnology, 31(12), 1102-1111. doi:10.1038/nbt.2749Khera, R., Dorsey, K. B., & Krumholz, H. M. (2018). Transition to the ICD-10 in the United States. JAMA, 320(2), 133. doi:10.1001/jama.2018.682

Data-driven discovery of changes in clinical code usage over time: a case-study on changes in cardiovascular disease recording in two English electronic health records databases (2001-2015)

[EN] Objectives To demonstrate how data-driven variability methods can be used to identify changes in disease recording in two English electronic health records databases between 2001 and 2015. Design Repeated cross-sectional analysis that applied data-driven temporal variability methods to assess month-by-month changes in routinely collected medical data. A measure of difference between months was calculated based on joint distributions of age, gender, socioeconomic status and recorded cardiovascular diseases. Distances between months were used to identify temporal trends in data recording. Setting 400 English primary care practices from the Clinical Practice Research Datalink (CPRD GOLD) and 451 hospital providers from the Hospital Episode Statistics (HES). Main outcomes The proportion of patients (CPRD GOLD) and hospital admissions (HES) with a recorded cardiovascular disease (CPRD GOLD: coronary heart disease, heart failure, peripheral arterial disease, stroke; HES: International Classification of Disease codes I20-I69/G45). Results Both databases showed gradual changes in cardiovascular disease recording between 2001 and 2008. The recorded prevalence of included cardiovascular diseases in CPRD GOLD increased by 47%-62%, which partially reversed after 2008. For hospital records in HES, there was a relative decrease in angina pectoris (-34.4%) and unspecified stroke (-42.3%) over the same time period, with a concomitant increase in chronic coronary heart disease (+14.3%). Multiple abrupt changes in the use of myocardial infarction codes in hospital were found in March/April 2010, 2012 and 2014, possibly linked to updates of clinical coding guidelines. Conclusions Identified temporal variability could be related to potentially non-medical causes such as updated coding guidelines. These artificial changes may introduce temporal correlation among diagnoses inferred from routine data, violating the assumptions of frequently used statistical methods. Temporal variability measures provide an objective and robust technique to identify, and subsequently account for, those changes in electronic health records studies without any prior knowledge of the data collection process.VN is funded by a Public Health England PhD Studentship. RWA is supported by a Wellcome Trust Clinical Research Career Development Fellowship (206602/Z/17/Z). JMGG and CS contributions to this work were partially supported by the MTS4up Spanish project (National Plan for Scientific and Technical Research and Innovation 2013-2016, No. DPI2016-80054-R), the CrowdHealth H2020-SC1-2016-CNECT project (No. 727560) (JMGG) and the Inadvance H2020-SC1-BHC-2018-2020 project (No. 825750). PR and DA did not receive any direct funding for this project. Access to the Clinical Practice Research Datalink was supported by the UK Economic and Social Research Council (ES/P008321/1). Access to aggregated Hospital Episode Statistics was provided by Public Health England. This work was further supported by Health Data Research UK, which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and the Wellcome Trust.Rockenschaub, P.; Nguyen, V.; Aldridge, RW.; Acosta, D.; Garcia-Gomez, JM.; Sáez Silvestre, C. (2020). Data-driven discovery of changes in clinical code usage over time: a case-study on changes in cardiovascular disease recording in two English electronic health records databases (2001-2015). BMJ Open. 10(2):1-9. https://doi.org/10.1136/bmjopen-2019-034396S19102Hripcsak, G., & Albers, D. J. (2013). Next-generation phenotyping of electronic health records. Journal of the American Medical Informatics Association, 20(1), 117-121. doi:10.1136/amiajnl-2012-001145Burton, P. R., Murtagh, M. J., Boyd, A., Williams, J. B., Dove, E. S., Wallace, S. E., … Knoppers, B. M. (2015). Data Safe Havens in health research and healthcare. Bioinformatics, 31(20), 3241-3248. doi:10.1093/bioinformatics/btv279Cruz-Correia R , Rodrigues P , Freitas A . Chapter: 4, Data quality and integration issues in electronic health records. In: Information discovery on electronic health records. CRC Press, 2009: 55–95.Massoudi, B. L., Goodman, K. W., Gotham, I. J., Holmes, J. H., Lang, L., Miner, K., … Fu, P. C. (2012). An informatics agenda for public health: summarized recommendations from the 2011 AMIA PHI Conference. Journal of the American Medical Informatics Association, 19(5), 688-695. doi:10.1136/amiajnl-2011-000507Schlegel, D. R., & Ficheur, G. (2017). Secondary Use of Patient Data: Review of the Literature Published in 2016. Yearbook of Medical Informatics, 26(01), 68-71. doi:10.15265/iy-2017-032Weiskopf, N. G., & Weng, C. (2013). Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association, 20(1), 144-151. doi:10.1136/amiajnl-2011-000681Herrett, E., Thomas, S. L., Schoonen, W. M., Smeeth, L., & Hall, A. J. (2010). Validation and validity of diagnoses in the General Practice Research Database: a systematic review. British Journal of Clinical Pharmacology, 69(1), 4-14. doi:10.1111/j.1365-2125.2009.03537.xSáez, C., Zurriaga, O., Pérez-Panadés, J., Melchor, I., Robles, M., & García-Gómez, J. M. (2016). Applying probabilistic temporal and multisite data quality control methods to a public health mortality registry in Spain: a systematic approach to quality control of repositories. Journal of the American Medical Informatics Association, 23(6), 1085-1095. doi:10.1093/jamia/ocw010Tate AR , Dungey S , Glew S , et al . Quality of recording of diabetes in the UK: how does the GP's method of coding clinical data affect incidence estimates? cross-sectional study using the CPRD database. BMJ Open 2017;7:e012905.doi:10.1136/bmjopen-2016-012905Calvert M , Shankar A , McManus RJ , et al . Effect of the quality and outcomes framework on diabetes care in the United Kingdom: retrospective cohort study. BMJ 2009;338:b1870.doi:10.1136/bmj.b1870Sáez, C., Robles, M., & García-Gómez, J. M. (2016). Stability metrics for multi-source biomedical data based on simplicial projections from probability distribution distances. Statistical Methods in Medical Research, 26(1), 312-336. doi:10.1177/0962280214545122Herrett, E., Gallagher, A. M., Bhaskaran, K., Forbes, H., Mathur, R., van Staa, T., & Smeeth, L. (2015). Data Resource Profile: Clinical Practice Research Datalink (CPRD). International Journal of Epidemiology, 44(3), 827-836. doi:10.1093/ije/dyv098Herbert, A., Wijlaars, L., Zylbersztejn, A., Cromwell, D., & Hardelid, P. (2017). Data Resource Profile: Hospital Episode Statistics Admitted Patient Care (HES APC). International Journal of Epidemiology, 46(4), 1093-1093i. doi:10.1093/ije/dyx015Chisholm J . The read clinical classification. BMJ 1990;300:1092.doi:10.1136/bmj.300.6732.1092Denaxas, S., Gonzalez-Izquierdo, A., Direk, K., Fitzpatrick, N. K., Fatemifar, G., Banerjee, A., … Hemingway, H. (2019). UK phenomics platform for developing and validating electronic health record phenotypes: CALIBER. Journal of the American Medical Informatics Association, 26(12), 1545-1559. doi:10.1093/jamia/ocz105Department for Communities and Local Government . The English Index of Multiple Deprivation (IMD) 2015 - Guidance. Available: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/464430/English_Index_of_Multiple_Deprivation_2015_-_Guidance.pdf [Accessed 8 Dec 2019].Sáez, C., Rodrigues, P. P., Gama, J., Robles, M., & García-Gómez, J. M. (2014). Probabilistic change detection and visualization methods for the assessment of temporal stability in biomedical data quality. Data Mining and Knowledge Discovery, 29(4), 950-975. doi:10.1007/s10618-014-0378-6Borg, I., & Groenen, P. (2003). Modern Multidimensional Scaling: Theory and Applications. Journal of Educational Measurement, 40(3), 277-280. doi:10.1111/j.1745-3984.2003.tb01108.xSáez, C., & García-Gómez, J. M. (2018). Kinematics of Big Biomedical Data to characterize temporal variability and seasonality of data repositories: Functional Data Analysis of data temporal evolution over non-parametric statistical manifolds. International Journal of Medical Informatics, 119, 109-124. doi:10.1016/j.ijmedinf.2018.09.015Conrad, N., Judge, A., Tran, J., Mohseni, H., Hedgecott, D., Crespillo, A. P., … Rahimi, K. (2018). Temporal trends and patterns in heart failure incidence: a population-based study of 4 million individuals. The Lancet, 391(10120), 572-580. doi:10.1016/s0140-6736(17)32520-5Herrett E , Shah AD , Boggon R , et al . Completeness and diagnostic validity of recording acute myocardial infarction events in primary care, hospital care, disease registry, and national mortality records: cohort study. BMJ 2013;346:f2350.doi:10.1136/bmj.f2350Pujades-Rodriguez M , Timmis A , Stogiannis D , et al . Socioeconomic deprivation and the incidence of 12 cardiovascular diseases in 1.9 million women and men: implications for risk prediction and prevention. PLoS One 2014;9:e104671.doi:10.1371/journal.pone.0104671Lee S , Shafe ACE , Cowie MR . Uk stroke incidence, mortality and cardiovascular risk management 1999-2008: time-trend analysis from the general practice research database. BMJ Open 2011;1:e000269.doi:10.1136/bmjopen-2011-000269Bhatnagar, P., Wickramasinghe, K., Williams, J., Rayner, M., & Townsend, N. (2015). The epidemiology of cardiovascular disease in the UK 2014. Heart, 101(15), 1182-1189. doi:10.1136/heartjnl-2015-307516Taylor, C. J., Ordóñez-Mena, J. M., Roalfe, A. K., Lay-Flurrie, S., Jones, N. R., Marshall, T., & Hobbs, F. D. R. (2019). Trends in survival after a diagnosis of heart failure in the United Kingdom 2000-2017: population based cohort study. BMJ, l223. doi:10.1136/bmj.l223Gho JMIH , Schmidt AF , Pasea L , et al . An electronic health records cohort study on heart failure following myocardial infarction in England: incidence and predictors. BMJ Open 2018;8:e018331.doi:10.1136/bmjopen-2017-018331Quint JK , Müllerova H , DiSantostefano RL , et al . Validation of chronic obstructive pulmonary disease recording in the clinical practice research Datalink (CPRD-GOLD). BMJ Open 2014;4:e005540.doi:10.1136/bmjopen-2014-005540Bhaskaran K , Forbes HJ , Douglas I , et al . Representativeness and optimal use of body mass index (BMI) in the UK clinical practice research Datalink (CPRD). BMJ Open 2013;3:e003389.doi:10.1136/bmjopen-2013-003389Booth, H. P., Prevost, A. T., & Gulliford, M. C. (2013). Validity of smoking prevalence estimates from primary care electronic health records compared with national population survey data for England, 2007 to 2011. Pharmacoepidemiology and Drug Safety, 22(12), 1357-1361. doi:10.1002/pds.3537Booth H , Dedman D , Wolf A . CPRD aurum frequently asked questions (FAQs). CPRD 2019.Burns, E. M., Rigby, E., Mamidanna, R., Bottle, A., Aylin, P., Ziprin, P., & Faiz, O. D. (2011). Systematic review of discharge coding accuracy. Journal of Public Health, 34(1), 138-148. doi:10.1093/pubmed/fdr054Marmot, M. G., Stansfeld, S., Patel, C., North, F., Head, J., White, I., … Smith, G. D. (1991). Health inequalities among British civil servants: the Whitehall II study. The Lancet, 337(8754), 1387-1393. doi:10.1016/0140-6736(91)93068-kKivimäki, M., Batty, G. D., Singh-Manoux, A., Britton, A., Brunner, E. J., & Shipley, M. J. (2017). Validity of Cardiovascular Disease Event Ascertainment Using Linkage to UK Hospital Records. Epidemiology, 28(5), 735-739. doi:10.1097/ede.0000000000000688Crosignani, P. G. (2003). Breast cancer and hormone-replacement therapy in the Million Women Study. Maturitas, 46(2), 91-92. doi:10.1016/j.maturitas.2003.09.002Wright FL , Green J , Canoy D , et al . Vascular disease in women: comparison of diagnoses in hospital episode statistics and general practice records in England. BMC Med Res Methodol 2012;12:161.doi:10.1186/1471-2288-12-161Herrett, E., Smeeth, L., Walker, L., & Weston, C. (2010). The Myocardial Ischaemia National Audit Project (MINAP). Heart, 96(16), 1264-1267. doi:10.1136/hrt.2009.192328Silver LE , Heneghan C , Mehta Z , et al . Substantial underestimation of incidence of acute myocardial infarction by hospital discharge diagnostic coding data: a prospective population-based study. Heart 2009;95.Health and Social Care Information Centre . Coding clinic guidance. 5th edn, 2010.Health & Social Care Information Centre . National Clinical Coding Standards - ICD-10. 4th edn, 2013. https://hscic.kahootz.com/connect.ti/t_c_home/view?objectId=31445829Wang, R. Y., & Strong, D. M. (1996). Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of Management Information Systems, 12(4), 5-33. doi:10.1080/07421222.1996.1151809

Applying probabilistic temporal and multi-site data quality control methods to a public health mortality registry in Spain: A systematic approach to quality control of repositories

OBJECTIVE: To assess the variability in data distributions among data sources and over time through a case study of a large multisite repository as a systematic approach to data quality (DQ). MATERIALS AND METHODS: Novel probabilistic DQ control methods based on information theory and geometry are applied to the Public Health Mortality Registry of the Region of Valencia, Spain, with 512 143 entries from 2000 to 2012, disaggregated into 24 health departments. The methods provide DQ metrics and exploratory visualizations for (1) assessing the variability among multiple sources and (2) monitoring and exploring changes with time. The methods are suited to big data and multitype, multivariate, and multimodal data. RESULTS: The repository was partitioned into 2 probabilistically separated temporal subgroups following a change in the Spanish National Death Certificate in 2009. Punctual temporal anomalies were noticed due to a punctual increment in the missing data, along with outlying and clustered health departments due to differences in populations or in practices. DISCUSSION: Changes in protocols, differences in populations, biased practices, or other systematic DQ problems affected data variability. Even if semantic and integration aspects are addressed in data sharing infrastructures, probabilistic variability may still be present. Solutions include fixing or excluding data and analyzing different sites or time periods separately. A systematic approach to assessing temporal and multisite variability is proposed. CONCLUSION: Multisite and temporal variability in data distributions affects DQ, hindering data reuse, and an assessment of such variability should be a part of systematic DQ procedures.This work was supported by the Spanish Ministry of Economy and Competitiveness grant numbers RTC-2014-1530-1 and TIN-2013-43457-R, and by the Universitat Politecnica de Valencia grant number SP20141432.Sáez Silvestre, C.; Zurriaga, O.; Pérez-Panadés, J.; Melchor, I.; Robles Viejo, M.; García Gómez, JM. (2016). Applying probabilistic temporal and multi-site data quality control methods to a public health mortality registry in Spain: A systematic approach to quality control of repositories. Journal of the American Medical Informatics Association. 23(6):1085-1095. https://doi.org/10.1093/jamia/ocw010S1085109523

Temporal variability analysis reveals biases in electronic health records due to hospital process reengineering interventions over seven years

[EN] Objective To evaluate the effects of Process-Reengineering interventions on the Electronic Health Records (EHR) of a hospital over 7 years. Materials and methods Temporal Variability Assessment (TVA) based on probabilistic data quality assessment was applied to the historic monthly-batched admission data of Hospital La Fe Valencia, Spain from 2010 to 2016. Routine healthcare data with a complete EHR was expanded by processed variables such as the Charlson Comorbidity Index. Results Four Process-Reengineering interventions were detected by quantifiable effects on the EHR: (1) the hospital relocation in 2011 involved progressive reduction of admissions during the next four months, (2) the hospital services re-configuration incremented the number of inter-services transfers, (3) the care-services re-distribution led to transfers between facilities (4) the assignment to the hospital of a new area with 80,000 patients in 2015 inspired the discharge to home for follow up and the update of the pre-surgery planned admissions protocol that produced a significant decrease of the patient length of stay. Discussion TVA provides an indicator of the effect of process re-engineering interventions on healthcare practice. Evaluating the effect of facilities¿ relocation and increment of citizens (findings 1, 3¿4), the impact of strategies (findings 2¿3), and gradual changes in protocols (finding 4) may help on the hospital management by optimizing interventions based on their effect on EHRs or on data reuse. Conclusions The effects on hospitals EHR due to process re-engineering interventions can be evaluated using the TVA methodology. Being aware of conditioned variations in EHR is of the utmost importance for the reliable reuse of routine hospitalization data.F.J.P.B, C.S., J.M.G.G. and J.A.C. were funded Universitat Politecnica de Valencia, project "ANALISIS DE LA CALIDAD Y VARIABILIDAD DE DATOS MEDICOS". www.upv.es. J.M.G.G.is also partially supported by: Ministerio de Economia y Competitividad of Spain through MTS4up project (National Plan for Scientific and Technical Research and Innovation 2013-2016, No. DPI2016-80054-R); and European Commission projects H2020-SC1-2016-CNECT Project (No. 727560) and H2020-SC1-BHC-2018-2020 (No. 825750). The funders did not play any role in the study design, data collection and analysis, decision to publish, nor preparation of the manuscript.Perez-Benito, FJ.; Sáez Silvestre, C.; Conejero, JA.; Tortajada, S.; Valdivieso, B.; Garcia-Gomez, JM. (2019). Temporal variability analysis reveals biases in electronic health records due to hospital process reengineering interventions over seven years. PLoS ONE. 14(8):1-19. https://doi.org/10.1371/journal.pone.0220369S119148Aguilar-Savén, R. S. (2004). Business process modelling: Review and framework. International Journal of Production Economics, 90(2), 129-149. doi:10.1016/s0925-5273(03)00102-6Poulymenopoulou, M. (2003). Journal of Medical Systems, 27(4), 325-335. doi:10.1023/a:1023701219563Dadam P, Reichert M, Kuhn K. Clinical Workflows -The Killer Application for Process-oriented Information Systems? Proceedings of the 4th International Conference on Business Information Systems. London: Springer London; 2000. pp. 36–59. doi: https://doi.org/10.1007/978-1-4471-0761-3Lenz, R., & Reichert, M. (2007). IT support for healthcare processes – premises, challenges, perspectives. Data & Knowledge Engineering, 61(1), 39-58. doi:10.1016/j.datak.2006.04.007Rebuge, Á., & Ferreira, D. R. (2012). Business process analysis in healthcare environments: A methodology based on process mining. Information Systems, 37(2), 99-116. doi:10.1016/j.is.2011.01.003Amour EAEH, Ghannouchi SA. Applying Data Mining Techniques to Discover KPIs Relationships in Business Process Context. 2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT). IEEE; 2017. pp. 230–237. doi: https://doi.org/10.1109/PDCAT.2017.00045Chou, Y.-C., Chen, B.-Y., Tang, Y.-Y., Qiu, Z.-J., Wu, M.-F., Wang, S.-C., … Chuang, W.-C. (2010). Prescription-Filling Process Reengineering of an Outpatient Pharmacy. Journal of Medical Systems, 36(2), 893-902. doi:10.1007/s10916-010-9553-5Leu, J.-D., & Huang, Y.-T. (2009). An Application of Business Process Method to the Clinical Efficiency of Hospital. Journal of Medical Systems, 35(3), 409-421. doi:10.1007/s10916-009-9376-4Gand K. Investigating on Requirements for Business Model Representations: The Case of Information Technology in Healthcare. 2017 IEEE 19th Conference on Business Informatics (CBI). IEEE; 2017. pp. 471–480. doi: https://doi.org/10.1109/CBI.2017.36Ferreira, G. S. A., Silva, U. R., Costa, A. L., & Pádua, S. I. D. de D. (2018). The promotion of BPM and lean in the health sector: main results. Business Process Management Journal, 24(2), 400-424. doi:10.1108/bpmj-06-2016-0115Abdulrahman Jabour RM. Cancer Reporting: Timeliness Analysis and Process. 2016; Available: https://search.proquest.com/openview/4ecf737c5ef6d2d503e948df8031fe54/1?pq-origsite=gscholar&cbl=18750&diss=yHewitt M, Simone J V. Enhancing Data Systems to Improve the Quality of Cancer Care [Internet]. National Academy Press; 2000. Available: http://www.nap.edu/catalog/9970.htmlWeiskopf, N. G., & Weng, C. (2013). Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association, 20(1), 144-151. doi:10.1136/amiajnl-2011-000681Saez C, Robles M, Garcia-Gomez JM. Comparative study of probability distribution distances to define a metric for the stability of multi-source biomedical research data. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS. IEEE; 2013. pp. 3226–3229. doi: https://doi.org/10.1109/EMBC.2013.6610228Sáez, C., Rodrigues, P. P., Gama, J., Robles, M., & García-Gómez, J. M. (2014). Probabilistic change detection and visualization methods for the assessment of temporal stability in biomedical data quality. Data Mining and Knowledge Discovery, 29(4), 950-975. doi:10.1007/s10618-014-0378-6Sáez, C., Zurriaga, O., Pérez-Panadés, J., Melchor, I., Robles, M., & García-Gómez, J. M. (2016). Applying probabilistic temporal and multisite data quality control methods to a public health mortality registry in Spain: a systematic approach to quality control of repositories. Journal of the American Medical Informatics Association, 23(6), 1085-1095. doi:10.1093/jamia/ocw010International Ethical Guidelines for Epidemiological Studies [Internet]. Geneva: Council for International Organizations of Medical Sciences (CIOMS) in collaboration with the World Health Organization; 2009. Available: https://cioms.ch/wp-content/uploads/2017/01/International_Ethical_Guidelines_LR.pdfResearch Ethics Committee of the Universitari i Politècnic La Fe Hospital [Internet]. Available: https://www.iislafe.es/en/research/ethics-committees/Charlson, M. E., Pompei, P., Ales, K. L., & MacKenzie, C. R. (1987). A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. Journal of Chronic Diseases, 40(5), 373-383. doi:10.1016/0021-9681(87)90171-8Schneeweiss, S., Wang, P. S., Avorn, J., & Glynn, R. J. (2003). Improved Comorbidity Adjustment for Predicting Mortality in Medicare Populations. Health Services Research, 38(4), 1103-1120. doi:10.1111/1475-6773.00165Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., … Ghali, W. A. (2005). Coding Algorithms for Defining Comorbidities in ICD-9-CM and ICD-10 Administrative Data. Medical Care, 43(11), 1130-1139. doi:10.1097/01.mlr.0000182534.19832.83Sáez Silvestre C. Probabilistic methods for multi-source and temporal biomedical data quality assessment [Internet]. Thesis. Universitat Politècnica de València. 2016. doi: https://doi.org/10.4995/Thesis/10251/62188Amari S, Nagaoka H. Methods of Information Geometry [Internet]. Amer. Math. Soc. and Oxford Univ. Press. American Mathematical Society; 2000. Available: https://books.google.es/books?hl=es&lr=&id=vc2FWSo7wLUC&oi=fnd&pg=PR7&dq=Methods+of+Information+geometry&ots=4HmyCCY4PX&sig=2-dpCuwMQvEC1iREjxdfIX0yEls#v=onepage&q=MethodsofInformationgeometry&f=falseCsiszár, I., & Shields, P. C. (2004). Information Theory and Statistics: A Tutorial. Foundations and Trends™ in Communications and Information Theory, 1(4), 417-528. doi:10.1561/0100000004Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1), 145-151. doi:10.1109/18.61115M.Cover T. Elements Of Information Theory Notes [Internet]. 2006. Available: http://books.google.fr/books?id=VWq5GG6ycxMC&printsec=frontcover&dq=intitle:Elements+of+Information+Theory&hl=&cd=1&source=gbs_api%5Cnpapers2://publication/uuid/BAF426F8-5A4F-44A4-8333-FA8187160D9BBrandes, U., & Pich, C. (s. f.). Eigensolver Methods for Progressive Multidimensional Scaling of Large Data. Lecture Notes in Computer Science, 42-53. doi:10.1007/978-3-540-70904-6_6Liaw, S. T., Rahimi, A., Ray, P., Taggart, J., Dennis, S., de Lusignan, S., … Talaei-Khoei, A. (2013). Towards an ontology for data quality in integrated chronic disease management: A realist review of the literature. International Journal of Medical Informatics, 82(1), 10-24. doi:10.1016/j.ijmedinf.2012.10.001Arts, D. G. T. (2002). Defining and Improving Data Quality in Medical Registries: A Literature Review, Case Study, and Generic Framework. Journal of the American Medical Informatics Association, 9(6), 600-611. doi:10.1197/jamia.m1087Bray, F., & Parkin, D. M. (2009). Evaluation of data quality in the cancer registry: Principles and methods. Part I: Comparability, validity and timeliness. European Journal of Cancer, 45(5), 747-755. doi:10.1016/j.ejca.2008.11.032Parkin, D. M., & Bray, F. (2009). Evaluation of data quality in the cancer registry: Principles and methods Part II. Completeness. European Journal of Cancer, 45(5), 756-764. doi:10.1016/j.ejca.2008.11.033Fernandez-Llatas, C., Ibanez-Sanchez, G., Celda, A., Mandingorra, J., Aparici-Tortajada, L., Martinez-Millana, A., … Traver, V. (2019). Analyzing Medical Emergency Processes with Process Mining: The Stroke Case. Lecture Notes in Business Information Processing, 214-225. doi:10.1007/978-3-030-11641-5_17Fernandez-Llatas, C., Lizondo, A., Monton, E., Benedi, J.-M., & Traver, V. (2015). Process Mining Methodology for Health Process Tracking Using Real-Time Indoor Location Systems. Sensors, 15(12), 29821-29840. doi:10.3390/s151229769Van der Aalst, W., Weijters, T., & Maruster, L. (2004). Workflow mining: discovering process models from event logs. IEEE Transactions on Knowledge and Data Engineering, 16(9), 1128-1142. doi:10.1109/tkde.2004.47Weijters AJMM, Van Der Aalst WMP, Alves De Medeiros AK. Process Mining with the HeuristicsMiner Algorithm [Internet]. Available: https://pdfs.semanticscholar.org/1cc3/d62e27365b8d7ed6ce93b41c193d0559d086.pdfShim, S. J., & Kumar, A. (2010). Simulation for emergency care process reengineering in hospitals. Business Process Management Journal, 16(5), 795-805. doi:10.1108/14637151011076476Svolba, G., & Bauer, P. (1999). Statistical Quality Control in Clinical Trials. Controlled Clinical Trials, 20(6), 519-530. doi:10.1016/s0197-2456(99)00029-xKahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F. (2012). A Pragmatic Framework for Single-site and Multisite Data Quality Assessment in Electronic Health Record-based Clinical Research. Medical Care, 50, S21-S29. doi:10.1097/mlr.0b013e318257dd67Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009). Methodologies for data quality assessment and improvement. ACM Computing Surveys, 41(3), 1-52. doi:10.1145/1541880.1541883Heinrich, B., Klier, M., & Kaiser, M. (2009). A Procedure to Develop Metrics for Currency and its Application in CRM. Journal of Data and Information Quality, 1(1), 1-28. doi:10.1145/1515693.1515697Sirgo, G., Esteban, F., Gómez, J., Moreno, G., Rodríguez, A., Blanch, L., … Bodí, M. (2018). Validation of the ICU-DaMa tool for automatically extracting variables for minimum dataset and quality indicators: The importance of data quality assessment. International Journal of Medical Informatics, 112, 166-172. doi:10.1016/j.ijmedinf.2018.02.007Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786), 504-507. doi:10.1126/science.1127647Kohn LT, Corrigan JM. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America. 2000. p. 287. National Academies Press

Subphenotyping of Mexican Patients With COVID-19 at Preadmission To Anticipate Severity Stratification: Age-Sex Unbiased Meta-Clustering Technique

[EN] Background: The COVID-19 pandemic has led to an unprecedented global health care challenge for both medical institutions and researchers. Recognizing different COVID-19 subphenotypes-the division of populations of patients into more meaningful subgroups driven by clinical features-and their severity characterization may assist clinicians during the clinical course, the vaccination process, research efforts, the surveillance system, and the allocation of limited resources. Objective: We aimed to discover age-sex unbiased COVID-19 patient subphenotypes based on easily available phenotypical data before admission, such as pre-existing comorbidities, lifestyle habits, and demographic features, to study the potential early severity stratification capabilities of the discovered subgroups through characterizing their severity patterns, including prognostic, intensive care unit (ICU), and morbimortality outcomes. Methods: We used the Mexican Government COVID-19 open data, including 778,692 SARS-CoV-2 population-based patient-level data as of September 2020. We applied a meta-clustering technique that consists of a 2-stage clustering approach combining dimensionality reduction (ie, principal components analysis and multiple correspondence analysis) and hierarchical clustering using the Ward minimum variance method with Euclidean squared distance. Results: In the independent age-sex clustering analyses, 56 clusters supported 11 clinically distinguishable meta-clusters (MCs). MCs 1-3 showed high recovery rates (90.27%-95.22%), including healthy patients of all ages, children with comorbidities and priority in receiving medical resources (ie, higher rates of hospitalization, intubation, and ICU admission) compared with other adult subgroups that have similar conditions, and young obese smokers. MCs 4-5 showed moderate recovery rates (81.30%-82.81%), including patients with hypertension or diabetes of all ages and obese patients with pneumonia, hypertension, and diabetes. MCs 6-11 showed low recovery rates (53.96%-66.94%), including immunosuppressed patients with high comorbidity rates, patients with chronic kidney disease with a poor survival length and probability of recovery, older smokers with chronic obstructive pulmonary disease, older adults with severe diabetes and hypertension, and the oldest obese smokers with chronic obstructive pulmonary disease and mild cardiovascular disease. Group outcomes conformed to the recent literature on dedicated age-sex groups. Mexican states and several types of clinical institutions showed relevant heterogeneity regarding severity, potentially linked to socioeconomic or health inequalities. Conclusions: The proposed 2-stage cluster analysis methodology produced a discriminative characterization of the sample and explainability over age and sex. These results can potentially help in understanding the clinical patient and their stratification for automated early triage before further tests and laboratory results are available and even in locations where additional tests are not available or to help decide resource allocation among vulnerable subgroups such as to prioritize vaccination or treatments.We sincerely thank the different types of clinical institutions and the Mexican government, which made a huge effort to make these data publicly available. We also thank the clinicians and epidemiologists from the Servicios de Salud de Nayarit for the useful discussions on specific aspects of the medical attention to hospitalized patients and the reporting of epidemiological data processes related to COVID-19. Furthermore, we would also like to thank Francisco Tomas Garcia Ruiz for his valuable help in data visualization design. This work was supported by Universitat Politecnica de Valencia contract no. UPV-SUB.2-1302 and FONDO SUPERA COVID-19 by CRUE-Santander Bank grant: "Severity Subgroup Discovery and Classification on COVID-19 Real World Data through Machine Learning and Data Quality assessment (SUBCOVERWD-19) ." The authors thank the Institute for Information and Communication Technologies (ITACA) at the Universitat Politecnica de Valencia for its support in the publication of this manuscript.Zhou, L.; Romero-Garcia, N.; Martínez-Miranda, J.; Conejero, JA.; Garcia-Gomez, JM.; Sáez Silvestre, C. (2022). Subphenotyping of Mexican Patients With COVID-19 at Preadmission To Anticipate Severity Stratification: Age-Sex Unbiased Meta-Clustering Technique. JMIR Public Health and Surveillance. 8(3):1-21. https://doi.org/10.2196/300321218

Smartphone sensors for monitoring cancer-related Quality of Life: App design, EORTC QLQ-C30 mapping and feasibility study in healthy subjects

[EN] Quality of life (QoL) indicators are now being adopted as clinical outcomes in clinical trials on cancer treatments. Technology-free daily monitoring of patients is complicated, time-consuming and expensive due to the need for vast amounts of resources and personnel. The alternative method of using the patients¿ own phones could reduce the burden of continuous monitoring of cancer patients in clinical trials. This paper proposes monitoring the patients¿ QoL by gathering data from their own phones. We considered that the continuous multiparametric acquisition of movement, location, phone calls, conversations and data use could be employed to simultaneously monitor their physical, psychological, social and environmental aspects. An open access phone app was developed (Human Dynamics Reporting Service (HDRS)) to implement this approach. We here propose a novel mapping between the standardized QoL items for these patients, the European Organization for the Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30) and define HDRS monitoring indicators. A pilot study with university volunteers verified the plausibility of detecting human activity indicators directly related to QoL.Funding for this study was provided by the authors' various departments, and partially by the CrowdHealth Project (Collective Wisdom Driving Public Health Policies (727560)) and the MTS4up project (DPI2016-80054-R).Asensio Cuesta, S.; Sánchez-García, Á.; Conejero, JA.; Sáez Silvestre, C.; Rivero-Rodriguez, A.; Garcia-Gomez, JM. (2019). Smartphone sensors for monitoring cancer-related Quality of Life: App design, EORTC QLQ-C30 mapping and feasibility study in healthy subjects. International Journal of Environmental research and Public Health. 16(3):1-18. https://doi.org/10.3390/ijerph16030461S118163Number of Smartphone Users Worldwide from 2014 to 2020 (in Billions)https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/Mirkovic, J., Kaufman, D. R., & Ruland, C. M. (2014). Supporting Cancer Patients in Illness Management: Usability Evaluation of a Mobile App. JMIR mHealth and uHealth, 2(3), e33. doi:10.2196/mhealth.3359Xing Su, Hanghang Tong, & Ping Ji. (2014). Activity recognition with smartphone sensors. Tsinghua Science and Technology, 19(3), 235-249. doi:10.1109/tst.2014.6838194Schmitz Weiss, A. (2013). Exploring News Apps and Location-Based Services on the Smartphone. Journalism & Mass Communication Quarterly, 90(3), 435-456. doi:10.1177/1077699013493788Higgins, J. P. (2016). Smartphone Applications for Patients’ Health and Fitness. The American Journal of Medicine, 129(1), 11-19. doi:10.1016/j.amjmed.2015.05.038Rivenson, Y., Ceylan Koydemir, H., Wang, H., Wei, Z., Ren, Z., Günaydın, H., … Ozcan, A. (2018). Deep Learning Enhanced Mobile-Phone Microscopy. ACS Photonics, 5(6), 2354-2364. doi:10.1021/acsphotonics.8b00146Priye, A., Ball, C. S., & Meagher, R. J. (2018). Colorimetric-Luminance Readout for Quantitative Analysis of Fluorescence Signals with a Smartphone CMOS Sensor. Analytical Chemistry, 90(21), 12385-12389. doi:10.1021/acs.analchem.8b03521Measuring Quality of Life for Cancer Patients: Where Are We Today and Where Are We Headed Tomorrow?http://blog.mdsol.com/measuring-quality-of-life-for-cancer-patients-where-are-we-today-and-where-are-we-headed-tomorrow/Zulueta, J., Piscitello, A., Rasic, M., Easter, R., Babu, P., Langenecker, S. A., … Leow, A. (2018). Predicting Mood Disturbance Severity with Mobile Phone Keystroke Metadata: A BiAffect Digital Phenotyping Study. Journal of Medical Internet Research, 20(7), e241. doi:10.2196/jmir.9775Caruso, R., GiuliaNanni, M., Riba, M. B., Sabato, S., & Grassi, L. (2017). Depressive Spectrum Disorders in Cancer: Diagnostic Issues and Intervention. A Critical Review. Current Psychiatry Reports, 19(6). doi:10.1007/s11920-017-0785-7THE WHOQOL GROUP. (1998). Development of the World Health Organization WHOQOL-BREF Quality of Life Assessment. Psychological Medicine, 28(3), 551-558. doi:10.1017/s0033291798006667Basic Issues Concerning Health-Related Quality of Life. (2017). Central European Journal of Urology, 70(2). doi:10.5173/ceju.2017.923Sloan, J. A. (2011). Metrics to Assess Quality of Life After Management of Early-Stage Lung Cancer. The Cancer Journal, 17(1), 63-67. doi:10.1097/ppo.0b013e31820e15dcBordoni, R., Ciardiello, F., von Pawel, J., Cortinovis, D., Karagiannis, T., Ballinger, M., … Rittmeyer, A. (2018). Patient-Reported Outcomes in OAK: A Phase III Study of Atezolizumab Versus Docetaxel in Advanced Non–Small-cell Lung Cancer. Clinical Lung Cancer, 19(5), 441-449.e4. doi:10.1016/j.cllc.2018.05.011Hartkopf, A. D., Graf, J., Simoes, E., Keilmann, L., Sickenberger, N., Gass, P., … Wallwiener, M. (2017). Electronic-Based Patient-Reported Outcomes: Willingness, Needs, and Barriers in Adjuvant and Metastatic Breast Cancer Patients. JMIR Cancer, 3(2), e11. doi:10.2196/cancer.6996Wallwiener, M., Matthies, L., Simoes, E., Keilmann, L., Hartkopf, A. D., Sokolov, A. N., … Brucker, S. Y. (2017). Reliability of an e-PRO Tool of EORTC QLQ-C30 for Measurement of Health-Related Quality of Life in Patients With Breast Cancer: Prospective Randomized Trial. Journal of Medical Internet Research, 19(9), e322. doi:10.2196/jmir.8210Gresham, G., Hendifar, A. E., Spiegel, B., Neeman, E., Tuli, R., Rimel, B. J., … Shinde, A. M. (2018). Wearable activity monitors to assess performance status and predict clinical outcomes in advanced cancer patients. npj Digital Medicine, 1(1). doi:10.1038/s41746-018-0032-6BOHANNON, R. W. (1997). Comfortable and maximum walking speed of adults aged 20—79 years: reference values and determinants. Age and Ageing, 26(1), 15-19. doi:10.1093/ageing/26.1.15Pérez-García, V. M., Fitzpatrick, S., Pérez-Romasanta, L. A., Pesic, M., Schucht, P., Arana, E., & Sánchez-Gómez, P. (2016). Applied mathematics and nonlinear sciences in the war on cancer. Applied Mathematics and Nonlinear Sciences, 1(2), 423-436. doi:10.21042/amns.2016.2.00036Shin, W., Song, S., Jung, S.-Y., Lee, E., Kim, Z., Moon, H.-G., … Lee, J. E. (2017). The association between physical activity and health-related quality of life among breast cancer survivors. Health and Quality of Life Outcomes, 15(1). doi:10.1186/s12955-017-0706-9Wearable Fitness Monitors Useful in Cancer Treatment, Study Findswww.sciencedaily.com/releases/2018/05/180501130856.htmBade, B. C., Brooks, M. C., Nietert, S. B., Ulmer, A., Thomas, D. D., Nietert, P. J., … Silvestri, G. A. (2016). Assessing the Correlation Between Physical Activity and Quality of Life in Advanced Lung Cancer. Integrative Cancer Therapies, 17(1), 73-79. doi:10.1177/1534735416684016Fortner, B. V., Stepanski, E. J., Wang, S. C., Kasprowicz, S., & Durrence, H. H. (2002). Sleep and Quality of Life in Breast Cancer Patients. Journal of Pain and Symptom Management, 24(5), 471-480. doi:10.1016/s0885-3924(02)00500-6Mishra, S. I., Scherer, R. W., Snyder, C., Geigle, P., & Gotay, C. (2014). Are Exercise Programs Effective for Improving Health-Related Quality of Life Among Cancer Survivors? A Systematic Review and Meta-Analysis. Oncology Nursing Forum, 41(6), E326-E342. doi:10.1188/14.onf.e326-e342Ratcliff, C. G., Lam, C. Y., Arun, B., Valero, V., & Cohen, L. (2014). Ecological momentary assessment of sleep, symptoms, and mood during chemotherapy for breast cancer. Psycho-Oncology, 23(11), 1220-1228. doi:10.1002/pon.3525Cox, S. M., Lane, A., & Volchenboum, S. L. (2018). Use of Wearable, Mobile, and Sensor Technology in Cancer Clinical Trials. JCO Clinical Cancer Informatics, (2), 1-11. doi:10.1200/cci.17.00147Brown, W., Yen, P.-Y., Rojas, M., & Schnall, R. (2013). Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology. Journal of Biomedical Informatics, 46(6), 1080-1087. doi:10.1016/j.jbi.2013.08.001Darlow, S., & Wen, K.-Y. (2016). Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal, 22(3), 633-650. doi:10.1177/1460458215577994Martin Sanchez, F., Gray, K., Bellazzi, R., & Lopez-Campos, G. (2014). Exposome informatics: considerations for the design of future biomedical research information systems. Journal of the American Medical Informatics Association, 21(3), 386-390. doi:10.1136/amiajnl-2013-001772Kim, H. H., Lee, S. Y., Baik, S. Y., & Kim, J. H. (2015). MELLO: Medical lifelog ontology for data terms from self-tracking and lifelog devices. International Journal of Medical Informatics, 84(12), 1099-1110. doi:10.1016/j.ijmedinf.2015.08.005Kessel, K. A., Vogel, M. M., Alles, A., Dobiasch, S., Fischer, H., & Combs, S. E. (2018). Mobile App Delivery of the EORTC QLQ-C30 Questionnaire to Assess Health-Related Quality of Life in Oncological Patients: Usability Study. JMIR mHealth and uHealth, 6(2), e45. doi:10.2196/mhealth.9486Elsbernd, A., Hjerming, M., Visler, C., Hjalgrim, L. L., Niemann, C. U., Boisen, K., & Pappot, H. (2018). Cocreated Smartphone App to Improve the Quality of Life of Adolescents and Young Adults with Cancer (Kræftværket): Protocol for a Quantitative and Qualitative Evaluation. JMIR Research Protocols, 7(5), e10098. doi:10.2196/1009