The reliability of a deep learning model in external memory clinic MRI data: A multi‐cohort study

AbstractBackgroundDeep learning (DL) has provided impressive results in numerous domains in recent years, including medical image analysis. Training DL models requires large data sets to yield good performance. Since medical data can be difficult to acquire, most studies rely on public research cohorts, which often have harmonized scanning protocols and strict exclusion criteria. This is not representative of a clinical setting. In this study, we investigated the performance of a DL model in out‐of‐distribution data from multiple memory clinics and research cohorts.MethodWe trained multiple versions of AVRA: a DL model trained to predict visual ratings of Scheltens' medial temporal atrophy (MTA) scale (Mårtensson et al., 2019). This was done on different combinations of training data—starting with only harmonized MRI data from public research cohorts, and further increasing image heterogeneity in the training set by including external memory clinic data. We assessed the performance in multiple test sets by comparing AVRA's MTA ratings to an experienced radiologist's (who rated all images in this study). Data came from Alzheimer's Disease Neuroimaging Initiative (ADNI), AddNeuroMed, and images from 13 European memory clinics in the E‐DLB consortium.ResultsModels trained only on research cohorts generalized well to new data acquired with similar protocols as the training data (weighted kappa κw between 0.70‐0.72), but worse to memory clinic data with more image variability (κw between 0.34‐0.66). This was most prominent in one specific memory clinic, where the DL model systematically predicted too low MTA scores. When including data from a wider range of scanners and protocols during training, the agreement to the radiologist's ratings in external memory clinics increased (κw between 0.51‐0.71).ConclusionIn this study we showed that increasing heterogeneity in training data improves generalization to out‐of‐distribution data. Our findings suggest that studies assessing reliability of a DL model should be done in multiple cohorts, and that softwares based on DL need to be rigorously evaluated prior to being certified for deployment to clinics. References: Mårtensson, G. et al. (2019) 'AVRA: Automatic Visual Ratings of Atrophy from MRI images using Recurrent Convolutional Neural Networks', NeuroImage: Clinical. Elsevier, 23(March), p. 101872

Description of the Method for Evaluating Digital Endpoints in Alzheimer Disease Study : Protocol for an Exploratory, Cross-sectional Study

©Jelena Curcic, Vanessa Vallejo, Jennifer Sorinas, Oleksandr Sverdlov, Jens Praestgaard, Mateusz Piksa, Mark Deurinck, Gul Erdemli, Maximilian Bügler, Ioannis Tarnanas, Nick Taptiklis, Francesca Cormack, Rebekka Anker, Fabien Massé, William Souillard-Mandar, Nathan Intrator, Lior Molcho, Erica Madero, Nicholas Bott, Mieko Chambers, Josef Tamory, Matias Shulz, Gerardo Fernandez, William Simpson, Jessica Robin, Jón G Snædal, Jang-Ho Cha, Kristin Hannesdottir. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 10.08.2022.BACKGROUND: More sensitive and less burdensome efficacy end points are urgently needed to improve the effectiveness of clinical drug development for Alzheimer disease (AD). Although conventional end points lack sensitivity, digital technologies hold promise for amplifying the detection of treatment signals and capturing cognitive anomalies at earlier disease stages. Using digital technologies and combining several test modalities allow for the collection of richer information about cognitive and functional status, which is not ascertainable via conventional paper-and-pencil tests. OBJECTIVE: This study aimed to assess the psychometric properties, operational feasibility, and patient acceptance of 10 promising technologies that are to be used as efficacy end points to measure cognition in future clinical drug trials. METHODS: The Method for Evaluating Digital Endpoints in Alzheimer Disease study is an exploratory, cross-sectional, noninterventional study that will evaluate 10 digital technologies' ability to accurately classify participants into 4 cohorts according to the severity of cognitive impairment and dementia. Moreover, this study will assess the psychometric properties of each of the tested digital technologies, including the acceptable range to assess ceiling and floor effects, concurrent validity to correlate digital outcome measures to traditional paper-and-pencil tests in AD, reliability to compare test and retest, and responsiveness to evaluate the sensitivity to change in a mild cognitive challenge model. This study included 50 eligible male and female participants (aged between 60 and 80 years), of whom 13 (26%) were amyloid-negative, cognitively healthy participants (controls); 12 (24%) were amyloid-positive, cognitively healthy participants (presymptomatic); 13 (26%) had mild cognitive impairment (predementia); and 12 (24%) had mild AD (mild dementia). This study involved 4 in-clinic visits. During the initial visit, all participants completed all conventional paper-and-pencil assessments. During the following 3 visits, the participants underwent a series of novel digital assessments. RESULTS: Participant recruitment and data collection began in June 2020 and continued until June 2021. Hence, the data collection occurred during the COVID-19 pandemic (SARS-CoV-2 virus pandemic). Data were successfully collected from all digital technologies to evaluate statistical and operational performance and patient acceptance. This paper reports the baseline demographics and characteristics of the population studied as well as the study's progress during the pandemic. CONCLUSIONS: This study was designed to generate feasibility insights and validation data to help advance novel digital technologies in clinical drug development. The learnings from this study will help guide future methods for assessing novel digital technologies and inform clinical drug trials in early AD, aiming to enhance clinical end point strategies with digital technologies. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/35442.Peer reviewe

Sex differences in brain atrophy in dementia with Lewy bodies

Publisher Copyright: © 2023 The Authors. Alzheimer's & Dementia published by Wiley Periodicals LLC on behalf of Alzheimer's Association.INTRODUCTION: Sex influences neurodegeneration, but it has been poorly investigated in dementia with Lewy bodies (DLB). We investigated sex differences in brain atrophy in DLB using magnetic resonance imaging (MRI). METHODS: We included 436 patients from the European-DLB consortium and the Mayo Clinic. Sex differences and sex-by-age interactions were assessed through visual atrophy rating scales (n = 327; 73 ± 8 years, 62% males) and automated estimations of regional gray matter volume and cortical thickness (n = 165; 69 ± 9 years, 72% males). RESULTS: We found a higher likelihood of frontal atrophy and smaller volumes in six cortical regions in males and thinner olfactory cortices in females. There were significant sex-by-age interactions in volume (six regions) and cortical thickness (seven regions) across the entire cortex. DISCUSSION: We demonstrate that males have more widespread cortical atrophy at younger ages, but differences tend to disappear with increasing age, with males and females converging around the age of 75. Highlights: Male DLB patients had higher odds for frontal atrophy on radiological visual rating scales. Male DLB patients displayed a widespread pattern of cortical gray matter alterations on automated methods. Sex differences in gray matter measures in DLB tended to disappear with increasing age.Peer reviewe

Survival of patients with dementia in an Icelandic daycare center

Neðst á síðunni er hægt að nálgast greinina í heild sinni með því að smella á hlekkinn View/OpenObjective: This study was undertaken to estimate the survival of patients with dementia and to evaluate change in prognosis over time. Material and methods: Individuals attending a day care center for dementia in a 10 year period were followed until December 1st 1996 or until death. These individuals are quite repres¬entative for demented patients living in this area. Results: Of 180 individuals, 167 had either been diagnosed with Alzheimer's disease or multi infarct dementia. Ninety seven had died during follow up. Crude survival as well as relative survival was calculated with regards to the onset of symptoms of dementia. Patients with Alzheimer's disease seemed to live 40% shorter than age and gender matched individuals in the society. Conclusion: Dementia is correlated with a definite shortening of survival. Patients in this time period seem to live longer than patients diagnosed 10-15 years earlier but any comparison is hampered by different methods of diagnosis.Tilgangur: Tilgangur rannsóknarinnar var að kanna lífshorfur sjúklinga með heilabilun og athuga hvort breyting hefði orðið á horfum þeirra síðustu tvo áratugi. Efniviður og aðferðir: Einstaklingar sem sótt höfðu dagvistun á höfuðborgarsvæðinu fyrir sjúklinga með heilabilun á 10 ára tímabili voru efniviður rannsóknarinnar. Þessir einstaklingar voru dæmigerðir fyrir þennan sjúklingahóp og því talið að afdrif hans gæfi almennt góða mynd af lífshorfum sjúklinga með heilabilun á þessu landsvæði. Niðurstöður: Af 180 innrituðum sjúklingum í dagvistinni höfðu 167 Alzheimers sjúkdóm eða blóðrásartruflanir í heila. Látist höfðu 97 sjúklingar og voru afdrif þeirra skoðuð beint (crude survival) og í samanburði við jafnaldra í íslensku þjóðfélagi (relative survival). Miðað var við tímann frá upphafi sjúkdómseinkenna. Sjúklingar með Alzheimers sjúkdóm virtust lifa að meðaltali 40% skemur en jafnaldrar þeirra. Helmingur sjúklinga með Alzheimers sjúkdóm var fallinn frá eftir liðlega átta ár en helmingur sjúklinga með blóðrásartruflanir eftir tæplega átta ár. Umræða: Heilabilun hefur í för með sér um-talsverða styttingu á ævilíkum. Sjúklingar á því tímabili sem hér var skoðað virtust þó lifa nokkru lengur en sjúklingar sem fengu heilabilun 10-15 árum fyrr, en mismunandi aðferðir við greiningu gera samanburð erfiðan

Survival of patients with dementia in an Icelandic daycare center

Neðst á síðunni er hægt að nálgast greinina í heild sinni með því að smella á hlekkinn View/OpenObjective: This study was undertaken to estimate the survival of patients with dementia and to evaluate change in prognosis over time. Material and methods: Individuals attending a day care center for dementia in a 10 year period were followed until December 1st 1996 or until death. These individuals are quite repres¬entative for demented patients living in this area. Results: Of 180 individuals, 167 had either been diagnosed with Alzheimer's disease or multi infarct dementia. Ninety seven had died during follow up. Crude survival as well as relative survival was calculated with regards to the onset of symptoms of dementia. Patients with Alzheimer's disease seemed to live 40% shorter than age and gender matched individuals in the society. Conclusion: Dementia is correlated with a definite shortening of survival. Patients in this time period seem to live longer than patients diagnosed 10-15 years earlier but any comparison is hampered by different methods of diagnosis.Tilgangur: Tilgangur rannsóknarinnar var að kanna lífshorfur sjúklinga með heilabilun og athuga hvort breyting hefði orðið á horfum þeirra síðustu tvo áratugi. Efniviður og aðferðir: Einstaklingar sem sótt höfðu dagvistun á höfuðborgarsvæðinu fyrir sjúklinga með heilabilun á 10 ára tímabili voru efniviður rannsóknarinnar. Þessir einstaklingar voru dæmigerðir fyrir þennan sjúklingahóp og því talið að afdrif hans gæfi almennt góða mynd af lífshorfum sjúklinga með heilabilun á þessu landsvæði. Niðurstöður: Af 180 innrituðum sjúklingum í dagvistinni höfðu 167 Alzheimers sjúkdóm eða blóðrásartruflanir í heila. Látist höfðu 97 sjúklingar og voru afdrif þeirra skoðuð beint (crude survival) og í samanburði við jafnaldra í íslensku þjóðfélagi (relative survival). Miðað var við tímann frá upphafi sjúkdómseinkenna. Sjúklingar með Alzheimers sjúkdóm virtust lifa að meðaltali 40% skemur en jafnaldrar þeirra. Helmingur sjúklinga með Alzheimers sjúkdóm var fallinn frá eftir liðlega átta ár en helmingur sjúklinga með blóðrásartruflanir eftir tæplega átta ár. Umræða: Heilabilun hefur í för með sér um-talsverða styttingu á ævilíkum. Sjúklingar á því tímabili sem hér var skoðað virtust þó lifa nokkru lengur en sjúklingar sem fengu heilabilun 10-15 árum fyrr, en mismunandi aðferðir við greiningu gera samanburð erfiðan

The reliability of a deep learning model in clinical out-of-distribution MRI data: A multicohort study.

To access publisher's full text version of this article, please click on the hyperlink in Additional Links field or click on the hyperlink at the top of the page marked DownloadDeep learning (DL) methods have in recent years yielded impressive results in medical imaging, with the potential to function as clinical aid to radiologists. However, DL models in medical imaging are often trained on public research cohorts with images acquired with a single scanner or with strict protocol harmonization, which is not representative of a clinical setting. The aim of this study was to investigate how well a DL model performs in unseen clinical datasets-collected with different scanners, protocols and disease populations-and whether more heterogeneous training data improves generalization. In total, 3117 MRI scans of brains from multiple dementia research cohorts and memory clinics, that had been visually rated by a neuroradiologist according to Scheltens' scale of medial temporal atrophy (MTA), were included in this study. By training multiple versions of a convolutional neural network on different subsets of this data to predict MTA ratings, we assessed the impact of including images from a wider distribution during training had on performance in external memory clinic data. Our results showed that our model generalized well to datasets acquired with similar protocols as the training data, but substantially worse in clinical cohorts with visibly different tissue contrasts in the images. This implies that future DL studies investigating performance in out-of-distribution (OOD) MRI data need to assess multiple external cohorts for reliable results. Further, by including data from a wider range of scanners and protocols the performance improved in OOD data, which suggests that more heterogeneous training data makes the model generalize better. To conclude, this is the most comprehensive study to date investigating the domain shift in deep learning on MRI data, and we advocate rigorous evaluation of DL models on clinical data prior to being certified for deployment. Keywords: Clinical application; Deep learning; Domain shift; Neuroimaging.Swedish Foundation for Strategic Research Swedish Research Council Strategic Research Programme in Neuroscience at Karolinska Institutet (StratNeuro) Swedish Brain Power Swedish Research Council Stockholm County Council Karolinska Institutet Hjarnfonden DOD ADNI (Department of Defense) W81XWH-12-2-0012 United States Department of Health & Human Services National Institutes of Health (NIH) - USA NIH National Institute on Aging (NIA) United States Department of Health & Human Services National Institutes of Health (NIH) - USA NIH National Institute of Biomedical Imaging & Bioengineering (NIBIB) AbbVie Alzheimer's Association Alzheimer's Drug Discovery Foundation Araclon Biotech BioClinica, Inc. Biogen Bristol-Myers Squibb CereSpir, Inc. Cogstate Eisai Co Ltd Elan Pharmaceuticals, Inc. Eli Lilly EuroImmun Hoffmann-La Roche Roche Holding Genentech Fujirebio GE Healthcare Janssen Alzheimer Immunotherapy Research & Development, LLC. Johnson & Johnson USA Lumosity Lundbeck Corporation Merck & Company Meso Scale Diagnostics, LLC. NeuroRx Research Neurotrack Technologies Novartis Pfizer Piramal Imaging Servier Takeda Pharmaceutical Company Ltd Transition Therapeutics Canadian Institutes of Health Research (CIHR) Centrum for innovativ medicin (CIMED) Alzheimerfonden Ake Wiberg Foundation Birgitta och Sten Westerberg United States Department of Health & Human Services National Institutes of Health (NIH) - USA NIH National Institute of Neurological Disorders & Stroke (NINDS