Objectives: To examine the association between hospitalization, critical illness, and infection occurring during middle- and late-life and structural brain abnormalities in older adults. Design: Prospective cohort study. Setting: Atherosclerosis Risk in Communities (ARIC) Study. Participants: A community sample of adults who were 44 to 66 years of age at study baseline. Measurements: Active surveillance of local hospitals and annual participant contact were used to gather hospitalization information (including International Classification of Diseases, Ninth Revision, codes) on all participants over a 24-year surveillance period. Subsequently, a subset of participants underwent 3-Tesla brain magnetic resonance imaging (MRI) to quantify total and regional brain volumes, white matter hyperintensity (WMH) volume, and white matter microstructural integrity (fractional anisotropy (FA) and mean diffusivity (MD) as measured using diffusion tensor imaging (DTI)). Results: Of the 1,689 participants included (mean age at MRI 76±5), 72% were hospitalized, 14% had a major infection, and 4% had a critical illness during the surveillance period. Using covariate-adjusted regression, hospitalization was associated with 0.12–standard deviation (SD) greater WMH volume (95% confidence interval (CI)=0.00–0.24) and poorer white matter microstructural integrity (0.17-SD lower FA, 95% CI=–0.27 to –0.06; 0.16-SD greater MD, 95% CI=0.07–0.25) than no hospitalization. There was a dose-dependent relationship between number of hospitalizations, smaller brain volumes, and lower white matter integrity (p-trends ≤.048). In hospitalized participants, critical illness was associated with smaller Alzheimer's disease (AD) signature region (–1.64 cm3, 95% CI=–3.16 to –0.12); major infection was associated with smaller AD signature region (–1.28 cm3, 95% CI=–2.21 to –0.35) and larger ventricular volume (3.79 cm3, 95% CI= 0.81–6.77). Conclusions: Whereas all-cause hospitalization was primarily associated with lower white matter integrity, critical illness and major infection were associated with smaller brain volume, particularly within regions implicated in AD