We characterize the associations of total cerebral small vessel disease (SVD) burden with brain structure, trajectories of vascular risk factors, and cognitive functions in mid-to-late life. Participants were 623 community-dwelling adults from the Whitehall II Imaging Sub-study with multi-modal MRI (mean age 69.96, SD = 5.18, 79% men). We used linear mixed-effects models to investigate associations of SVD burden with up to 25-year retrospective trajectories of vascular risk and cognitive performance. General linear modelling was used to investigate concurrent associations with grey matter (GM) density and white matter (WM) microstructure, and whether these associations were modified by cognitive status (Montreal Cognitive Asessment [MoCA] scores of < 26 vs. ≥ 26). Severe SVD burden in older age was associated with higher mean arterial pressure throughout midlife (β = 3.36, 95% CI [0.42-6.30]), and faster cognitive decline in letter fluency (β = -0.07, 95% CI [-0.13--0.01]), and verbal reasoning (β = -0.05, 95% CI [-0.11--0.001]). Moreover, SVD burden was related to lower GM volumes in 9.7% of total GM, and widespread WM microstructural decline (FWE-corrected p < 0.05). The latter association was most pronounced in individuals who demonstrated cognitive impairments on MoCA (MoCA < 26; F3,608 = 2.14, p = 0.007). These findings highlight the importance of managing midlife vascular health to preserve brain structure and cognitive function in old age