Cardiovascular health in midlife is an established risk factor for cognitive function later in life. Knowing mechanisms of this association may allow preventative steps to be taken to preserve brain health and cognitive performance in older age. In this study, we investigated the association of the Framingham stroke-risk score, a validated multifactorial predictor of 10-year risk of stroke, with brain measures and cognitive performance in stroke-free individuals. We used a large (N = 800) longitudinal cohort of community-dwelling adults of the Whitehall II imaging sub-study with no obvious structural brain abnormalities, who had Framingham stroke risk measured five times between 1991 and 2013 and MRI measures of structural integrity, and cognitive function performed between 2012 and 2016 [baseline mean age 47.9 (5.2) years, range 39.7-62.7 years; MRI mean age 69.81 (5.2) years, range 60.3-84.6 years; 80.6% men]. Unadjusted linear associations were assessed between the Framingham stroke-risk score in each wave and voxelwise grey matter density, fractional anisotropy and mean diffusivity at follow-up. These analyses were repeated including socio-demographic confounders as well as stroke risk in previous waves to examine the effect of residual risk acquired between waves. Finally, we used structural equation modelling to assess whether stroke risk negatively affects cognitive performance via specific brain measures. Higher unadjusted stroke risk measured at each of the five waves over 20 years prior to the MRI scan was associated with lower voxelwise grey and white matter measures. After adjusting for socio-demographic variables, higher stroke risk from 1991 to 2009 was associated with lower grey matter volume in the medial temporal lobe. Higher stroke risk from 1997 to 2013 was associated with lower fractional anisotropy along the corpus callosum. In addition, higher stroke risk from 2012 to 2013, sequentially adjusted for risk measured in 1991-94, 1997-98 and 2002-04 (i.e. 'residual risks' acquired from the time of these examinations onwards), was associated with widespread lower fractional anisotropy, and lower grey matter volume in sub-neocortical structures. Structural equation modelling suggested that such reductions in brain integrity were associated with cognitive impairment. These findings highlight the importance of considering cerebrovascular health in midlife as important for brain integrity and cognitive function later in life (ClinicalTrials.gov Identifier: NCT03335696)