Sickle cell disease (SCD) is the most common monogenic disorder in the world. Notably, there is extensive clinical heterogeneity in SCD that cannot be fully accounted for by known factors, and in particular, the extent to which the phenotypic diversity of SCD can be explained by genetic variation has not been reliably quantified. Here, in a family-based cohort of 449 patients with SCD and 755 relatives, we first show that 5 known modifiers affect 11 adverse outcomes in SCD to varying degrees. We then utilize a restricted maximum likelihood procedure to estimate the heritability of 20 hematologic traits, including fetal hemoglobin (HbF) and white blood cell count (WBC), in the clinically relevant context of inheritance from healthy carriers to SCD patients. We report novel estimations of heritability for HbF at 31.6% (±5.4%) and WBC at 41.2% (±6.8%) in our cohort. Finally, we demonstrate shared genetic bases between HbF, WBC, and other hematologic traits, but surprisingly little overlap between HbF and WBC themselves. In total, our analyses show that HbF and WBC have significant heritable components among individuals with SCD and their relatives, demonstrating the value of using family-based studies to better understand modifiers of SCD