Transient activation of the hypothalamic–pituitary–gonadal axis in early infancy plays an important role in male genital development and sexual differentiation of the brain, but factors contributing to individual variation in testosterone levels during this period are poorly understood. We measured salivary testosterone levels in 222 infants (119 males, 103 females, 108 singletons, 114 twins) between 2.70 and 4.80 months of age. We tested 16 major demographic and medical history variables for effects on inter-individual variation in salivary testosterone. Using the subset of twins, we estimated genetic and environmental contributions to salivary testosterone levels. Finally, we tested single nucleotide polymorphisms (SNPs) within ±5 kb of genes involved in testosterone synthesis, transport, signaling, and metabolism for associations with salivary testosterone using univariate tests and random forest (RF) analysis. We report an association between 5 min APGAR scores and salivary testosterone levels in males. Twin modeling indicated that individual variability in testosterone levels was primarily explained by environmental factors. Regarding genetic variation, univariate tests did not reveal any variants significantly associated with salivary testosterone after adjusting for false discovery rate. The top hit in males was rs10923844, an SNP of unknown function located downstream of HSD3B1 and HSD3B2. The top hits in females were two SNPs located upstream of ESR1 (rs3407085 and rs2295190). RF analysis, which reflects joint and conditional effects of multiple variants, indicated that genes involved in regulation of reproductive function, particularly LHCGR, are related to salivary testosterone levels in male infants, as are genes involved in cholesterol production, transport, and removal, while genes involved in estrogen signaling are related to salivary testosterone levels in female infants