We model sex-structured population dynamics to analyze pairwise competition
between groups differing both genetically and culturally. A sex-ratio allele is
expressed in the heterogametic sex only, so that assumptions of Fisher's
analysis do not apply. Sex-ratio evolution drives cultural evolution of a
group-associated trait governing mortality in the homogametic sex. The two-sex
dynamics under resource limitation induces a strong Allee effect that depends
on both sex ratio and cultural trait values. We describe the resulting
threshold, separating extinction from positive growth, as a function of female
and male densities. When initial conditions avoid extinction due to the Allee
effect, different sex ratios cannot coexist; in our model, greater female
allocation always invades and excludes a lesser allocation. But the culturally
transmitted trait interacts with the sex ratio to determine the ecological
consequences of successful invasion. The invading female allocation may permit
population persistence at self-regulated equilibrium. For this case, the
resident culture may be excluded, or may coexist with the invader culture. That
is, a single sex-ratio allele in females and a cultural dimorphism in male
mortality can persist; a low-mortality resident trait is maintained by
father-to-son cultural transmission. Otherwise, the successfully invading
female allocation excludes the resident allele and culture, and then drives the
population to extinction via a shortage of males. Finally, we show that the
results obtained under homogeneous mixing hold, with caveats, in a spatially
explicit model with local mating and diffusive dispersal in both sexes.Comment: final version, reflecting changes in response to referees' comment