Social cognition includes understanding the mental states (thoughts, feelings, intentions, desires, and beliefs) of others - so-called 'theory of mind' or 'mindreading'. Recent studies have shown an impact of age and sex. Here, we applied the 'Reading the Mind in the Eyes' Test (RMET) that measures the ability to identify mental states from the eye region of the face. RMET accuracy was measured and analyzed in a large population-based sample (N = 1603) across the whole adult age-range from 19 to 79 years with effect size analyses (Hedges' g). Overall test performance was lower in older than younger women and men, whereas differences between women and men were almost negligible across the whole cohort. In a further analysis focusing on age-specific sex differences, RMET accuracy was higher for women below 45 years compared to men. This sex effect nearly vanished in older people above 45 years of age. Results were verified in a sub-cohort after excluding participants with neurological and psychiatric conditions, and with another cut-off, i.e. 50 years of age. In conclusion, results suggest that mindreading declines with age. Overall sex effects were small and results suggest that age-related hormonal and social factors may impact mental state perception. Future mega-analyses and longitudinal studies including hormonal and social measures are needed to validate the interaction between RMET performance, aging and sex