Exploring the Usefulness of a Non-Random Holdout Sample for Model Validation: Welfare Effects on Female Behavior

Abstract

Opportunities for external validation of behavioral models in the social sciences that are based on randomized social experiments or on large regime shifts, that can be treated as experiments for the purpose of model validation, are extremely rare. In this paper, we consider an alternative approach, namely mimicking the essential element of regime change by non-randomly holding out from estimation a portion of the sample that faces a significantly different policy regime. The non-random holdout sample is used for model validation/selection. We illustrate the non-random holdout sample approach to model validation in the context of a model of welfare program participation. The policy heterogeneity that we exploit to generate a non-random holdout sample takes advantage of the wide variation across states that has existed in welfare policy.Model validation, Hold-out sample, Public welfare