Randomized goodness of fit tests

summary:Classical goodness of fit tests are no longer asymptotically distributional free if parameters are estimated. For a parametric model and the maximum likelihood estimator the empirical processes with estimated parameters is asymptotically transformed into a time transformed Brownian bridge by adding an independent Gaussian process that is suitably constructed. This randomization makes the classical tests distributional free. The power under local alternatives is investigated. Computer simulations compare the randomized Cramér-von Mises test with tests specially designed for location-scale families, such as the Shapiro-Wilk and the Shenton-Bowman test for normality and with the Epps-Pulley test for exponentiality

Recursive Partitioning for Heterogeneous Causal Effects

In this paper we study the problems of estimating heterogeneity in causal effects in experimental or observational studies and conducting inference about the magnitude of the differences in treatment effects across subsets of the population. In applications, our method provides a data-driven approach to determine which subpopulations have large or small treatment effects and to test hypotheses about the differences in these effects. For experiments, our method allows researchers to identify heterogeneity in treatment effects that was not specified in a pre-analysis plan, without concern about invalidating inference due to multiple testing. In most of the literature on supervised machine learning (e.g. regression trees, random forests, LASSO, etc.), the goal is to build a model of the relationship between a unit's attributes and an observed outcome. A prominent role in these methods is played by cross-validation which compares predictions to actual outcomes in test samples, in order to select the level of complexity of the model that provides the best predictive power. Our method is closely related, but it differs in that it is tailored for predicting causal effects of a treatment rather than a unit's outcome. The challenge is that the "ground truth" for a causal effect is not observed for any individual unit: we observe the unit with the treatment, or without the treatment, but not both at the same time. Thus, it is not obvious how to use cross-validation to determine whether a causal effect has been accurately predicted. We propose several novel cross-validation criteria for this problem and demonstrate through simulations the conditions under which they perform better than standard methods for the problem of causal effects. We then apply the method to a large-scale field experiment re-ranking results on a search engine

New goodness-of-fit diagnostics for conditional discrete response models

This paper proposes new specification tests for conditional models with discrete responses, which are key to apply efficient maximum likelihood methods, to obtain consistent estimates of partial effects and to get appropriate predictions of the probability of future events. In particular, we test the static and dynamic ordered choice model specifications and can cover infinite support distributions for e.g. count data. The traditional approach for specification testing of discrete response models is based on probability integral transforms of a jittered discrete data which leads to continuous uniform iid series under the true conditional distribution. Then, standard specification testing techniques for continuous variables could be applied to the transformed series, but the extra randomness from jitters affects the power properties of these methods. We investigate in this paper an alternative transformation based only on original discrete data that avoids any randomization. We analyze the asymptotic properties of goodness-of-fit tests based on this new transformation and explore the properties in finite samples of a bootstrap algorithm to approximate the critical values of test statistics which are model and parameter dependent. We show analytically and in simulations that our approach dominates the methods based on randomization in terms of power. We apply the new tests to models of the monetary policy conducted by the Federal Reserve

The Factor Structure of the Shortened Version of the Working Alliance Inventory

1st Place in Denman Undergraduate Research Forum for PsychologyIn research on the process of change in psychotherapy, perhaps no variable has received more attention than the therapeutic alliance. Measures of the alliance characterize the level of agreement between therapist and client on treatment goals, the level of agreement on how to accomplish those goals, and the affective bond between therapist and client. One of the most widely used measures of the alliance is the 12-item Working Alliance Inventory (WAI-S, shortened version). However, the factor structure underlying the WAI-S remains unclear. Most often researchers have used a total score from the WAI-S, implying a single latent factor. The authors of the WAI-S originally suggested the WAI-S was composed of three distinct factors (i.e., Task, Goal, and Bond). An exploratory factor analysis of the WAI-S in a relatively small sample suggested two factors: Agreement and Relationship (Andrusyna, Tang, DeRubeis, & Luborsky, 2001). To examine the different factor structures proposed, we drew data from three independent samples of depressed patients participating in cognitive therapy for depression. In this combined sample of 207 patients, we used confirmatory factor analyses to compare the fit of the previously proposed one, two, and three factor models of the WAI-S. Using item scores from the third therapy session, our results support a two-factor solution consisting of Agreement and Relationship factors. All fit indices examined favored the two-factor model over competing models. Additional analyses suggest this factor structure applied to ratings of the alliance made by therapists, clients and observers. Our results clarify the factor structure of the WAI-S and should inform future research on the therapeutic alliance.A five-year embargo was granted for this item.Academic Major: Psycholog