How Good are Our Measures? Investigating the Appropriate Use of Factor Analysis for Survey Instruments

Background: Evaluation work frequently utilizes factor analysis to establish the dimensionality, reliability, and stability of surveys. However, survey data is typically ordinal, violating the assumptions of most statistical methods, and thus is often factor-analyzed inappropriately. Purpose: This study illustrates the salient analytical decisions for factor-analyzing ordinal survey data appropriately and demonstrates the repercussions of inappropriate analyses. Setting: The data used for this study are drawn from an evaluation of the efficacy of a drama-based approach to teaching Shakespeare in elementary and middle school.  Intervention: Not applicable. Research Design: Survey research. Data Collection and Analysis: Four factor analytic methods were compared: a traditional exploratory factor analysis (EFA), a full-information EFA, and two EFAs within the confirmatory factor analysis framework (E/CFA) conducted according to the Jöreskog method and the Gugiu method. Findings: Methods appropriate for ordinal data produce better models, the E/CFAs outperform the EFAs, and the Gugiu method demonstrates greater model interpretability and stability than the Jöreskog method. These results suggest that the Gugiu E/CFA may be the preferable factor analytic method for use with ordinal data. Practical applications of these findings are discussed. Keywords: factor analysis; ordinal data; E/CFA; survey research

Analysis-of-marginal-Tail-Means (ATM): a robust method for discrete black-box optimization

We present a new method, called Analysis-of-marginal-Tail-Means (ATM), for effective robust optimization of discrete black-box problems. ATM has important applications to many real-world engineering problems (e.g., manufacturing optimization, product design, molecular engineering), where the objective to optimize is black-box and expensive, and the design space is inherently discrete. One weakness of existing methods is that they are not robust: these methods perform well under certain assumptions, but yield poor results when such assumptions (which are difficult to verify in black-box problems) are violated. ATM addresses this via the use of marginal tail means for optimization, which combines both rank-based and model-based methods. The trade-off between rank- and model-based optimization is tuned by first identifying important main effects and interactions, then finding a good compromise which best exploits additive structure. By adaptively tuning this trade-off from data, ATM provides improved robust optimization over existing methods, particularly in problems with (i) a large number of factors, (ii) unordered factors, or (iii) experimental noise. We demonstrate the effectiveness of ATM in simulations and in two real-world engineering problems: the first on robust parameter design of a circular piston, and the second on product family design of a thermistor network

Social Welfare in One-sided Matching Markets without Money

We study social welfare in one-sided matching markets where the goal is to efficiently allocate n items to n agents that each have a complete, private preference list and a unit demand over the items. Our focus is on allocation mechanisms that do not involve any monetary payments. We consider two natural measures of social welfare: the ordinal welfare factor which measures the number of agents that are at least as happy as in some unknown, arbitrary benchmark allocation, and the linear welfare factor which assumes an agent's utility linearly decreases down his preference lists, and measures the total utility to that achieved by an optimal allocation. We analyze two matching mechanisms which have been extensively studied by economists. The first mechanism is the random serial dictatorship (RSD) where agents are ordered in accordance with a randomly chosen permutation, and are successively allocated their best choice among the unallocated items. The second mechanism is the probabilistic serial (PS) mechanism of Bogomolnaia and Moulin [8], which computes a fractional allocation that can be expressed as a convex combination of integral allocations. The welfare factor of a mechanism is the infimum over all instances. For RSD, we show that the ordinal welfare factor is asymptotically 1/2, while the linear welfare factor lies in the interval [.526, 2/3]. For PS, we show that the ordinal welfare factor is also 1/2 while the linear welfare factor is roughly 2/3. To our knowledge, these results are the first non-trivial performance guarantees for these natural mechanisms

A Bayesian semiparametric latent variable model for mixed responses

In this article we introduce a latent variable model (LVM) for mixed ordinal and continuous responses, where covariate effects on the continuous latent variables are modelled through a flexible semiparametric predictor. We extend existing LVM with simple linear covariate effects by including nonparametric components for nonlinear effects of continuous covariates and interactions with other covariates as well as spatial effects. Full Bayesian modelling is based on penalized spline and Markov random field priors and is performed by computationally efficient Markov chain Monte Carlo (MCMC) methods. We apply our approach to a large German social science survey which motivated our methodological development