Model Diagnostics meets Forecast Evaluation: Goodness-of-Fit, Calibration, and Related Topics

Principled forecast evaluation and model diagnostics are vital in fitting probabilistic models and forecasting outcomes of interest. A common principle is that fitted or predicted distributions ought to be calibrated, ideally in the sense that the outcome is indistinguishable from a random draw from the posited distribution. Much of this thesis is centered on calibration properties of various types of forecasts. In the first part of the thesis, a simple algorithm for exact multinomial goodness-of-fit tests is proposed. The algorithm computes exact $p$-values based on various test statistics, such as the log-likelihood ratio and Pearson\u27s chi-square. A thorough analysis shows improvement on extant methods. However, the runtime of the algorithm grows exponentially in the number of categories and hence its use is limited. In the second part, a framework rooted in probability theory is developed, which gives rise to hierarchies of calibration, and applies to both predictive distributions and stand-alone point forecasts. Based on a general notion of conditional T-calibration, the thesis introduces population versions of T-reliability diagrams and revisits a score decomposition into measures of miscalibration, discrimination, and uncertainty. Stable and efficient estimators of T-reliability diagrams and score components arise via nonparametric isotonic regression and the pool-adjacent-violators algorithm. For in-sample model diagnostics, a universal coefficient of determination is introduced that nests and reinterprets the classical $R^2$ in least squares regression. In the third part, probabilistic top lists are proposed as a novel type of prediction in classification, which bridges the gap between single-class predictions and predictive distributions. The probabilistic top list functional is elicited by strictly consistent evaluation metrics, based on symmetric proper scoring rules, which admit comparison of various types of predictions

Monte Carlo Confidence Sets for Identified Sets

In complicated/nonlinear parametric models, it is generally hard to know whether the model parameters are point identified. We provide computationally attractive procedures to construct confidence sets (CSs) for identified sets of full parameters and of subvectors in models defined through a likelihood or a vector of moment equalities or inequalities. These CSs are based on level sets of optimal sample criterion functions (such as likelihood or optimally-weighted or continuously-updated GMM criterions). The level sets are constructed using cutoffs that are computed via Monte Carlo (MC) simulations directly from the quasi-posterior distributions of the criterions. We establish new Bernstein-von Mises (or Bayesian Wilks) type theorems for the quasi-posterior distributions of the quasi-likelihood ratio (QLR) and profile QLR in partially-identified regular models and some non-regular models. These results imply that our MC CSs have exact asymptotic frequentist coverage for identified sets of full parameters and of subvectors in partially-identified regular models, and have valid but potentially conservative coverage in models with reduced-form parameters on the boundary. Our MC CSs for identified sets of subvectors are shown to have exact asymptotic coverage in models with singularities. We also provide results on uniform validity of our CSs over classes of DGPs that include point and partially identified models. We demonstrate good finite-sample coverage properties of our procedures in two simulation experiments. Finally, our procedures are applied to two non-trivial empirical examples: an airline entry game and a model of trade flows

Contributions to Mediation Analysis and First Principles Modeling for Mechanistic Statistical Analysis

This thesis contains three projects that propose novel methods for studying mechanisms that explain statistical relationships. The ultimate goal of each of these methods is to help researchers describe how or why complex relationships between observed variables exist. The first project proposes and studies a method for recovering mediation structure in high dimensions. We take a dimension reduction approach that generalizes the ``product of coefficients'' concept for univariate mediation analysis through the optimization of a loss function. We devise an efficient algorithm for optimizing the product-of-coefficients inspired loss function. Through extensive simulation studies, we show that the method is capable of consistently identifying mediation structure. Finally, two case studies are presented that demonstrate how the method can be used to conduct multivariate mediation analysis. The second project uses tools from conditional inference to improve the calibration of tests of univariate mediation hypotheses. The key insight of the project is that the non-Euclidean geometry of the null parameter space causes the test statistic’s sampling distribution to depend on a nuisance parameter. After identifying a statistic that is both sufficient for the nuisance parameter and approximately ancillary for the parameter of interest, we derive the test statistic’s limiting conditional sampling distribution. We additionally develop a non-standard bootstrap procedure for calibration in finite samples. We demonstrate through simulation studies that improved evidence calibration leads to substantial power increases over existing methods. This project suggests that conditional inference might be a useful tool in evidence calibration for other non-standard or otherwise challenging problems. In the last project, we present a methodological contribution to a pharmaceutical science study of {em in vivo} ibuprofen pharmacokinetics. We demonstrate how model misspecification in a first-principles analysis can be addressed by augmenting the model to include a term corresponding to an omitted source of variation. In previously used first-principles models, gastric emptying, which is pulsatile and stochastic, is modeled as first-order diffusion for simplicity. However, analyses suggest that the actual gastric emptying process is expected to be a unimodal smooth function, with phase and amplitude varying by subject. Therefore, we adopt a flexible approach in which a highly idealized parametric version of gastric emptying is combined with a Gaussian process to capture deviations from the idealized form. These functions are characterized by their distributions, which allows us to learn their common and unique features across subjects despite that these features are not directly observed. Through simulation studies, we show that the proposed approach is able to identify certain features of latent function distributions.PHDStatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/163026/1/josephdi_1.pd

Untangling hotel industry’s inefficiency: An SFA approach applied to a renowned Portuguese hotel chain

The present paper explores the technical efficiency of four hotels from Teixeira Duarte Group - a renowned Portuguese hotel chain. An efficiency ranking is established from these four hotel units located in Portugal using Stochastic Frontier Analysis. This methodology allows to discriminate between measurement error and systematic inefficiencies in the estimation process enabling to investigate the main inefficiency causes. Several suggestions concerning efficiency improvement are undertaken for each hotel studied.info:eu-repo/semantics/publishedVersio

Some contributions to the analysis of skew data on the line and circle

In the first part of this thesis we consider the skew-normal class of distributions on the line and its limiting general half-normal distribution. Inferential procedures based on the methods of moments and maximum likelihood are developed and their performance assessed using simulation. Data on the strength of glass fibre and the body fat of elite athletes are used to illustrate some of the inferential issues raised. The second part of the thesis is devoted to a consideration of the analysis of skew circular data. First, we derive the large-sample distribution of certain key circular statistics and show how this result provides a basis for inference for the corresponding population measures. Next, tests for circular reflective symmetry about an unknown central direction are investigated. A large-sample test and computer intensive variants of it are developed, and their operating characteristics explored both theoretically and empirically. Subsequently, we consider tests for circular reflective symmetry about a known or specified median axis. Two new procedures are developed for testing for symmetry about a known median axis against skew alternatives, and their operating characteristics compared in a simulation experiment with those of the circular analogues of three linear tests. On the basis of the results obtained from the latter, a simple testing strategy is identified. The performance of the tests against rotation alternatives is also investigated. Throughout, the use of the various tests of symmetry is illustrated using a wide range of circular data sets. Finally, we propose the wrapped skew-normal distribution on the circle as a potential model for circular data. The distribution’s fundamental properties are presented and inference based on the methods of moments and maximum likelihood is explored. Tests for limiting cases of the class are proposed, and a potential use of the distribution is illustrated in the mixture based modelling of data on bird migration

Latent data modeling with biostatistical applications

This dissertation consists of three projects that make use of latent variable modeling techniques. One of the focuses of this dissertation research has been in the area of spatial and spatio-temporal modeling. The specific topics and motivating problems in this study have been fully supported and motivated by the Companion Animal Parasite Council (CAPC). In particular, the CAPC has developed a rather extensive database, which houses several common dog disease data sets collected throughout the conterminous United States. This data exists at a county level and was collected monthly over a span of 5 consecutive years, and exhibits strong spatial and temporal correlation structures. Further, due to non-reporting counties a significant portion of the data is missing, both in the spatial and temporal domain. The goal of our work in this area was to identify risk factors significantly related to the prevalence of the various diseases and to develop models which could be used to accurately forecast future disease trends nationwide. No similar work has been completed for these diseases on the spatio-temporal scale that we consider. To accomplish this task, we developed and implemented a Bayesian spatio-temporal regression model to analyze the data. Due to the relatively large spatial scale and complex structure of the data, a key challenge was developing computationally efficient algorithms that could be used to implement Markov chain Monte Carlo (MCMC) techniques. Once this was completed, we implemented our models to assess the relevance of the considered covariates and to forecast future trends. In addition to the spatial and spatio-temporal modeling problems, this dissertation research also focus on developing new modeling techniques for data collected on pooled specimens. The concept of using pooling as a more cost effective data collection technique is becoming pervasive in the biological sciences and elsewhere. In particular pooled data is collected by first amalgamating several specimens (e.g., blood, urine, etc.), collected from individuals, into a pooled sample, this pooled sample is then measured for a characteristic of interest; e.g., in infectious disease studies the pooled outcome is typically binary indicating disease status and in biological marker (i.e., biomarker) evaluation studies the outcome is continuous. In either case, information on several individuals is obtained at the expense of making only one measurement, thus reducing the cost of data collection. However, the statistical analysis of measurements (either binary or continuous) taken on pools is often fraught with many challenges. In my dissertation research, I have considered developing regression methods for both continuous and binary outcomes measured on pools. For continuous outcomes, I proposed a general regression framework which can be used to analyze pooled outcomes under practically all parametric models. This was accomplished through the use of an advanced Monte Carlo sampling algorithm, which was implemented to approximate the observed data likelihood. Proceeding in this fashion, also allows us to account for measurement error, which has not been accounted for previously, and led to the development of computationally efficient software which can be used to implement the proposed approach. For binary outcomes (usually referred to as group testing data), I developed a novel Bayesian generalized additive model. Specifically, the proposed approach assumes the linear predictor depends on several unknown smooth functions of some covariates as well as linear combinations of other covariates. In addition, our model can account for imperfect testing, and can be used to analyze data collected according to any group testing process

Cross-Validated Decision Trees with Targeted Maximum Likelihood Estimation for Nonparametric Causal Mixtures Analysis

Exposure to mixtures of chemicals, such as drugs, pollutants, and nutrients, is common in real-world exposure or treatment scenarios. To understand the impact of these exposures on health outcomes, an interpretable and important approach is to estimate the causal effect of exposure regions that are most associated with a health outcome. This requires a statistical estimator that can identify these exposure regions and provide an unbiased estimate of a causal target parameter given the region. In this work, we present a methodology that uses decision trees to data-adaptively determine exposure regions and employs cross-validated targeted maximum likelihood estimation to unbiasedly estimate the average regional-exposure effect (ARE). This results in a plug-in estimator with an asymptotically normal distribution and minimum variance, from which confidence intervals can be derived. The methodology is implemented in the open-source software, CVtreeMLE, a package in R. Analysts put in a vector of exposures, covariates and an outcome and tables are given for regions in the exposures, such as lead > 2.1 & arsenic > 1.4, with an associated ARE which represents the mean outcome difference if all individuals were exposed to this region compared to if none were exposed to this region. CVtreeMLE enables researchers to discover interpretable exposure regions in mixed exposure scenarios and provides robust statistical inference for the impact of these regions. The resulting quantities offer interpretable thresholds that can inform public health policies, such as pollutant regulations, or aid in medical decision-making, such as identifying the most effective drug combinations

Multiple tests of association with biological annotation metadata

We propose a general and formal statistical framework for multiple tests of association between known fixed features of a genome and unknown parameters of the distribution of variable features of this genome in a population of interest. The known gene-annotation profiles, corresponding to the fixed features of the genome, may concern Gene Ontology (GO) annotation, pathway membership, regulation by particular transcription factors, nucleotide sequences, or protein sequences. The unknown gene-parameter profiles, corresponding to the variable features of the genome, may be, for example, regression coefficients relating possibly censored biological and clinical outcomes to genome-wide transcript levels, DNA copy numbers, and other covariates. A generic question of great interest in current genomic research regards the detection of associations between biological annotation metadata and genome-wide expression measures. This biological question may be translated as the test of multiple hypotheses concerning association measures between gene-annotation profiles and gene-parameter profiles. A general and rigorous formulation of the statistical inference question allows us to apply the multiple hypothesis testing methodology developed in [Multiple Testing Procedures with Applications to Genomics (2008) Springer, New York] and related articles, to control a broad class of Type I error rates, defined as generalized tail probabilities and expected values for arbitrary functions of the numbers of Type I errors and rejected hypotheses. The resampling-based single-step and stepwise multiple testing procedures of [Multiple Testing Procedures with Applications to Genomics (2008) Springer, New York] take into account the joint distribution of the test statistics and provide Type I error control in testing problems involving general data generating distributions (with arbitrary dependence structures among variables), null hypotheses, and test statistics.Comment: Published in at http://dx.doi.org/10.1214/193940307000000446 the IMS Collections (http://www.imstat.org/publications/imscollections.htm) by the Institute of Mathematical Statistics (http://www.imstat.org