MML Probabilistic Principal Component Analysis

Principal component analysis (PCA) is perhaps the most widely method for data dimensionality reduction. A key question in PCA decomposition of data is deciding how many factors to retain. This manuscript describes a new approach to automatically selecting the number of principal components based on the Bayesian minimum message length method of inductive inference. We also derive a new estimate of the isotropic residual variance and demonstrate, via numerical experiments, that it improves on the usual maximum likelihood approach

On Statistical Modelling and Hypothesis Testing by Information Theoretic Methods

The main objective of this thesis is to study various information theoretic methods and criteria in the context of statistical model selection. The focus in this research is on Rissanen’s Minimum Description Length (MDL) principle and its variants, with a special emphasis on the Normalized Maximum Likelihood (NML).We extend the Rissanen methodology for coping with infinite parametric complexity and discuss two particular cases. This is applied for deriving four NMLcriteria and investigate their performance. Furthermore, we find the connection between Stochastic Complexity (SC), defined as minus logarithm of NML, and other model selection criteria.We also study the use of information theoretic criteria (ITC) for selecting the order of autoregressive (AR) models in the presence of nonstationarity. In particular, we give a modified version of Sequentially NML (SNML) when the model parameters are estimated by forgetting factor LS algorithm.Another contribution of the thesis is in connection with the new approach for composite hypothesis testing using Optimally Distinguishable Distributions (ODD). The ODD-detector for subspace signals in Gaussian noise is introduced and its performance is evaluated.Additionally, we exploit the Kolmogorov Structure Function (KSF) to derive a new criterion for cepstral nulling, which has been recently applied to the problem of periodogram smoothing.Finally, the problem of fairness in multiaccess communication systems is investigated and a new method is proposed. The new approach is based on partitioning the network into subnetworks and employing two different multiple-access schemes within and across subnetworks. It is also introduced an algorithm for selecting optimally the subnetworks such that to achieve the max-min fairness

Statistical and Information Theoretic Approaches to Model Selection and Averaging

In this thesis we consider model selection (MS) and its alternative, model averaging (MA), in seven research articles and in an introductory summary of the articles. The utilization of the minimum description length (MDL) principle is a common theme in five articles. In three articles we approach MA by estimating model weights using MDL and by making use of the idea of shrinkage estimation with special emphasis on the weighted average least squares (WALS) and penalized least squares (PenLS) estimation. We also apply MS and MA techniques to data on hip fracture treatment costs in seven hospital districts in Finland. Implementation of the MDL principle for MS is put into action by using the normalized maximum likelihood (NML). However, the straightforward use of the NML technique in Gaussian linear regression fails because the normalization coeffcient is not finite. Rissanen has proposed an elegant solution to the problem by constraining the data space properly. We demonstrate the effect of data constraints on the MS criterion and present a general convex constraint in data space and disscuss two particular cases: the rhomboidal and ellipsoidal constraints. From these findings we derive four new NML-based criteria. One particular constraint is related to the case when collinearity is present in data. We study WALS estimation which has the potential for a good risk profile. WALS is attractive in regression especially when the number of explanatory variables is large because its computational burden is light. We also apply WALS to estimation and comparison of hip fracture treatment costs between hospital districts in Finland. We present the WALS estimators as a special case of shrinkage estimators and we characterize a class of shrinkage estimators for which we derive the effciency bound. We demonstrate how shrinkage estimators are obtained by using the PenLS technique and we prove suffcient conditions for the PenLS estimator to belong to the class of shrinkage estimators. Through this connection we may derive new MA estimators and effectively utilize certain previously known estimators in MA. We also study the performance of the estimators by using simulation experiments based on hip fracture treatment cost data. We propose an MA estimator with weights selected by the NML criterion. The resulting mixture estimator usually performs better than the corresponding MS estimator. We report on simulation experiments where the performance potential of MDL weight selection is compared with the corresponding potential of the AIC, BIC and Mallow's MA estimators. We also exploit the finding that a smoothing spline estimator may be rewritten as a linear mixed model (LMM). We present the NML criterion for LMM's and propose an automatic data-based smoothing method based on this criterion. The performance of the MDL criterion is compared to AIC, BIC and generalized cross-validation criteria in simulation experiments. Finally we consider the sequential NML (sNML) criterion in logistic regression. We show that while the NML criterion becomes quickly computationally infeasible as the number of covariates and amount of data increases, the sNML criterion can still be exploited in MS. We also develop a risk adjustment model for hip fracture mortality in Finland by choosing comorbidities that have an effect on mortality after hip fracture