Parametric estimation and tests through divergences and duality technique

We introduce estimation and test procedures through divergence optimization for discrete or continuous parametric models. This approach is based on a new dual representation for divergences. We treat point estimation and tests for simple and composite hypotheses, extending maximum likelihood technique. An other view at the maximum likelihood approach, for estimation and test, is given. We prove existence and consistency of the proposed estimates. The limit laws of the estimates and test statistics (including the generalized likelihood ratio one) are given both under the null and the alternative hypotheses, and approximation of the power functions is deduced. A new procedure of construction of confidence regions, when the parameter may be a boundary value of the parameter space, is proposed. Also, a solution to the irregularity problem of the generalized likelihood ratio test pertaining to the number of components in a mixture is given, and a new test is proposed, based on $\chi ^{2}$-divergence on signed finite measures and duality technique

Number of Instances for Reliable Feature Ranking in a Given Problem

Background: In practical use of machine learning models, users may add new features to an existing classification model, reflecting their (changed) empirical understanding of a field. New features potentially increase classification accuracy of the model or improve its interpretability. Objectives: We have introduced a guideline for determination of the sample size needed to reliably estimate the impact of a new feature. Methods/Approach: Our approach is based on the feature evaluation measure ReliefF and the bootstrap-based estimation of confidence intervals for feature ranks. Results: We test our approach using real world qualitative business-to-business sales forecasting data and two UCI data sets, one with missing values. The results show that new features with a high or a low rank can be detected using a relatively small number of instances, but features ranked near the border of useful features need larger samples to determine their impact. Conclusions: A combination of the feature evaluation measure ReliefF and the bootstrap-based estimation of confidence intervals can be used to reliably estimate the impact of a new feature in a given problem

Recover Subjective Quality Scores from Noisy Measurements

Simple quality metrics such as PSNR are known to not correlate well with subjective quality when tested across a wide spectrum of video content or quality regime. Recently, efforts have been made in designing objective quality metrics trained on subjective data (e.g. VMAF), demonstrating better correlation with video quality perceived by human. Clearly, the accuracy of such a metric heavily depends on the quality of the subjective data that it is trained on. In this paper, we propose a new approach to recover subjective quality scores from noisy raw measurements, using maximum likelihood estimation, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together. We also derive closed-from expression for the confidence interval of each estimate. Compared to previous methods which partially exploit the subjective information, our approach is able to exploit the information in full, yielding tighter confidence interval and better handling of outliers without the need for z-scoring or subject rejection. It also handles missing data more gracefully. Finally, as side information, it provides interesting insights on the test subjects and video contents.Comment: 16 pages; abridged version appeared in Data Compression Conference (DCC) 201

A new statistical test based on the wavelet cross-spectrum to detect time–frequency dependence between non-stationary signals: Application to the analysis of cortico-muscular interactions

The study of the correlations that may exist between neurophysiological signals is at the heart of modern techniques for data analysis in neuroscience. Wavelet coherence is a popular method to construct a time-frequency map that can be used to analyze the time-frequency correlations be- tween two time series. Coherence is a normalized measure of dependence, for which it is possible to construct confidence intervals, and that is commonly considered as being more interpretable than the wavelet cross-spectrum (WCS). In this paper, we provide empirical and theoretical arguments to show that a significant level of wavelet coherence does not necessarily correspond to a significant level of dependence between random signals, especially when the number of trials is small. In such cases, we demonstrate that the WCS is a much better measure of statistical dependence, and a new statistical test to detect significant values of the cross-spectrum is proposed. This test clearly outperforms the limitations of coherence analysis while still allowing a consistent estimation of the time-frequency correlations between two non-stationary stochastic processes. Simulated data are used to investigate the advantages of this new approach over coherence analysis. The method is also applied to experimental data sets to analyze the time-frequency correlations that may exist between electroencephalogram (EEG) and surface electromyogram (EMG)

Towards Using Model Averaging To Construct Confidence Intervals In Logistic Regression Models

Regression analyses in epidemiological and medical research typically begin with a model selection process, followed by inference assuming the selected model has generated the data at hand. It is well-known that this two-step procedure can yield biased estimates and invalid confidence intervals for model coefficients due to the uncertainty associated with the model selection. To account for this uncertainty, multiple models may be selected as a basis for inference. This method, commonly referred to as model-averaging, is increasingly becoming a viable approach in practice. Previous research has demonstrated the advantage of model-averaging in reducing bias of parameter estimates. However, there is lack of methods for constructing confidence intervals around parameter estimates using model-averaging. In the context of multiple logistic regression models, we propose and evaluate new confidence interval estimation approaches for regression coefficients. Specifically, we study the properties of confidence intervals constructed by averaging tail errors arising from confidence limits obtained from all models included in model-averaging for parameter estimation. We propose model-averaging confidence intervals based on the score test. For selection of models to be averaged, we propose the bootstrap inclusion fractions method. We evaluate the performance of our proposed methods using simulation studies, in a comparison with model-averaging interval procedures based on likelihood ratio and Wald tests, traditional stepwise procedures, the bootstrap approach, penalized regression, and the Bayesian model-averaging approach. Methods with good performance have been implemented in the \u27mataci\u27 R package, and illustrated using data from a low birth weight study

Tie-respecting bootstrap methods for estimating distributions of sets and functions of eigenvalues

Bootstrap methods are widely used for distribution estimation, although in some problems they are applicable only with difficulty. A case in point is that of estimating the distributions of eigenvalue estimators, or of functions of those estimators, when one or more of the true eigenvalues are tied. The $m$-out-of-$n$ bootstrap can be used to deal with problems of this general type, but it is very sensitive to the choice of $m$. In this paper we propose a new approach, where a tie diagnostic is used to determine the locations of ties, and parameter estimates are adjusted accordingly. Our tie diagnostic is governed by a probability level, $\beta$, which in principle is an analogue of $m$ in the $m$-out-of-$n$ bootstrap. However, the tie-respecting bootstrap (TRB) is remarkably robust against the choice of $\beta$. This makes the TRB significantly more attractive than the $m$-out-of-$n$ bootstrap, where the value of $m$ has substantial influence on the final result. The TRB can be used very generally; for example, to test hypotheses about, or construct confidence regions for, the proportion of variability explained by a set of principal components. It is suitable for both finite-dimensional data and functional data.Comment: Published in at http://dx.doi.org/10.3150/08-BEJ154 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm

CONFIDENCE INTERVAL ESTIMATION FOR A DIFFERENCE BETWEEN CORRELATED INTRACLASS CORRELATION COEFFICIENTS

The intraclass correlation coefficient (ICC), an index of similarity, plays an important role in a wide range of disciplines, for example in the assessment of instrument reliability. In this case, the study design may involve recruiting a sample of subjects each of whom are assessed several times with a new device and the standard. The ICC estimates for the two devices may then be compared using a test of hypothesis. However it is well known that conclusions drawn from hypothesis testing are confounded by sample size, i.e., a significant p-value can result from a sufficiently large sample size. In such cases, a confidence interval for a difference between two ICCs is more informative since it combines point estimation and hypothesis testing into a single inference statement. The sampling distribution for the ICC is well known to be left-skewed and thus confidence limits are usually constructed using Fisher’s Z-transformation or the F- distribution. Unfortunately, such an approach is not applicable to a difference between two ICCs. The remaining alternative is to apply a simple asymptotic approach, i.e., point estimate plus/minus normal quantile multiplied by the estimate of standard error. However this method is known to perform poorly because it ignores the features of the underlying sampling distribution. In this thesis I develop a confidence interval procedure using the method of variance estimate recovery (MOVER). Specifically, the variance estimates required for the upper and lower limits of a difference are recovered from those obtained for separate ICCs. An advantage of this approach is that it provides a confidence interval that reflects the underlying sampling distribution. Simulation results show that the MOVER method performs very well in terms of overall coverage percentage and tail errors. Two data sets are used to illustrate this procedure

CONFIDENCE INTERVAL ESTIMATION FOR A DIFFERENCE BETWEEN CORRELATED INTRACLASS CORRELATION COEFFICIENTS

The intraclass correlation coefficient (ICC), an index of similarity, plays an important role in a wide range of disciplines, for example in the assessment of instrument reliability. In this case, the study design may involve recruiting a sample of subjects each of whom are assessed severe^ times with a new device and the standard. The ICC estimates for the two devices may then be compared using a test of hypothesis. However it is well known that conclusions drawn from hypothesis testing are confounded by sample size, i.e., a significant p-value can result from a sufficiently large sample size. In such cases, a confidence interval for a difference between two ICCs is more informative since it combines point estimation and hypothesis testing into a single inference statement. The sampling distribution for the ICC is well known to be left-skewed and thus confidence limits are usually constructed using Fisher’s Z-transformation or the F- distribution. Unfortunately, such an approach is not applicable to a difference between two ICCs. The remaining alternative is to apply a simple asymptotic approach, i.e., point estimate plus/minus normal quantile multiplied by the estimate of standard error. However this method is known to perform poorly because it ignores the features of the underlying sampling distribution. In this thesis I develop a confidence interval procedure using the method of variance estimate recovery (MOVER). Specifically, the variance estimates required for the upper and lower limits of a difference are iii recovered from those obtained for separate ICCs. An advantage of this approach is that it provides a confidence interval that reflects the underlying sampling distribution. Simulation results show that the MOVER method performs very well in terms of overall coverage percentage and tail errors. Two data sets are used to illustrate this procedure