51,739 research outputs found
Kronecker Sum Decompositions of Space-Time Data
In this paper we consider the use of the space vs. time Kronecker product
decomposition in the estimation of covariance matrices for spatio-temporal
data. This decomposition imposes lower dimensional structure on the estimated
covariance matrix, thus reducing the number of samples required for estimation.
To allow a smooth tradeoff between the reduction in the number of parameters
(to reduce estimation variance) and the accuracy of the covariance
approximation (affecting estimation bias), we introduce a diagonally loaded
modification of the sum of kronecker products representation [1]. We derive a
Cramer-Rao bound (CRB) on the minimum attainable mean squared predictor
coefficient estimation error for unbiased estimators of Kronecker structured
covariance matrices. We illustrate the accuracy of the diagonally loaded
Kronecker sum decomposition by applying it to video data of human activity.Comment: 5 pages, 8 figures, accepted to CAMSAP 201
From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation
In statistical prediction, classical approaches for model selection and model
evaluation based on covariance penalties are still widely used. Most of the
literature on this topic is based on what we call the "Fixed-X" assumption,
where covariate values are assumed to be nonrandom. By contrast, it is often
more reasonable to take a "Random-X" view, where the covariate values are
independently drawn for both training and prediction. To study the
applicability of covariance penalties in this setting, we propose a
decomposition of Random-X prediction error in which the randomness in the
covariates contributes to both the bias and variance components. This
decomposition is general, but we concentrate on the fundamental case of least
squares regression. We prove that in this setting the move from Fixed-X to
Random-X prediction results in an increase in both bias and variance. When the
covariates are normally distributed and the linear model is unbiased, all terms
in this decomposition are explicitly computable, which yields an extension of
Mallows' Cp that we call . also holds asymptotically for certain
classes of nonnormal covariates. When the noise variance is unknown, plugging
in the usual unbiased estimate leads to an approach that we call ,
which is closely related to Sp (Tukey 1967), and GCV (Craven and Wahba 1978).
For excess bias, we propose an estimate based on the "shortcut-formula" for
ordinary cross-validation (OCV), resulting in an approach we call .
Theoretical arguments and numerical simulations suggest that is
typically superior to OCV, though the difference is small. We further examine
the Random-X error of other popular estimators. The surprising result we get
for ridge regression is that, in the heavily-regularized regime, Random-X
variance is smaller than Fixed-X variance, which can lead to smaller overall
Random-X error
A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models
Generative models, like large language models, are becoming increasingly
relevant in our daily lives, yet a theoretical framework to assess their
generalization behavior and uncertainty does not exist. Particularly, the
problem of uncertainty estimation is commonly solved in an ad-hoc manner and
task dependent. For example, natural language approaches cannot be transferred
to image generation. In this paper we introduce the first
bias-variance-covariance decomposition for kernel scores and their associated
entropy. We propose unbiased and consistent estimators for each quantity which
only require generated samples but not the underlying model itself. As an
application, we offer a generalization evaluation of diffusion models and
discover how mode collapse of minority groups is a contrary phenomenon to
overfitting. Further, we demonstrate that variance and predictive kernel
entropy are viable measures of uncertainty for image, audio, and language
generation. Specifically, our approach for uncertainty estimation is more
predictive of performance on CoQA and TriviaQA question answering datasets than
existing baselines and can also be applied to closed-source models.Comment: Preprin
Data Unfolding with Wiener-SVD Method
Data unfolding is a common analysis technique used in HEP data analysis.
Inspired by the deconvolution technique in the digital signal processing, a new
unfolding technique based on the SVD technique and the well-known Wiener filter
is introduced. The Wiener-SVD unfolding approach achieves the unfolding by
maximizing the signal to noise ratios in the effective frequency domain given
expectations of signal and noise and is free from regularization parameter.
Through a couple examples, the pros and cons of the Wiener-SVD approach as well
as the nature of the unfolded results are discussed.Comment: 26 pages, 12 figures, match the accepted version by JINS
- …