Regularization with Approximated L2L^2 Maximum Entropy Method

We tackle the inverse problem of reconstructing an unknown finite measure $\mu$ from a noisy observation of a generalized moment of $\mu$ defined as the integral of a continuous and bounded operator $\Phi$ with respect to $\mu$. When only a quadratic approximation $\Phi_m$ of the operator is known, we introduce the $L^2$ approximate maximum entropy solution as a minimizer of a convex functional subject to a sequence of convex constraints. Under several assumptions on the convex functional, the convergence of the approximate solution is established and rates of convergence are provided.Comment: 16 page

Takeuchi's Information Criteria as a form of Regularization

Takeuchi's Information Criteria (TIC) is a linearization of maximum likelihood estimator bias which shrinks the model parameters towards the maximum entropy distribution, even when the model is mis-specified. In statistical machine learning, $L_2$ regularization (a.k.a. ridge regression) also introduces a parameterized bias term with the goal of minimizing out-of-sample entropy, but generally requires a numerical solver to find the regularization parameter. This paper presents a novel regularization approach based on TIC; the approach does not assume a data generation process and results in a higher entropy distribution through more efficient sample noise suppression. The resulting objective function can be directly minimized to estimate and select the best model, without the need to select a regularization parameter, as in ridge regression. Numerical results applied to a synthetic high dimensional dataset generated from a logistic regression model demonstrate superior model performance when using the TIC based regularization over a $L_1$ and a $L_2$ penalty term

Fast optimization of Multithreshold Entropy Linear Classifier

Multithreshold Entropy Linear Classifier (MELC) is a density based model which searches for a linear projection maximizing the Cauchy-Schwarz Divergence of dataset kernel density estimation. Despite its good empirical results, one of its drawbacks is the optimization speed. In this paper we analyze how one can speed it up through solving an approximate problem. We analyze two methods, both similar to the approximate solutions of the Kernel Density Estimation querying and provide adaptive schemes for selecting a crucial parameters based on user-specified acceptable error. Furthermore we show how one can exploit well known conjugate gradients and L-BFGS optimizers despite the fact that the original optimization problem should be solved on the sphere. All above methods and modifications are tested on 10 real life datasets from UCI repository to confirm their practical usability.Comment: Presented at Theoretical Foundations of Machine Learning 2015 (http://tfml.gmum.net), final version published in Schedae Informaticae Journa

A realizability-preserving high-order kinetic scheme using WENO reconstruction for entropy-based moment closures of linear kinetic equations in slab geometry

We develop a high-order kinetic scheme for entropy-based moment models of a one-dimensional linear kinetic equation in slab geometry. High-order spatial reconstructions are achieved using the weighted essentially non-oscillatory (WENO) method, and for time integration we use multi-step Runge-Kutta methods which are strong stability preserving and whose stages and steps can be written as convex combinations of forward Euler steps. We show that the moment vectors stay in the realizable set using these time integrators along with a maximum principle-based kinetic-level limiter, which simultaneously dampens spurious oscillations in the numerical solutions. We present numerical results both on a manufactured solution, where we perform convergence tests showing our scheme converges of the expected order up to the numerical noise from the numerical optimization, as well as on two standard benchmark problems, where we show some of the advantages of high-order solutions and the role of the key parameter in the limiter