Takeuchi's Information Criteria (TIC) is a linearization of maximum
likelihood estimator bias which shrinks the model parameters towards the
maximum entropy distribution, even when the model is mis-specified. In
statistical machine learning, L2​ regularization (a.k.a. ridge regression)
also introduces a parameterized bias term with the goal of minimizing
out-of-sample entropy, but generally requires a numerical solver to find the
regularization parameter. This paper presents a novel regularization approach
based on TIC; the approach does not assume a data generation process and
results in a higher entropy distribution through more efficient sample noise
suppression. The resulting objective function can be directly minimized to
estimate and select the best model, without the need to select a regularization
parameter, as in ridge regression. Numerical results applied to a synthetic
high dimensional dataset generated from a logistic regression model demonstrate
superior model performance when using the TIC based regularization over a L1​
and a L2​ penalty term