138,388 research outputs found
A Short Introduction to Model Selection, Kolmogorov Complexity and Minimum Description Length (MDL)
The concept of overfitting in model selection is explained and demonstrated
with an example. After providing some background information on information
theory and Kolmogorov complexity, we provide a short explanation of Minimum
Description Length and error minimization. We conclude with a discussion of the
typical features of overfitting in model selection.Comment: 20 pages, Chapter 1 of The Paradox of Overfitting, Master's thesis,
Rijksuniversiteit Groningen, 200
Quantifying Overfitting: Introducing the Overfitting Index
In the rapidly evolving domain of machine learning, ensuring model
generalizability remains a quintessential challenge. Overfitting, where a model
exhibits superior performance on training data but falters on unseen data, is a
recurrent concern. This paper introduces the Overfitting Index (OI), a novel
metric devised to quantitatively assess a model's tendency to overfit. Through
extensive experiments on the Breast Ultrasound Images Dataset (BUS) and the
MNIST dataset using architectures such as MobileNet, U-Net, ResNet, Darknet,
and ViT-32, we illustrate the utility and discernment of the OI. Our results
underscore the variable overfitting behaviors across architectures and
highlight the mitigative impact of data augmentation, especially on smaller and
more specialized datasets. The ViT-32's performance on MNIST further emphasizes
the robustness of certain models and the dataset's comprehensive nature. By
providing an objective lens to gauge overfitting, the OI offers a promising
avenue to advance model optimization and ensure real-world efficacy
Replica analysis of overfitting in regression models for time-to-event data
Overfitting, which happens when the number of parameters in a model is too
large compared to the number of data points available for determining these
parameters, is a serious and growing problem in survival analysis. While modern
medicine presents us with data of unprecedented dimensionality, these data
cannot yet be used effectively for clinical outcome prediction. Standard error
measures in maximum likelihood regression, such as p-values and z-scores, are
blind to overfitting, and even for Cox's proportional hazards model (the main
tool of medical statisticians), one finds in literature only rules of thumb on
the number of samples required to avoid overfitting. In this paper we present a
mathematical theory of overfitting in regression models for time-to-event data,
which aims to increase our quantitative understanding of the problem and
provide practical tools with which to correct regression outcomes for the
impact of overfitting. It is based on the replica method, a statistical
mechanical technique for the analysis of heterogeneous many-variable systems
that has been used successfully for several decades in physics, biology, and
computer science, but not yet in medical statistics. We develop the theory
initially for arbitrary regression models for time-to-event data, and verify
its predictions in detail for the popular Cox model.Comment: 37 pages, 9 figure
- …