79,693 research outputs found
Empirical Risk Minimization with Approximations of Probabilistic Grammars
Probabilistic grammars are generative statistical models that are useful for compositional and sequential structures. We present a framework, reminiscent of structural risk minimization, for empirical risk minimization of the parameters of a fixed probabilistic grammar using the log-loss. We derive sample complexity bounds in this framework that apply both to the supervised setting and the unsupervised setting.
Empirical Risk Minimization for Probabilistic Grammars: Sample Complexity and Hardness of Learning
Probabilistic grammars are generative statistical models that are useful for compositional and sequential structures. They are used ubiquitously in computational linguistics. We present a framework, reminiscent of structural risk minimization, for empirical risk minimization of probabilistic grammars using the log-loss. We derive sample complexity bounds in this framework that apply both to the supervised setting and the unsupervised setting. By making assumptions about the underlying distribution that are appropriate for natural language scenarios, we are able to derive distribution-dependent sample complexity bounds for probabilistic grammars. We also give simple algorithms for carrying out empirical risk minimization using this framework in both the supervised and unsupervised settings. In the unsupervised case, we show that the problem of minimizing empirical risk is NP-hard. We therefore suggest an approximate algorithm, similar to expectation-maximization, to minimize the empirical risk. Learning from data is central to contemporary computational linguistics. It is in common in such learning to estimate a model in a parametric family using the maximum likelihood principle. This principle applies in the supervised case (i.e., using annotate
Structural Risk Minimization for Learning Nonlinear Dynamics
Recent advances in learning or identification of nonlinear dynamics focus on
learning a suitable model within a pre-specified model class. However, a key
difficulty that remains is the choice of the model class from which the
dynamics will be learned. The fundamental challenge is trading the richness of
the model class with the learnability within the model class. Toward addressing
the so-called model selection problem, we introduce a novel notion of
Structural Risk Minimization (SRM) for learning nonlinear dynamics. Inspired by
classical SRM for classification, we minimize a bound on the true prediction
error over hierarchies of model classes. The class selected by our SRM scheme
is shown to achieve a nearly optimal learning guarantee among all model classes
contained in the hierarchy. Employing the proposed scheme along with computable
model class complexity bounds, we derive explicit SRM schemes for learning
nonlinear dynamics under hierarchies of: i) norm-constrained Reproducing Kernel
Hilbert Spaces, and ii) norm-constrained Neural Network classes. We empirically
show that even though too loose to be used as absolute estimates, our SRM
bounds on the true prediction error are able to track its relative behavior
across different model classes of the hierarchy
Support Vector Machines in High Energy Physics
This lecture will introduce the Support Vector algorithms for classification
and regression. They are an application of the so called kernel trick, which
allows the extension of a certain class of linear algorithms to the non linear
case. The kernel trick will be introduced and in the context of structural risk
minimization, large margin algorithms for classification and regression will be
presented. Current applications in high energy physics will be discussed.Comment: 11 pages, 12 figures. Part of the proceedings of the Track
'Computational Intelligence for HEP Data Analysis' at iCSC 200
Structural Minimization of Risk on Estimation of Heterogeneity Distributions
Population heterogeneity dynamics is one of the research directions in IIASA's Population Program. One typical and practical problem related to hidden heterogeneity is the estimation of the heterogeneity distribution.
This paper describes the approach to such an estimation which is based on the method of structural minimization of mean risk. It is shown how this method can be implemented to some real data. The main ideas of the method are also described
- …