The Vapnik-Chervonenkis Dimension: Information versus Complexity in Learning

When feasible, learning is a very attractive alternative to explicit programming. This is particularly true in areas where the problems do not lend themselves to systematic programming, such as pattern recognition in natural environments. The feasibility of learning an unknown function from examples depends on two questions: 1. Do the examples convey enough information to determine the function? 2. Is there a speedy way of constructing the function from the examples? These questions contrast the roles of information and complexity in learning. While the two roles share some ground, they are conceptually and technically different. In the common language of learning, the information question is that of generalization and the complexity question is that of scaling. The work of Vapnik and Chervonenkis (1971) provides the key tools for dealing with the information issue. In this review, we develop the main ideas of this framework and discuss how complexity fits in

Hints

The systematic use of hints in the learning-from-examples paradigm is the subject of this review. Hints are the properties of the target function that are known to us independently of the training examples. The use of hints is tantamount to combining rules and data in learning, and is compatible with different learning models, optimization techniques, and regularization techniques. The hints are represented to the learning process by virtual examples, and the training examples of the target function are treated on equal footing with the rest of the hints. A balance is achieved between the information provided by the different hints through the choice of objective functions and learning schedules. The Adaptive Minimization algorithm achieves this balance by relating the performance on each hint to the overall performance. The application of hints in forecasting the very noisy foreign-exchange markets is illustrated. On the theoretical side, the information value of hints is contrasted to the complexity value and related to the VC dimension

An algorithm for learning from hints

To take advantage of prior knowledge (hints) about the function one wants to learn, we introduce a method that generalizes learning from examples to learning from hints. A canonical representation of hints is defined and illustrated. All hints are represented to the learning process by examples, and examples of the function are treated on equal footing with the rest of the hints. During learning, examples from different hints are selected for processing according to a given schedule. We present two types of schedules; fixed schedules that specify the relative emphasis of each hint, and adaptive schedules that are based on how well each hint has been learned so far. Our learning method is compatible with any descent technique

Financial model calibration using consistency hints

We introduce a technique for forcing the calibration of a financial model to produce valid parameters. The technique is based on learning from hints. It converts simple curve fitting into genuine calibration, where broad conclusions can be inferred from parameter values. The technique augments the error function of curve fitting with consistency hint error functions based on the Kullback-Leibler distance. We introduce an efficient EM-type optimization algorithm tailored to this technique. We also introduce other consistency hints, and balance their weights using canonical errors. We calibrate the correlated multifactor Vasicek model of interest rates, and apply it successfully to Japanese Yen swaps market and US dollar yield market

Effect of Liquid Droplets on Turbulence Structure in a Round Gaseous Jet

A second-order model which predicts the modulation of turbulence in jets laden with uniform size solid particles or liquid droplets is discussed. The approach followed is to start from the separate momentum and continuity equations of each phase and derive two new conservation equations. The first is for the carrier fluid's kinetic energy of turbulence and the second for the dissipation rate of that energy. Closure of the set of transport equations is achieved by modeling the turbulence correlations up to a third order. The coefficients (or constants) appearing in the modeled equations are then evaluated by comparing the predictions with LDA-measurements obtained recently in a turbulent jet laden with 200 microns solid particles. This set of constants is then used to predict the same jet flow but laden with 50 microns solid particles. The agreement with the measurement in this case is very good

The capacity of multilevel threshold functions

Lower and upper bounds for the capacity of multilevel threshold elements are estimated, using two essentially different enumeration techniques. It is demonstrated that the exact number of multilevel threshold functions depends strongly on the relative topology of the input set. The results correct a previously published estimate and indicate that adding threshold levels enhances the capacity more than adding variables

Pruning training sets for learning of object categories

Training datasets for learning of object categories are often contaminated or imperfect. We explore an approach to automatically identify examples that are noisy or troublesome for learning and exclude them from the training set. The problem is relevant to learning in semi-supervised or unsupervised setting, as well as to learning when the training data is contaminated with wrongly labeled examples or when correctly labeled, but hard to learn examples, are present. We propose a fully automatic mechanism for noise cleaning, called ’data pruning’, and demonstrate its success on learning of human faces. It is not assumed that the data or the noise can be modeled or that additional training examples are available. Our experiments show that data pruning can improve on generalization performance for algorithms with various robustness to noise. It outperforms methods with regularization properties and is superior to commonly applied aggregation methods, such as bagging

Hints and the VC Dimension

Learning from hints is a generalization of learning from examples that allows for a variety of information about the unknown function to be used in the learning process. In this paper, we use the VC dimension, an established tool for analyzing learning from examples, to analyze learning from hints. In particular, we show how the VC dimension is affected by the introduction of a hint. We also derive a new quantity that defines a VC dimension for the hint itself. This quantity is used to estimate the number of examples needed to "absorb" the hint. We carry out the analysis for two types of hints, invariances and catalysts. We also describe how the same method can be applied to other types of hints

Effect of liquid droplets on turbulence in a round gaseous jet

The main objective of this investigation is to develop a two-equation turbulence model for dilute vaporizing sprays or in general for dispersed two-phase flows including the effects of phase changes. The model that accounts for the interaction between the two phases is based on rigorously derived equations for turbulence kinetic energy (K) and its dissipation rate epsilon of the carrier phase using the momentum equation of that phase. Closure is achieved by modeling the turbulent correlations, up to third order, in the equations of the mean motion, concentration of the vapor in the carrier phase, and the kinetic energy of turbulence and its dissipation rate for the carrier phase. The governing equations are presented in both the exact and the modeled formes. The governing equations are solved numerically using a finite-difference procedure to test the presented model for the flow of a turbulent axisymmetric gaseous jet laden with either evaporating liquid droplets or solid particles. The predictions include the distribution of the mean velocity, volume fractions of the different phases, concentration of the evaporated material in the carrier phase, turbulence intensity and shear stress of the carrier phase, droplet diameter distribution, and the jet spreading rate. The predictions are in good agreement with the experimental data