A Realizable Learning Task which Exhibits Overfitting

In this paper we examine a perceptron learning task. The task is realizable since it is provided by another perceptron with identical architecture. Both perceptrons have nonlinear sigmoid output functions. The gain of the output function determines the level of nonlinearity of the learning task. It is observed that a high level of nonlinearity leads to overfitting. We give an explanation for this rather surprising observation and develop a method to avoid the overfitting. This method has two possible interpretations, one is learning with noise, the other cross--validated early stopping. 1 Learning Rules from Examples The property which makes feedforward neural nets interesting for many practical applications is their ability to approximate functions, which are given only by examples. Feed-forward networks with at least one hidden layer of nonlinear units are able to approximate each continuous function on a N-dimensional hypercube arbitrarily well. While the existence of neural functi..

Optimization under uncertainty with application to data clustering

A new optimization technique with uncertainty that extends the pure nested partition (NP) algorithm is presented in this thesis. This method is called the nested partition with inheritance. The basic idea of a NP algorithm is very simple. At each iteration, the most promising region is partitioned and the performance of the partitioned region is evaluated using sampling. Based on the performance evaluation, the most promising region is chosen for the next iteration. These procedures are repeated until it satisfies the termination condition.;Even though the pure NP method guarantees the convergence to the optimal solution, it has several shortcomings. To handle these shortcomings, two extensions to the pure NP are suggested. To rigorously determine the required sample effort, some statistical selection methods are implemented, which include the Nelson Matejcik procedure, the Rinott procedure, and the Dudewicz and Dalal procedure, as well as a subset procedure. In addition, Genetic Algorithms (GAs) are used to speed convergence and to overcome the difficulty in the backtracking stage of the NP algorithm.;As an application of the new methodology, this work also suggests the methods to be applied to a data clustering problem. This is a very hard problem with two of the main difficulties being lack of scalability with respect to amount of data and problems with high dimensionality. The new algorithms are found to be effective for solving this problem. Random sampling enhances scalability and the iterative partitioning addresses the dimensionality