Parametric Polynomial Time Perceptron Rescaling Algorithm

Let us consider a linear feasibility problem with a possibly infinite number of inequality constraints posed in an on-line setting: an algorithm suggests a candidate solution, and the oracle either confirms its feasibility, or outputs a violated constraint vector. This model can be solved by subgradient optimisation algorithms for non-smooth functions, also known as the perceptron algorithms in the machine learning community, and its solvability depends on the problem dimension and the radius of the constraint set. The classical perceptron algorithm may have an exponential complexity in the worst case when the radius is infinitesimal [1]. To overcome this difficulty, the space dilation technique was exploited in the ellipsoid algorithm to make its running time polynomial [3]. A special case of the space dilation, the rescaling procedure is utilised in the perceptron rescaling algorithm [2] with a probabilistic approach to choosing the direction of dilation. A parametric version of the perceptron rescaling algorithm is the focus of this work. It is demonstrated that some fixed parameters of the latter algorithm (the initial estimate of the radius and the relaxation parameter) may be modified and adapted for particular problems. The generalised theoretical framework allows to determine convergence of the algorithm with any chosen set of values of these parameters, and suggests a potential way of decreasing the complexity of the algorithm which remains the subject of current research

Entropy landscape of solutions in the binary perceptron problem

The statistical picture of the solution space for a binary perceptron is studied. The binary perceptron learns a random classification of input random patterns by a set of binary synaptic weights. The learning of this network is difficult especially when the pattern (constraint) density is close to the capacity, which is supposed to be intimately related to the structure of the solution space. The geometrical organization is elucidated by the entropy landscape from a reference configuration and of solution-pairs separated by a given Hamming distance in the solution space. We evaluate the entropy at the annealed level as well as replica symmetric level and the mean field result is confirmed by the numerical simulations on single instances using the proposed message passing algorithms. From the first landscape (a random configuration as a reference), we see clearly how the solution space shrinks as more constraints are added. From the second landscape of solution-pairs, we deduce the coexistence of clustering and freezing in the solution space.Comment: 21 pages, 6 figures, version accepted by Journal of Physics A: Mathematical and Theoretica

The impact of architecture on the performance of artificial neural networks

A number of researchers have investigated the impact of network architecture on the performance of artificial neural networks. Particular attention has been paid to the impact on the performance of the multi-layer perceptron of architectural issues, and the use of various strategies to attain an optimal network structure. However, there are still perceived limitations with the multi-layer perceptron and networks that employ a different architecture to the multi-layer perceptron have gained in popularity in recent years, particularly, networks that implement a more localised solution, where the solution in one area of the problem space does not impact, or has a minimal impact, on other areas of the space. In this study, we discuss the major architectural issues affecting the performance of a multi-layer perceptron, before moving on to examine in detail the performance of a new localised network, namely the bumptree. The work presented here examines the impact on the performance of artificial neural networks of employing alternative networks to the long established multi-layer perceptron. In particular, networks that impose a solution where the impact of each parameter in the final network architecture has a localised impact on the problem space being modelled are examined. The alternatives examined are the radial basis function and bumptree neural networks, and the impact of architectural issues on the performance of these networks is examined. Particular attention is paid to the bumptree, with new techniques for both developing the bumptree structure and employing this structure to classify patterns being examined

Constructive Preference Elicitation over Hybrid Combinatorial Spaces

Preference elicitation is the task of suggesting a highly preferred configuration to a decision maker. The preferences are typically learned by querying the user for choice feedback over pairs or sets of objects. In its constructive variant, new objects are synthesized "from scratch" by maximizing an estimate of the user utility over a combinatorial (possibly infinite) space of candidates. In the constructive setting, most existing elicitation techniques fail because they rely on exhaustive enumeration of the candidates. A previous solution explicitly designed for constructive tasks comes with no formal performance guarantees, and can be very expensive in (or unapplicable to) problems with non-Boolean attributes. We propose the Choice Perceptron, a Perceptron-like algorithm for learning user preferences from set-wise choice feedback over constructive domains and hybrid Boolean-numeric feature spaces. We provide a theoretical analysis on the attained regret that holds for a large class of query selection strategies, and devise a heuristic strategy that aims at optimizing the regret in practice. Finally, we demonstrate its effectiveness by empirical evaluation against existing competitors on constructive scenarios of increasing complexity.Comment: AAAI 2018, computing methodologies, machine learning, learning paradigms, supervised learning, structured output

Playing Billiard in Version Space

A ray-tracing method inspired by ergodic billiards is used to estimate the theoretically best decision rule for a set of linear separable examples. While the Bayes-optimum requires a majority decision over all Perceptrons separating the example set, the problem considered here corresponds to finding the single Perceptron with best average generalization probability. For randomly distributed examples the billiard estimate agrees with known analytic results. In real-life classification problems the generalization error is consistently reduced compared to the maximal stability Perceptron.Comment: uuencoded, gzipped PostScript file, 127576 bytes To recover 1) save file as bayes.uue. Then 2) uudecode bayes.uue and 3) gunzip bayes.ps.g

Multifractality and percolation in the coupling space of perceptrons

The coupling space of perceptrons with continuous as well as with binary weights gets partitioned into a disordered multifractal by a set of $p=\gamma N$ random input patterns. The multifractal spectrum $f(\alpha)$ can be calculated analytically using the replica formalism. The storage capacity and the generalization behaviour of the perceptron are shown to be related to properties of $f(\alpha)$ which are correctly described within the replica symmetric ansatz. Replica symmetry breaking is interpreted geometrically as a transition from percolating to non-percolating cells. The existence of empty cells gives rise to singularities in the multifractal spectrum. The analytical results for binary couplings are corroborated by numerical studies.Comment: 13 pages, revtex, 4 eps figures, version accepted for publication in Phys. Rev.

Stability of the replica symmetric solution in diluted perceptron learning

We study the role played by the dilution in the average behavior of a perceptron model with continuous coupling with the replica method. We analyze the stability of the replica symmetric solution as a function of the dilution field for the generalization and memorization problems. Thanks to a Gardner like stability analysis we show that at any fixed ratio $\alpha$ between the number of patterns M and the dimension N of the perceptron ($\alpha=M/N$), there exists a critical dilution field $h_c$ above which the replica symmetric ansatz becomes unstable.Comment: Stability of the solution in arXiv:0907.3241, 13 pages, (some typos corrected