Tuning the distribution dependent prior in the PAC-Bayes framework based on empirical data

In this paper we further develop the idea that the PAC-Bayes prior can be defined based on the data-generating distribution. In particular, following Catoni [1], we refine some recent generalisation bounds on the risk of the Gibbs Classifier, when the prior is defined in terms of the data generating distribution, and the posterior is defined in terms of the observed one. Moreover we show that the prior and the posterior distributions can be tuned based on the observed samples without worsening the convergence rate of the bounds and with a marginal impact on their constants

The Digital Kernel Perceptron

In this paper, we show that a kernel-based perceptron can be efficiently implemented in digital hardware using very few components. Despite its simplicity, the experimental results on standard data sets show remarkable performance in terms of generalization error

Quantum-computing optimization for K-Winner Machines

While addressing Vector Quantization (VQ) as a general paradigm for data representation, the paper adopts the K-winner Machine model as a case study, which provides a reference for analyzing both theoretical and implementation aspects. The design of vector quantizers often requires that the (often overlooked) dichotomy between \u2018analogue\u2019 modeling and \u2018digital\u2019 implementation be taken in account. In the case of digital VQ systems, optimal design can bring about NP-hard problems that prove intractable in terms of computational complexity. The paper discusses the possibility of using advanced paradigms such as Quantum Computing for digital optimization processes in order to overcome the limitations of conventional machinery. The presented research provides analytical criteria determining the relative advantages of conventional over quantum-computing approaches

Representation and generalization properties of class-entropy networks

Using conditional class entropy (CCE) as a cost function allows feedforward networks to fully exploit classification-relevant information. CCE-based networks arrange the data space into partitions, which are assigned unambiguous symbols and are labeled by class information. By this labeling mechanism the network can model the empirical data distribution at the local level. Region labeling evolves with the network-training process, which follows a plastic algorithm. The paper proves several theoretical properties about the performance of CCE-based networks, and considers both convergence during training and generalization ability at run-time. In addition, analytical criteria and practical procedures are proposed to enhance the generalization performance of the trained networks. Experiments on artificial and real-world domains confirm the accuracy of this class of networks and witness the validity of the described method

Differential privacy and generalization: Sharper bounds with applications

In this paper we deal with the problem of improving the recent milestone results on the estimation of the generalization capability of a randomized learning algorithm based on Differential Privacy (DP). In particular, we derive new DP based multiplicative Chernoff and Bennett type generalization bounds, which improve over the current state-of-the-art Hoeffding type bound. Then, we prove that a randomized algorithm based on the data generating dependent prior and data dependent posterior Boltzmann distributions of Catoni (2007) [10] is Differentially Private and shows better generalization properties than the Gibbs classifier associated to the same distributions. With this aim, we also exploit a simple example. Finally, we discuss the advantages of using the Thresholdout procedure, one of the main results generated by the DP theory, for Model Selection and Error Estimation purposes, and we derive a new result which exploits our new generalization bounds

PAC-bayesian analysis of distribution dependent priors: Tighter risk bounds and stability analysis

In this paper we bound the risk of the Gibbs and Bayes classifiers (GC and BC), when the prior is defined in terms of the data generating distribution, and the posterior is defined in terms of the observed one, as proposed by Catoni (2007). We deal with this problem from two different perspectives. From one side we briefly review and further develop the classical PAC-Bayes analysis by refining the current state-of-the-art risk bounds. From the other side we propose a novel approach, based on the concept of Algorithmic Stability, which we call Distribution Stability (DS), and develop some new risk bounds over the GC and BC based on the DS. Finally, we show that the data dependent posterior distribution associated to the data generating prior has also attractive and previously unknown properties

Testing the Augmented Binary Multiclass SVM on Microarray Data

In this paper we test a new multicategory SVM method, called Augmented Binary (AB), on microarray gene expression data. The AB SVM is one of the methods generating a multicategory classifier in one step, without dividing the multiclass problem into binary subproblems. This approach can be useful when the number of samples is very low, like in this kind of application. Furthermore, the use of a single SVM, instead of several binary ones, simplifies the search for optimal hyperparameters and allows a consistent output for all the classes