The complexity of algorithmic hypothesis class

University of Technology Sydney. Faculty of Engineering and Information Technology.Statistical learning theory provides the mathematical and theoretical foundations for statistical learning algorithms and inspires the development of more efficient methods. It is observed that learning algorithms may not output some hypotheses in the predefined hypothesis class. Therefore, in this thesis, we focus on statistical learning theory and study how to measure the complexity of the algorithmic hypothesis class, which is a subset of the predefined hypothesis class that a learning algorithm will (or is likely to) output. By designing complexity measures for the algorithmic hypothesis class, we provide new generalization bounds for k-dimensional coding schemes and multi-task learning and propose two frameworks to derive tighter generalization bounds than the current state-of-the-art. We take k-dimensional coding schemes, a set of unsupervised learning algorithms, and multi-task learning, a set of supervised learning algorithms, as examples to demonstrate that learning algorithm outputs may have special properties and are therefore included in a subset of the predefined hypothesis class. By analyzing the subsets (or the algorithmic hypothesis classes), we shed new light on learning problems and derive tighter generalization bounds than the current state-of-the-art. Specifically, for k-dimensional coding schemes, we show that the induced algorithmic loss function classes are sets of Lipschitz-continuous hypotheses and that a dimensionality-dependent complexity measure helps to derive small Lipschitz constants and thus improve the generalization bounds. For multi-task learning, we prove that tasks can act as regularizer and that feature structures can contribute to a small algorithmic hypothesis class and also help to improve the generalization bounds. To more precisely exploit algorithmic hypothesis class complexity by considering the hypothesis and feature structure properties, we extend algorithmic robustness and stability to complexity measures for the hypothesis class. Inspired by the idea of algorithmic robustness, we propose the complexity measure of uniform robustness. Compared to the Rademacher complexity, our measure more finely considers the geometric information of data. For example, when the sample space is covered by a small number of small radius and widely separated balls, the uniform robustness can be very small while the Rademacher complexity can be very large. Moreover, based on the definition of uniform robustness, we also provide a framework to derive generalization bounds for a very general class of learning algorithms. We exploit the algorithmic hypothesis class of stable algorithms by studying the definition of algorithmic stability. Stable learning algorithms have the property that their outputs will not change much when one training example is changed. This implies that their outputs will not be sufficiently far apart, even though the training sample is completely altered. Thus, stable learning algorithms often have small algorithmic hypothesis classes. However, since measuring the complexity of the small algorithmic hypothesis class is unknown, we design a novel complexity measure called the algorithmic Rademacher complexity to measure the algorithmic hypothesis class of stable learning algorithms and provide sharper error bounds than the current state-of-the-art

Implementable Quantum Classifier for Nonlinear Data

In this Letter, we propose a quantum machine learning scheme for the classification of classical nonlinear data. The main ingredients of our method are variational quantum perceptron (VQP) and a quantum generalization of classical ensemble learning. Our VQP employs parameterized quantum circuits to learn a Grover search (or amplitude amplification) operation with classical optimization, and can achieve quadratic speedup in query complexity compared to its classical counterparts. We show how the trained VQP can be used to predict future data with $O(1)$ {query} complexity. Ultimately, a stronger nonlinear classifier can be established, the so-called quantum ensemble learning (QEL), by combining a set of weak VQPs produced using a subsampling method. The subsampling method has two significant advantages. First, all $T$ weak VQPs employed in QEL can be trained in parallel, therefore, the query complexity of QEL is equal to that of each weak VQP multiplied by $T$. Second, it dramatically reduce the {runtime} complexity of encoding circuits that map classical data to a quantum state because this dataset can be significantly smaller than the original dataset given to QEL. This arguably provides a most satisfactory solution to one of the most criticized issues in quantum machine learning proposals. To conclude, we perform two numerical experiments for our VQP and QEL, implemented by Python and pyQuil library. Our experiments show that excellent performance can be achieved using a very small quantum circuit size that is implementable under current quantum hardware development. Specifically, given a nonlinear synthetic dataset with $4$ features for each example, the trained QEL can classify the test examples that are sampled away from the decision boundaries using $146$ single and two qubits quantum gates with $92\%$ accuracy.Comment: 9 page