Mind change efficient learning

This paper studies efficient learning with respect to mind changes. Our starting point is the idea that a learner that is efficient with respect to mind changes minimizes mind changes not only globally in the entire learning problem, but also locally in subproblems after receiving some evidence. Formalizing this idea leads to the notion of uniform mind change optimality. We characterize the structure of language classes that can be identified with at most α mind changes by some learner (not necessarily effective): A language class L is identifiable with α mind changes iff the accumulation order of L is at most α. Accumulation order is a classic concept from point-set topology. To aid the construction of learning algorithms, we show that the characteristic property of uniformly mind change optimal learners is that they output conjectures (languages) with maximal accumulation order. We illustrate the theory by describing mind change optimal learners for various problems such as identifying linear subspaces and one-variable patterns

Indefinitely Oscillating Martingales

We construct a class of nonnegative martingale processes that oscillate indefinitely with high probability. For these processes, we state a uniform rate of the number of oscillations and show that this rate is asymptotically close to the theoretical upper bound. These bounds on probability and expectation of the number of upcrossings are compared to classical bounds from the martingale literature. We discuss two applications. First, our results imply that the limit of the minimum description length operator may not exist. Second, we give bounds on how often one can change one's belief in a given hypothesis when observing a stream of data.Comment: ALT 2014, extended technical repor

Levels of discontinuity, limit-computability, and jump operators

We develop a general theory of jump operators, which is intended to provide an abstraction of the notion of "limit-computability" on represented spaces. Jump operators also provide a framework with a strong categorical flavor for investigating degrees of discontinuity of functions and hierarchies of sets on represented spaces. We will provide a thorough investigation within this framework of a hierarchy of $\Delta^0_2$-measurable functions between arbitrary countably based $T_0$-spaces, which captures the notion of computing with ordinal mind-change bounds. Our abstract approach not only raises new questions but also sheds new light on previous results. For example, we introduce a notion of "higher order" descriptive set theoretical objects, we generalize a recent characterization of the computability theoretic notion of "lowness" in terms of adjoint functors, and we show that our framework encompasses ordinal quantifications of the non-constructiveness of Hilbert's finite basis theorem

Quasi-Polish Spaces

We investigate some basic descriptive set theory for countably based completely quasi-metrizable topological spaces, which we refer to as quasi-Polish spaces. These spaces naturally generalize much of the classical descriptive set theory of Polish spaces to the non-Hausdorff setting. We show that a subspace of a quasi-Polish space is quasi-Polish if and only if it is level \Pi_2 in the Borel hierarchy. Quasi-Polish spaces can be characterized within the framework of Type-2 Theory of Effectivity as precisely the countably based spaces that have an admissible representation with a Polish domain. They can also be characterized domain theoretically as precisely the spaces that are homeomorphic to the subspace of all non-compact elements of an \omega-continuous domain. Every countably based locally compact sober space is quasi-Polish, hence every \omega-continuous domain is quasi-Polish. A metrizable space is quasi-Polish if and only if it is Polish. We show that the Borel hierarchy on an uncountable quasi-Polish space does not collapse, and that the Hausdorff-Kuratowski theorem generalizes to all quasi-Polish spaces

Indefinitely oscillating martingales

We construct a class of nonnegative martingale processes that oscillate indefinitely with high probability. For these processes, we state a uniform rate of the number of oscillations for a given magnitude and show that this rate is asymptotically close to the theoretical upper bound. These bounds on probability and expectation of the number of upcrossings are compared to classical bounds from the martingale literature. We discuss two applications. First, our results imply that the limit of the minimum description length operator may not exist. Second, we give bounds on how often one can change one’s belief in a given hypothesis when observing a stream of data

Set systems: order types, continuous nondeterministic deformations, and quasi-orders

By reformulating a learning process of a set system L as a game between Teacher and Learner, we define the order type of L to be the order type of the game tree, if the tree is well-founded. The features of the order type of L (dim L in symbol) are (1) We can represent any well-quasi-order (wqo for short) by the set system L of the upper-closed sets of the wqo such that the maximal order type of the wqo is equal to dim L. (2) dim L is an upper bound of the mind-change complexity of L. dim L is defined iff L has a finite elasticity (fe for short), where, according to computational learning theory, if an indexed family of recursive languages has fe then it is learnable by an algorithm from positive data. Regarding set systems as subspaces of Cantor spaces, we prove that fe of set systems is preserved by any continuous function which is monotone with respect to the set-inclusion. By it, we prove that finite elasticity is preserved by various (nondeterministic) language operators (Kleene-closure, shuffle-closure, union, product, intersection,. . ..) The monotone continuous functions represent nondeterministic computations. If a monotone continuous function has a computation tree with each node followed by at most n immediate successors and the order type of a set system L is {\alpha}, then the direct image of L is a set system of order type at most n-adic diagonal Ramsey number of {\alpha}. Furthermore, we provide an order-type-preserving contravariant embedding from the category of quasi-orders and finitely branching simulations between them, into the complete category of subspaces of Cantor spaces and monotone continuous functions having Girard's linearity between them. Keyword: finite elasticity, shuffle-closur

Many Problems, Different Frameworks: Classification of Problems in Computable Analysis and Algorithmic Learning Theory

In this thesis, we study the complexity of some mathematical problems, in particular those arising in \emph{computable analysis} and \emph{algorithmic learning theory for algebraic structures}. We highlight that our study is not limited to these two areas: indeed, in both cases, the results we obtain are tightly connected to ideas and tools coming from different areas of mathematical logic, including for example descriptive set theory and reverse mathematics. After giving the necessary preliminaries, the rest of the thesis is divided into two parts one concerning computable analysis and the other algorithmic learning theory for algebraic structures. In the first part we start studying the uniform computational strength of the Cantor-Bendixson theorem in the Weihrauch lattice. This work falls into the program connecting reverse mathematics and computable analysis via the framework of Weihrauch reducibility. We concentrate on problems related to perfect subsets of Polish spaces, studying the perfect set theorem, the Cantor-Bendixson theorem, and various problems arising from them. In the framework of reverse mathematics, these theorems are equivalent respectively to $\mathsf{ATR}_0$ and \PiCA and, as far as we know, this is the first systematic study of problems at the level of \PiCA in the Weihrauch lattice. We show that the strength of some of the problems we study depends on the topological properties of the Polish space under consideration, while others have the same strength once the space is rich enough. The first part continues considering problems related to (induced) subgraphs. We provide results on the (effective) Wadge complexity of sets of graphs, that are also used to determine the Weihrauch degree of certain decision problems. The decision problems we consider are defined for a fixed graph $G$, and they take as input a graph $H$, answering whether $G$ is an (induced) subgraph of $H$: we also consider the opposite problem (i.e.\ answering whether $H$ is an induced subgraph of $G$). Our study in this context is not limited to decision problems, and we also study the Weihrauch degree of problems that, for a fixed graph $G$ and given in input a graph $H$ such that $G$ is an (induced) subgraph $H$, they output a copy of $G$ in $H$. In both cases, we highlight differences and analogies between the subgraph and the induced subgraph relation. In the second part, we introduce algorithmic learning theory, and we present the framework we use to study the learnability of families of algebraic structures: here, given a countable family of pairwise nonisomorphic structures \K, a learner receives larger and larger pieces of an arbitrary copy of a structure in \K and, at each stage, is required to output a conjecture about the isomorphism type of such a structure. We say that \K is learnable if there exists a learner which eventually stabilizes to a correct guess. The framework was lacking a method for comparing the complexity of nonlearnable families, and so we propose a solution to this problem using tools coming from invariant descriptive set theory. To do so, we first prove that a family of structures is learnable if and only if its learning domain is continuously reducible to the relation $E_0$ of eventual agreement on infinite binary sequences and then, replacing $E_0$ with Borel equivalence relations of higher complexity, we obtain a new hierarchy of learning problems. This leads to the notion of \emph{$E$-learnability}, where a family of structures \K is {$E$-learnable}, for a Borel equivalence relation $E$, if there is a continuous reduction from the isomorphism relation associated with \K to $E$. It is then natural to ask how the notion of $E$-learnability interacts with "classical" learning paradigms. We conclude the second part (and the overall thesis) studying the number of mind changes that a learner needs to learn a given family, both from a topological and a combinatorial point of view

Mind change efficient learning

This paper studies efficient learning with respect to mind changes. Our starting point is the idea that a learner that is efficient with respect to mind changes minimizes mind changes not only globally in the entire learning problem, but also locally in subproblems after receiving some evidence. Formalizing this idea leads to the notion of uniform mind change optimality. We characterize the structure of language classes that can be identified with at most α mind changes by some learner (not necessarily effective): A language class L is identifiable with α mind changes iff the accumulation order of L is at most α. Accumulation order is a classic concept from point-set topology. To aid the construction of learning algorithms, we show that the characteristic property of uniformly mind change optimal learners is that they output conjectures (languages) with maximal accumulation order. We illustrate the theory by describing mind change optimal learners for various problems such as identifying linear subspaces and one-variable patterns