A Moment-Matching Approach to Testable Learning and a New Characterization of Rademacher Complexity

A remarkable recent paper by Rubinfeld and Vasilyan (2022) initiated the study of \emph{testable learning}, where the goal is to replace hard-to-verify distributional assumptions (such as Gaussianity) with efficiently testable ones and to require that the learner succeed whenever the unknown distribution passes the corresponding test. In this model, they gave an efficient algorithm for learning halfspaces under testable assumptions that are provably satisfied by Gaussians. In this paper we give a powerful new approach for developing algorithms for testable learning using tools from moment matching and metric distances in probability. We obtain efficient testable learners for any concept class that admits low-degree \emph{sandwiching polynomials}, capturing most important examples for which we have ordinary agnostic learners. We recover the results of Rubinfeld and Vasilyan as a corollary of our techniques while achieving improved, near-optimal sample complexity bounds for a broad range of concept classes and distributions. Surprisingly, we show that the information-theoretic sample complexity of testable learning is tightly characterized by the Rademacher complexity of the concept class, one of the most well-studied measures in statistical learning theory. In particular, uniform convergence is necessary and sufficient for testable learning. This leads to a fundamental separation from (ordinary) distribution-specific agnostic learning, where uniform convergence is sufficient but not necessary.Comment: 34 page

Multiclass Learnability Beyond the PAC Framework: Universal Rates and Partial Concept Classes

In this paper we study the problem of multiclass classification with a bounded number of different labels $k$, in the realizable setting. We extend the traditional PAC model to a) distribution-dependent learning rates, and b) learning rates under data-dependent assumptions. First, we consider the universal learning setting (Bousquet, Hanneke, Moran, van Handel and Yehudayoff, STOC '21), for which we provide a complete characterization of the achievable learning rates that holds for every fixed distribution. In particular, we show the following trichotomy: for any concept class, the optimal learning rate is either exponential, linear or arbitrarily slow. Additionally, we provide complexity measures of the underlying hypothesis class that characterize when these rates occur. Second, we consider the problem of multiclass classification with structured data (such as data lying on a low dimensional manifold or satisfying margin conditions), a setting which is captured by partial concept classes (Alon, Hanneke, Holzman and Moran, FOCS '21). Partial concepts are functions that can be undefined in certain parts of the input space. We extend the traditional PAC learnability of total concept classes to partial concept classes in the multiclass setting and investigate differences between partial and total concepts

Exploring the potential of using undergraduates’ knowledge, skills and experience in research methods as a proxy for capturing learning gain

Learning gain is a politicised concept within contemporary HE, and as such has been aligned with agendas of teaching excellence and learning outcomes but the extent to which it captures actual learning has yet to be clarified. Here, we report the outcomes of a learning gain study which examines how students’ knowledge, skills and experiences as researchers develops throughout their studies. We examine data from a self-reporting survey administered across a university and college-based HE providers during students’ second year of undergraduate study. The data highlight disciplinary differences in student engagement with research methods and the significance of perceived relevance of research methods to students’ learning. These findings do have a bearing on the development of measures of learning gain as they are demonstrating the complexity of capturing student learning across disciplines. Our findings can be employed to develop a method of capturing learning gain that can be integrated into undergraduates’ research methods education

Algorithmic cache of sorted tables for feature selection: Speeding up methods based on consistency and information theory measures

Feature selection is a mechanism used in Machine Learning to re-duce the complexity and improve the speed of the learning process by usinga subset of features from the data set. There are several measures which areused to assign a score to a subset of features and, therefore, are able to com-pare them and decide which one is the best. The bottle neck of consistencemeasures is having the information of the different examples available to checktheir class by groups. To handle it, this paper proposes the concept of an al-gorithmic cache, which stores sorted tables to speed up the access to exampleinformation. The work carries out an empirical study using 34 real-world datasets and four representative search strategies combined with different tablecaching strategies and three sorting methods. The experiments calculate fourdifferent consistency and one information measures, showing that the proposedsorted tables cache reduces computation time and it is competitive with hashtable structures

Assessment-driven Learning through Serious Games: Guidance and Effective Outcomes

Evaluation in serious games is an important aspect; it aims to evaluate the good transmission of pedagogical objectives, the performance of student in relation to these objectives defined in the pedagogical scenario, the content of the course and the predefined criteria. However, the effectiveness of learning is under-studied due to the complexity involved to gamify the assessment concept, particularly when it comes to intangible measures related to the progression of learning outcomes, which is among the most important aspects of evaluation in serious games. This paper reviews the literature regarding assessment due to their importance in the learning process with a detailed assessment plan applied on serious game. Then, it presents a framework used to facilitate the assessment design integrated in serious games. Finally, a significant example of how the proposed framework proved successful with corresponding results will conclude the paper