Comparing Sample-wise Learnability Across Deep Neural Network Models

Estimating the relative importance of each sample in a training set has important practical and theoretical value, such as in importance sampling or curriculum learning. This kind of focus on individual samples invokes the concept of sample-wise learnability: How easy is it to correctly learn each sample (cf. PAC learnability)? In this paper, we approach the sample-wise learnability problem within a deep learning context. We propose a measure of the learnability of a sample with a given deep neural network (DNN) model. The basic idea is to train the given model on the training set, and for each sample, aggregate the hits and misses over the entire training epochs. Our experiments show that the sample-wise learnability measure collected this way is highly linearly correlated across different DNN models (ResNet-20, VGG-16, and MobileNet), suggesting that such a measure can provide deep general insights on the data's properties. We expect our method to help develop better curricula for training, and help us better understand the data itself.Comment: Accepted to AAAI 2019 Student Abstrac

The Consistency dimension and distribution-dependent learning from queries

We prove a new combinatorial characterization of polynomial learnability from equivalence queries, and state some of its consequences relating the learnability of a class with the learnability via equivalence and membership queries of its subclasses obtained by restricting the instance space. Then we propose and study two models of query learning in which there is a probability distribution on the instance space, both as an application of the tools developed from the combinatorial characterization and as models of independent interest.Postprint (published version

Robustifying Learnability

In recent years, the learnability of rational expectations equilibria (REE) and determinacy of economic structures have rightfully joined the usual performance criteria among the sought after goals of policy design. And while some contributions to the literature (for example Bullard and Mitra (2001) and Evans and Honkapohja (2002)) have made significant headway in establishing certain features of monetary policy rules that facilitate learning, a comprehensive treatment of policy design for learnability has yet to surface, especially for cases in which agents have potentially misspecified their learning models. This paper provides such a treatment. We argue that since even among professional economists a generally acceptable workhorse model of the economy has not been agreed upon, it is unreasonable to expect private agents to have collective rational expectations. We assume instead that agents have an approximate understanding of the workings of the economy and that their task of learning true reduced forms of the economy is subject to potentially destabilizing errors. We then ask: can a central bank set policy that accounts for learning errors but also succeeds in bounding them in a way that allows eventual learnability of the model, given policy. For different parameterizations of a given policy rule applied to a New Keynesian model, we use structured singular value analysis (from robust control) to find the largest ranges of misspecifications that can be tolerated in a learning model without compromising convergence to an REE. A parallel set of experiments seeks to determine the optimal stance (strong inflation as opposed to strong output stabilization) that allows for the greatest scope of errors in learning without leading to expectational instabilty in cases when the central bank designs both optimal and robust policy rules with commitment. We compare the features of all the rules contemplated in the paper with those that maximize economic performance in the true model, and we measure the performance cost of maximizing learnability under the various conditions mentioned here.monetary policy, learning, E-stability, model uncertainty, robustness

Robustifying learnability

In recent years, the learnability of rational expectations equilibria (REE) and determinacy of economic structures have rightfully joined the usual performance criteria among the sought-after goals of policy design. Some contributions to the literature, including Bullard and Mitra (2001) and Evans and Honkapohja (2002), have made significant headway in establishing certain features of monetary policy rules that facilitate learning. However a treatment of policy design for learnability in worlds where agents have potentially misspecified their learning models has yet to surface. This paper provides such a treatment. We begin with the notion that because the profession has yet to settle on a consensus model of the economy, it is unreasonable to expect private agents to have collective rational expectations. We assume that agents have only an approximate understanding of the workings of the economy and that their learning the reduced forms of the economy is subject to potentially destabilizing perturbations. The issue is then whether a central bank can design policy to account for perturbations and still assure the learnability of the model. Our test case is the standard New Keynesian business cycle model. For different parameterizations of a given policy rule, we use structured singular value analysis (from robust control theory) to find the largest ranges of misspecifications that can be tolerated in a learning model without compromising convergence to an REE. In addition, we study the cost, in terms of performance in the steady state of a central bank that acts to robustify learnability on the transition path to REE. (Note: This paper contains full-color graphics) JEL Classification: C6, E5E-stability, learnability, Learning, monetary policy, robust control

Robustifying learnability

In recent years, the learnability of rational expectations equilibria (REE) and determinacy of economic structures have rightfully joined the usual performance criteria among the sought-after goals of policy design. Some contributions to the literature, including Bullard and Mitra (2001) and Evans and Honkapohja (2002), have made significant headway in establishing certain features of monetary policy rules that facilitate learning. However a treatment of policy design for learnability in worlds where agents have potentially misspecified their learning models has yet to surface. This paper provides such a treatment. We begin with the notion that because the profession has yet to settle on a consensus model of the economy, it is unreasonable to expect private agents to have collective rational expectations. We assume that agents have only an approximate understanding of the workings of the economy and that their learning the reduced forms of the economy is subject to potentially destabilizing perturbations. The issue is then whether a central bank can design policy to account for perturbations and still assure the learnability of the model. Our test case is the standard New Keynesian business cycle model. For different parameterizations of a given policy rule, we use structured singular value analysis (from robust control theory) to find the largest ranges of misspecifications that can be tolerated in a learning model without compromising convergence to an REE.Robust control ; Monetary policy

What is usability in the context of the digital library and how can it be measured?

This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes

Learnability makes things click : a grounded theory approach to the software product evaluation

The aim of this doctoral dissertation is to investigate the phenomenon of learnability more deeply in order to better understand the learnability process. Grounded theory was used to determine the ground concepts based on fifteen users’ (N=15) actions (N=1836) in the WebCT Campus Edition’s virtual environment. Based on this study, the phenomenon of learnability and the learnability process is understood in greater detail and defined from the human centric of view, where the human being is the key actor. This doctoral dissertation answers the following research problem: 1. How learnable is the WebCT Campus Edition’s virtual environment? 2. How can the phenomenon of learnability be defined in a new way? The WebCT Campus Edition virtual environment’s learnability was measured with performance time and directions of action. In addition, the traditional learnability metrics of performance time and direction of users’ actions was used to verify the theoretical model of learnability and its nonlinearity. The result of this study showed that the variety of the WebCT Campus Edition’s learnability was higher between the individual users than it was between the different tasks. Therefore, the variety of task difficulty, i.e. the complexity or easiness of the different part of the user interface, have less influence on learnability in the WebCT Campus Edition than do the individual users’characteristics. Thus, the research results confirm the results found in earlier studies, where two important issues for usability evaluation and therefore evaluation on learnability, are the tasks and users individual characteristics. The theoretical model of learnability with following phases of information search, data collection, knowledge management, knowledge form, knowledge build and the result of action were determined from data. The theoretical model of learnability and its main patterns of a) data collection-information search, b)knowledge build-knowledge form and c) information searchknowledge management proves that learnability is a non-linear process. Therefore, the phenomenon of learnability cannot purely be defined by the separate properties of learnability, i.e. the properties of a user interface and a progressively enhanced linear process illustrated with learning curves. The theoretical model of learnability is one of the rare models of learnability that is based on empirical data. The use of grounded theory methodology means that the phenomenon of learnability is studied through tacit knowledge, i.e. through users’ real actions and though explicit knowledge, the users cognitive processes during interaction i.e. the phenomenon of learnability is approached from the holistic point of view, where the phenomenon of learnability is seen as one holistic process. Thus triangulation, where the phenomenon is interpreted through several split case studies are therefore unnecessary in this research setting. In conclusion, too many studies are still conducted in a laboratory situation using traditional methodological paradigms. More learnability studies with new methodological approaches in the natural environments are needed were the human, learning and non-linear process of learnability are in focus. It is important to understand more deeply the process of learnability and investigate more in greater detail the key elements that enhance learnability and on the other hand, cause learnability problems for users. Finally, based on the theoretical models of learnability, we can develop tools for the commercial user interface world in order to measure and test the learnability process more precisely and better understand how skills are actually learnt and how “to click” learnability