Improving Representation Learning for Deep Clustering and Few-shot Learning

The amounts of data in the world have increased dramatically in recent years, and it is quickly becoming infeasible for humans to label all these data. It is therefore crucial that modern machine learning systems can operate with few or no labels. The introduction of deep learning and deep neural networks has led to impressive advancements in several areas of machine learning. These advancements are largely due to the unprecedented ability of deep neural networks to learn powerful representations from a wide range of complex input signals. This ability is especially important when labeled data is limited, as the absence of a strong supervisory signal forces models to rely more on intrinsic properties of the data and its representations. This thesis focuses on two key concepts in deep learning with few or no labels. First, we aim to improve representation quality in deep clustering - both for single-view and multi-view data. Current models for deep clustering face challenges related to properly representing semantic similarities, which is crucial for the models to discover meaningful clusterings. This is especially challenging with multi-view data, since the information required for successful clustering might be scattered across many views. Second, we focus on few-shot learning, and how geometrical properties of representations influence few-shot classification performance. We find that a large number of recent methods for few-shot learning embed representations on the hypersphere. Hence, we seek to understand what makes the hypersphere a particularly suitable embedding space for few-shot learning. Our work on single-view deep clustering addresses the susceptibility of deep clustering models to find trivial solutions with non-meaningful representations. To address this issue, we present a new auxiliary objective that - when compared to the popular autoencoder-based approach - better aligns with the main clustering objective, resulting in improved clustering performance. Similarly, our work on multi-view clustering focuses on how representations can be learned from multi-view data, in order to make the representations suitable for the clustering objective. Where recent methods for deep multi-view clustering have focused on aligning view-specific representations, we find that this alignment procedure might actually be detrimental to representation quality. We investigate the effects of representation alignment, and provide novel insights on when alignment is beneficial, and when it is not. Based on our findings, we present several new methods for deep multi-view clustering - both alignment and non-alignment-based - that out-perform current state-of-the-art methods. Our first work on few-shot learning aims to tackle the hubness problem, which has been shown to have negative effects on few-shot classification performance. To this end, we present two new methods to embed representations on the hypersphere for few-shot learning. Further, we provide both theoretical and experimental evidence indicating that embedding representations as uniformly as possible on the hypersphere reduces hubness, and improves classification accuracy. Furthermore, based on our findings on hyperspherical embeddings for few-shot learning, we seek to improve the understanding of representation norms. In particular, we ask what type of information the norm carries, and why it is often beneficial to discard the norm in classification models. We answer this question by presenting a novel hypothesis on the relationship between representation norm and the number of a certain class of objects in the image. We then analyze our hypothesis both theoretically and experimentally, presenting promising results that corroborate the hypothesis

Explainable AI: The new 42?

Explainable AI is not a new field. Since at least the early exploitation of C.S. Pierce’s abductive reasoning in expert systems of the 1980s, there were reasoning architectures to support an explanation function for complex AI systems, including applications in medical diagnosis, complex multi-component design, and reasoning about the real world. So explainability is at least as old as early AI, and a natural consequence of the design of AI systems. While early expert systems consisted of handcrafted knowledge bases that enabled reasoning over narrowly well-defined domains (e.g., INTERNIST, MYCIN), such systems had no learning capabilities and had only primitive uncertainty handling. But the evolution of formal reasoning architectures to incorporate principled probabilistic reasoning helped address the capture and use of uncertain knowledge. There has been recent and relatively rapid success of AI/machine learning solutions arises from neural network architectures. A new generation of neural methods now scale to exploit the practical applicability of statistical and algebraic learning approaches in arbitrarily high dimensional spaces. But despite their huge successes, largely in problems which can be cast as classification problems, their effectiveness is still limited by their un-debuggability, and their inability to “explain” their decisions in a human understandable and reconstructable way. So while AlphaGo or DeepStack can crush the best humans at Go or Poker, neither program has any internal model of its task; its representations defy interpretation by humans, there is no mechanism to explain their actions and behaviour, and furthermore, there is no obvious instructional value.. the high performance systems can not help humans improve. Even when we understand the underlying mathematical scaffolding of current machine learning architectures, it is often impossible to get insight into the internal working of the models; we need explicit modeling and reasoning tools to explain how and why a result was achieved. We also know that a significant challenge for future AI is contextual adaptation, i.e., systems that incrementally help to construct explanatory models for solving real-world problems. Here it would be beneficial not to exclude human expertise, but to augment human intelligence with artificial intelligence

Metacognition and Reflection by Interdisciplinary Experts: Insights from Cognitive Science and Philosophy

Interdisciplinary understanding requires integration of insights from different perspectives, yet it appears questionable whether disciplinary experts are well prepared for this. Indeed, psychological and cognitive scientific studies suggest that expertise can be disadvantageous because experts are often more biased than non-experts, for example, or fixed on certain approaches, and less flexible in novel situations or situations outside their domain of expertise. An explanation is that experts’ conscious and unconscious cognition and behavior depend upon their learning and acquisition of a set of mental representations or knowledge structures. Compared to beginners in a field, experts have assembled a much larger set of representations that are also more complex, facilitating fast and adequate perception in responding to relevant situations. This article argues how metacognition should be employed in order to mitigate such disadvantages of expertise: By metacognitively monitoring and regulating their own cognitive processes and representations, experts can prepare themselves for interdisciplinary understanding. Interdisciplinary collaboration is further facilitated by team metacognition about the team, tasks, process, goals, and representations developed in the team. Drawing attention to the need for metacognition, the article explains how philosophical reflection on the assumptions involved in different disciplinary perspectives must also be considered in a process complementary to metacognition and not completely overlapping with it. (Disciplinary assumptions are here understood as determining and constraining how the complex mental representations of experts are chunked and structured.) The article concludes with a brief reflection on how the process of Reflective Equilibrium should be added to the processes of metacognition and philosophical reflection in order for experts involved in interdisciplinary collaboration to reach a justifiable and coherent form of interdisciplinary integration. An Appendix of “Prompts or Questions for Metacognition” that can elicit metacognitive knowledge, monitoring, or regulation in individuals or teams is included at the end of the article

Concepts, Introspection, and Phenomenal Consciousness: An Information-Theoretical Approach

This essay is a sustained information-theoretic attempt to bring new light on some of the perennial problems in the philosophy of mind surrounding phenomenal consciousness and introspection. Following Dretske (1981), we present and develop an informational psychosemantics as it applies to what we call <em>sensory concepts</em>, concepts that apply, roughly, to so-called secondary qualities of objects. We show that these concepts have a special informational character and semantic structure that closely tie them to the brain states realizing conscious qualitative experiences. We then develop an account of introspection which exploits this special nature of sensory concepts. The result is a new class of concepts, which, following recent terminology, we call <em>phenomenal concepts</em>: these concepts refer to phenomenal experience itself and are the vehicles used in introspection. On our account, the connection between sensory and phenomenal concepts is very tight: it consists in different semantic uses of the same cognitive structures underlying the sensory concepts, like RED. Contrary to widespread opinion, we show that information theory contains all the resources to satisfy internalist intuitions about phenomenal consciousness, while not offending externalist ones. A consequence of this account is that it explains and predicts the so-called conceivability arguments against physicalism on the basis of the special nature of sensory and phenomenal concepts. Thus we not only show why physicalism is not threatened by such arguments, but also demonstrate its strength in virtue of its ability to predict and explain away such arguments in a principled way. However, we take the main contribution of this work to be what it provides in addition to a response to those conceivability arguments, namely, a substantive account of the interface between sensory and conceptual systems and the mechanisms of introspection as based on the special nature of the information flow between them

Mental Structures

An ongoing philosophical discussion concerns how various types of mental states fall within broad representational genera—for example, whether perceptual states are “iconic” or “sentential,” “analog” or “digital,” and so on. Here, I examine the grounds for making much more specific claims about how mental states are structured from constituent parts. For example, the state I am in when I perceive the shape of a mountain ridge may have as constituent parts my representations of the shapes of each peak and saddle of the ridge. More specific structural claims of this sort are a guide to how mental states fall within broader representational kinds. Moreover, these claims have significant implications of their own about semantic, functional, and epistemic features of our mental lives. But what are the conditions on a mental state's having one type of constituent structure rather than another? Drawing on explanatory strategies in vision science, I argue that, other things being equal, the constituent structure of a mental state determines what I call its distributional properties—namely, how mental states of that type can, cannot, or must co‐occur with other mental states in a given system. Distributional properties depend critically on and are informative about the underlying structures of mental states, they abstract in important ways from aspects of how mental states are processed, and they can yield significant insights into the variegation of psychological capacities

Neural Models of Seeing and Thinking

Air Force Office of Scientific Research (F49620-01-1-0397); Office of Naval Research (N00014-01-1-0624

Embodied cognition and temporally extended agency

According to radical versions of embodied cognition, human cognition and agency should be explained without the ascription of representational mental states. According to a standard reply, accounts of embodied cognition can explain only instances of cognition and agency that are not “representation-hungry”. Two main types of such representation-hungry phenomena have been discussed: cognition about “the absent” and about “the abstract”. Proponents of representationalism have maintained that a satisfactory account of such phenomena requires the ascription of mental representations. Opponents have denied this. I will argue that there is another important representation-hungry phenomenon that has been overlooked in this debate: temporally extended planning agency. In particular, I will argue that it is very difficult to see how planning agency can be explained without the ascription of mental representations, even if we grant, for the sake of argument, that cognition about the absent and abstract can. We will see that this is a serious challenge for the radical as well as the more modest anti-representationalist versions of embodied cognition, and we will see that modest anti-representationalism is an unstable position

Why 'scaffolding' is the wrong metaphor : the cognitive usefulness of mathematical representations.

The metaphor of scaffolding has become current in discussions of the cognitive help we get from artefacts, environmental affordances and each other. Consideration of mathematical tools and representations indicates that in these cases at least (and plausibly for others), scaffolding is the wrong picture, because scaffolding in good order is immobile, temporary and crude. Mathematical representations can be manipulated, are not temporary structures to aid development, and are refined. Reflection on examples from elementary algebra indicates that Menary is on the right track with his ‘enculturation’ view of mathematical cognition. Moreover, these examples allow us to elaborate his remarks on the uniqueness of mathematical representations and their role in the emergence of new thoughts.Peer reviewe