Making sense of sensory input

This paper attempts to answer a central question in unsupervised learning: what does it mean to "make sense" of a sensory sequence? In our formalization, making sense involves constructing a symbolic causal theory that both explains the sensory sequence and also satisfies a set of unity conditions. The unity conditions insist that the constituents of the causal theory -- objects, properties, and laws -- must be integrated into a coherent whole. On our account, making sense of sensory input is a type of program synthesis, but it is unsupervised program synthesis. Our second contribution is a computer implementation, the Apperception Engine, that was designed to satisfy the above requirements. Our system is able to produce interpretable human-readable causal theories from very small amounts of data, because of the strong inductive bias provided by the unity conditions. A causal theory produced by our system is able to predict future sensor readings, as well as retrodict earlier readings, and impute (fill in the blanks of) missing sensory readings, in any combination. We tested the engine in a diverse variety of domains, including cellular automata, rhythms and simple nursery tunes, multi-modal binding problems, occlusion tasks, and sequence induction intelligence tests. In each domain, we test our engine's ability to predict future sensor values, retrodict earlier sensor values, and impute missing sensory data. The engine performs well in all these domains, significantly out-performing neural net baselines. We note in particular that in the sequence induction intelligence tests, our system achieved human-level performance. This is notable because our system is not a bespoke system designed specifically to solve intelligence tests, but a general-purpose system that was designed to make sense of any sensory sequence

Sensory augmentation and the tactile sublime

This paper responds to recent developments in the field of sensory augmentation by analysing several technological devices that augment the sensory apparatus using the tactile sense. First, I will define the term sensory augmentation, as the use of technological modification to enhance the sensory apparatus, and elaborate on the preconditions for successful tactile sensory augmentation. These are the adaptability of the brain to unfamiliar sensory input and the specific qualities of the skin lending themselves to be used for the perception of additional sensory information. Two devices, Moon Ribas’ Seismic Sense and David Eagleman’s vest, will then be discussed as potential facilitators of aesthetic experiences in virtue of the tactile sensory augmentation that these devices allow. I will connect the experiences afforded by these devices to the Kantian categories of the mathematical and the dynamical sublime, and to existing accounts of tactile sublimity. Essentially, the objects these devices make sensible, earthquakes for the Seismic Sense and digital information for the vest, produce pleasurable feelings of potential danger, awe, and respect. The subsequent acclimation to this new way of sensing and the aim to comprehend its sensed object are then discussed as possible objections to the interpretation of these experiences as sublime, and as aesthetic in general. To exemplify these issues and concretise my thesis of tactile sensory augmentation as a trigger of the sublime, I will outline an experiment to use the vest as an aid for faster decision making on the stock market

Kant's cognitive architecture

Imagine a machine, equipped with sensors, receiving a stream of sensory information. It must, somehow, make sense of this stream of sensory data. But what, exactly, does this involve? We have an intuitive understanding of what is involved in “making sense” of sensory data – but can we specify precisely what is involved? Can this intuitive notion be formalized? In this thesis, we make three contributions. First, we provide a precise formalization of what it means to “make sense” of a sensory sequence. According to our definition, making sense means constructing a symbolic causal theory that explains the sensory sequence and satisfies a set of unity conditions that were inspired by Kant’s discussion in the first half of the Critique of Pure Reason. According to our interpretation, making sense of sensory input is a type of program synthesis, but it is unsupervised program synthesis. Our second contribution is a computer implementation, the Apperception Engine, that was designed to satisfy our requirements for making sense of a sensory sequence. Our system is able to produce interpretable human-readable causal theories from very small amounts of data, because of the strong inductive bias provided by the Kantian unity constraints. A causal theory produced by our system is able to predict future sensor readings, as well as retrodict earlier readings, and impute missing sensory readings. In fact, it is able to do all three tasks simultaneously. The engine is implemented in Answer Set Programming (ASP) and induces theories expressed in an extension of Datalog that includes causal rules and constraints. We test the engine in a diverse variety of domains, including cellular automata, rhythms and simple nursery tunes, multi-modal binding problems, occlusion tasks, and sequence induction IQ tests. In each domain, we test our engine’s ability to predict future sensor values, retrodict earlier sensor values, and impute missing sensory data. The Apperception Engine performs well in all these domains, significantly out-performing neural net baselines. These results are significant because neural nets typically struggle to solve the binding problem (where information from different modalities must somehow be combined together into different aspects of one unified object) and fail to solve occlusion tasks (in which objects are sometimes visible and sometimes obscured from view). We note in particular that in the sequence induction IQ tasks, our system achieves human-level performance. This is notable because the Apperception Engine was not designed to solve these IQ tasks; it is not a bespoke hand-engineered solution to this particular domain. – Rather, it is a general purpose system that attempts to make sense of any sensory sequence, that just happens to be able to solve these IQ tasks “out of the box”. Our third contribution is a major extension of the engine to handle noisy and ambiguous data. While the initial implementation assumes the sensory input has already been preprocessed into ground atoms of first-order logic, our extension makes sense of raw unprocessed input – a sequence of pixel images from a video camera, for example. The resulting system is a neuro-symbolic framework for distilling interpretable theories out of streams of raw, unprocessed sensory experience.Open Acces

Music listening as coping behavior : from reactive response to sense-making

Coping is a survival mechanism of living organisms. It is not merely reactive, but also involves making sense of the environment by rendering sensory information into percepts that have meaning in the context of an organism's cognitions. Music listening, on the other hand, is a complex task that embraces sensory, physiological, behavioral, and cognitive levels of processing. Being both a dispositional process that relies on our evolutionary toolkit for coping with the world and a more elaborated skill for sense-making, it goes beyond primitive action-reaction couplings by the introduction of higher-order intermediary variables between sensory input and effector reactions. Consideration of music-listening from the perspective of coping treats music as a sound environment and listening as a process that involves exploration of this environment as well as interactions with the sounds. Several issues are considered in this regard such as the conception of music as a possible stressor, the role of adaptive listening, the relation between coping and reward, the importance of self-regulation strategies in the selection of music, and the instrumental meaning of music in the sense that it can be used to modify the internal and external environment of the listener

A Functional Model of the Aesthetic Response

In a process of somatic evolution, the brain semi-randomly generates initially-unstable neural circuits that are selectively stabilized if they succeed in making sense out of raw sensory input. The human aesthetic response serves the function of stabilizing the circuits that successfully mediate perception and interpretation, making those faculties more agile, conferring selective advantage. It is triggered by structures in art and nature that provoke the making of sense. Art is deliberate human action aimed at triggering the aesthetic response in others; thus, if successful, it serves the same function of making perception and interpretation more agile. These few principles initiate a cascade of emergent phenomena which account for many observed qualities of aesthetics, including universality and idiosyncrasy of taste, the relevance of artists’ intentions, the virtues of openness and resonance, the dysfunction of formulaic art, and the fact that methods of art correspond to modes of perceptual transformation

An intuitive tangible game controller

This paper outlines the development of a sensory feedback device providing a low cost, versatile and intuitive interface for controlling digital environments, in this example a flight simulator. Gesture based input allows for a more immersive experience, so rather than making the user feel like they are controlling an aircraft the intuitive interface allows the user to become the aircraft that is controlled by the movements of the user's hand. The movements are designed to feel intuitive and allow for a sense of immersion that would be difficult to achieve with an alternative interface. In this example the user's hand can become the aircraft much the same way that a child would imagine it

Measuring the World: Olfaction as a Process Model of Perception

How much does stimulus input shape perception? The common-sense view is that our perceptions are representations of objects and their features and that the stimulus structures the perceptual object. The problem for this view concerns perceptual biases as responsible for distortions and the subjectivity of perceptual experience. These biases are increasingly studied as constitutive factors of brain processes in recent neuroscience. In neural network models the brain is said to cope with the plethora of sensory information by predicting stimulus regularities on the basis of previous experiences. Drawing on this development, this chapter analyses perceptions as processes. Looking at olfaction as a model system, it argues for the need to abandon a stimulus-centred perspective, where smells are thought of as stable percepts, computationally linked to external objects such as odorous molecules. Perception here is presented as a measure of changing signal ratios in an environment informed by expectancy effects from top-down processes

Prediction of Choice from Competing Mechanosensory and Choice-Memory Cues during Active Tactile Decision Making

Perceptual decision making is an active process where animals move their sense organs to extract task-relevant information. To investigate how the brain translates sensory input into decisions during active sensation, we developed a mouse active touch task where the mechanosensory input can be precisely measured and that challenges animals to use multiple mechanosensory cues. Male mice were trained to localize a pole using a single whisker and to report their decision by selecting one of three choices. Using high-speed imaging and machine vision, we estimated whisker–object mechanical forces at millisecond resolution. Mice solved the task by a sensory-motor strategy where both the strength and direction of whisker bending were informative cues to pole location. We found competing influences of immediate sensory input and choice memory on mouse choice. On correct trials, choice could be predicted from the direction and strength of whisker bending, but not from previous choice. In contrast, on error trials, choice could be predicted from previous choice but not from whisker bending. This study shows that animal choices during active tactile decision making can be predicted from mechanosensory and choice-memory signals, and provides a new task well suited for the future study of the neural basis of active perceptual decisions

Models, Brains, and Scientific Realism

Prediction Error Minimization theory (PEM) is one of the most promising attempts to model perception in current science of mind, and it has recently been advocated by some prominent philosophers as Andy Clark and Jakob Hohwy. Briefly, PEM maintains that “the brain is an organ that on aver-age and over time continually minimizes the error between the sensory input it predicts on the basis of its model of the world and the actual sensory input” (Hohwy 2014, p. 2). An interesting debate has arisen with regard to which is the more adequate epistemological interpretation of PEM. Indeed, Hohwy maintains that given that PEM supports an inferential view of perception and cognition, PEM has to be considered as conveying an internalist epistemological perspective. Contrary to this view, Clark maintains that it would be incorrect to interpret in such a way the indirectness of the link between the world and our inner model of it, and that PEM may well be combined with an externalist epistemological perspective. The aim of this paper is to assess those two opposite interpretations of PEM. Moreover, it will be suggested that Hohwy’s position may be considerably strengthened by adopting Carlo Cellucci’s view on knowledge (2013)

Beyond representations: towards an action-centric perspective on tangible interaction

In the light of theoretical as well as concrete technical development, we discuss a conceptual shift from an information-centric to an action-centric perspective on tangible interactive technology. We explicitly emphasise the qualities of shareable use, and the importance of designing tangibles that allow for meaningful manipulation and control of the digital material. This involves a broadened focus from studying properties of the interface, to instead aim for qualities of the activity of using a system, a general tendency towards designing for social and sharable use settings and an increased openness towards multiple and subjective interpretations. An effect of this is that tangibles are not designed as representations of data, but as resources for action. We discuss four ways that tangible artefacts work as resources for action: (1) for physical manipulation; (2) for referential, social and contextually oriented action; (3) for perception and sensory experience; (4) for digitally mediated action