Multi-label affordance mapping from egocentric vision

Accurate affordance detection and segmentation with pixel precision is an important piece in many complex systems based on interactions, such as robots and assitive devices. We present a new approach to affordance perception which enables accurate multi-label segmentation. Our approach can be used to automatically extract grounded affordances from first person videos of interactions using a 3D map of the environment providing pixel level precision for the affordance location. We use this method to build the largest and most complete dataset on affordances based on the EPIC-Kitchen dataset, EPIC-Aff, which provides interaction-grounded, multi-label, metric and spatial affordance annotations. Then, we propose a new approach to affordance segmentation based on multi-label detection which enables multiple affordances to co-exists in the same space, for example if they are associated with the same object. We present several strategies of multi-label detection using several segmentation architectures. The experimental results highlight the importance of the multi-label detection. Finally, we show how our metric representation can be exploited for build a map of interaction hotspots in spatial action-centric zones and use that representation to perform a task-oriented navigation.Comment: International Conference on Computer Vision (ICCV) 202

Nonconceptual content and the nature of perceptual experience

Recent philosophy of mind and epistemology has seen an important and influential trend towards accounting for at least some features of experiences in content-involving terms. It is a contested point whether ascribing content to experiences can account for all the intrinsic properties of experiences, but on many theories of experiences there are close links between the ascription of content and the ways in which experiences are ascribed and typed. The issues here have both epistemological and psychological dimensions. On the one hand, a theory of experiential content has a fundamental role in explaining how knowledge of the world can be acquired through experience. On the other hand, there are important psychological questions about the phenomenology of experiences and the conditions under which content ascriptions are made. The debate about whether experiences have conceptual or nonconceptual content is highly relevant to both the epistemological and psychological issues. This is apparent in the work of Gareth Evans who is often cited as the first philosopher to propose a theory of nonconceptual content. This paper will explicate and examine Evans' theory. In particular, attention will be paid to Evans' claim that perceptually-derived information with nonconceptual content can only qualify as a perceptual experience when it serves as input to a thinking, concept-applying, and reasoning system. We will sketch out a theoretical approach to the nature of nonconceptual content that builds on Evans' insight but does not follow him in this restrictive claim

Diversity in Spatial Language Within Communities: The Interplay of Culture, Language and Landscape in Representations of Space (Short Paper)

Significant diversity exists in the way languages structure spatial reference, and this has been shown to correlate with diversity in non-linguistic spatial behaviour. However, most research in spatial language has focused on diversity between languages: on which spatial referential strategies are represented in the grammar, and to a lesser extent which of these strategies are preferred overall in a given language. However, comparing languages as a whole and treating each language as a single data point provides a very partial picture of linguistic spatial behaviour, failing to recognise the very significant diversity that exists within languages, a largely under-investigated but now emerging field of research. This paper focuses on language-internal diversity, and on the central role of a range of sociocultural and demographic factors that intervene in the relationship between humans, languages, and the physical environments in which communities live

Planning and Explanations with a Learned Spatial Model

This paper reports on a robot controller that learns and applies a cognitively-based spatial model as it travels in challenging, real-world indoor spaces. The model not only describes indoor space, but also supports robust, model-based planning. Together with the spatial model, the controller\u27s reasoning framework allows it to explain and defend its decisions in accessible natural language. The novel contributions of this paper are an enhanced cognitive spatial model that facilitates successful reasoning and planning, and the ability to explain navigation choices for a complex environment. Empirical evidence is provided by simulation of a commercial robot in a large, complex, realistic world

The road more travelled: The differential effects of spatial experience in young and elderly participants

Our spatial mental representations allow us to give refined descriptions of the environment in terms of the relative locations and distances between objects and landmarks. In this study, we investigated the effects of familiarity with the everyday environment, in terms of frequency of exploration and mode of transportation, on categorical and coordinate spatial relations, on young and elderly participants, controlling for socio-demographic factors. Participants were tested with a general anamnesis, a neuropsychological assessment, measures of explorations and the Landmark Positioning on a Map task. The results showed: (a) a modest difference in performance with categorical spatial relations; (b) a larger difference in coordinate spatial relations; (c) a significant moderating effect of age on the relationship between familiarity and spatial relations, with a stronger relation among the elderly than the young. Ceteris paribus, the role of direct experience with exploring their hometown on spatial mental representations appeared to be more important in the elderly than in the young. This advantage appears to make the elderly wiser and likely protects them from the detrimental effects of aging on spatial mental representations

Action-Related Representations

Theories of grounded cognition state that there is a meaningful connection between action and cognition. Although these claims are widely accepted, the nature and structure of this connection is far from clear and is still a matter of controversy. This book argues for a type of cognitive representation that essentially combines cognition and action, and which is foundational for higher-order cognitive capacities

Visual world studies of conversational perspective taking: similar findings, diverging interpretations

Visual-world eyetracking greatly expanded the potential for insight into how listeners access and use common ground during situated language comprehension. Past reviews of visual world studies on perspective taking have largely taken the diverging findings of the various studies at face value, and attributed these apparently different findings to differences in the extent to which the paradigms used by different labs afford collaborative interaction. Researchers are asking questions about perspective taking of an increasingly nuanced and sophisticated nature, a clear indicator of progress. But this research has the potential not only to improve our understanding of conversational perspective taking. Grappling with problems of data interpretation in such a complex domain has the unique potential to drive visual world researchers to a deeper understanding of how to best map visual world data onto psycholinguistic theory. I will argue against this interactional affordances explanation, on two counts. First, it implies that interactivity affects the overall ability to form common ground, and thus provides no straightforward explanation of why, within a single noninteractive study, common ground can have very large effects on some aspects of processing (referential anticipation) while having negligible effects on others (lexical processing). Second, and more importantly, the explanation accepts the divergence in published findings at face value. However, a closer look at several key studies shows that the divergences are more likely to reflect inconsistent practices of analysis and interpretation that have been applied to an underlying body of data that is, in fact, surprisingly consistent. The diverging interpretations, I will argue, are the result of differences in the handling of anticipatory baseline effects (ABEs) in the analysis of visual world data. ABEs arise in perspective-taking studies because listeners have earlier access to constraining information about who knows what than they have to referential speech, and thus can already show biases in visual attention even before the processing of any referential speech has begun. To be sure, these ABEs clearly indicate early access to common ground; however, access does not imply integration, since it is possible that this information is not used later to modulate the processing of incoming speech. Failing to account for these biases using statistical or experimental controls leads to over-optimistic assessments of listeners’ ability to integrate this information with incoming speech. I will show that several key studies with varying degrees of interactional affordances all show similar temporal profiles of common ground use during the interpretive process: early anticipatory effects, followed by bottom-up effects of lexical processing that are not modulated by common ground, followed (optionally) by further late effects that are likely to be post-lexical. Furthermore, this temporal profile for common ground radically differs from the profile of contextual effects related to verb semantics. Together, these findings are consistent with the proposal that lexical processes are encapsulated from common ground, but cannot be straightforwardly accounted for by probabilistic constraint-based approaches