To perceive an affordance is to perceive an object or situation as presenting an opportunity for action. The concept of affordances has been taken up across wide range of disciplines, including AI. I explore an interesting extension of the concept of affordances in robotics. Among the affordances that artificial systems have been engineered to detect are affordances to deliberate. In psychology, affordances are typically limited to bodily action, so the it is noteworthy that AI researchers have found it helpful to extend the concept to encompass mental actions. I propose that psychologists can learn from this extension, and argue that human subjects can perceive mental affordances, such as affordances to attend, affordances to imagine and affordances to count

McClelland, T

English

Apollo (Cambridge)

AI and Affordances for Mental Action (Extended Abstract) Dr Tom McClelland 11. INTRODUCTION The relationship between behavioural psychology and artificial intelligence is reciprocal: just as AI researchers can apply lessons from psychology to artificial behaviour, psychologists can apply lessons from AI to human behaviour. In some cases these interactions will have a cyclic structure, with one discipline inspiring new ideas in the other, then those ideas in turn being taken up by the original discipline. Although this reciprocal arrangement has yielded a wealth of results, there are doubtless a vast range of lessons that remain unrecognised. Put another way, there are surely insights in each discipline that could be fruitfully taken up by the other, but which have not yet been extracted. My aim in this paper is extract one such lesson from AI and to present some proposals about how it might be applied to human behaviour. I start with an insight from psychology – the role of affordance perception in human behaviour – and consider how this insight has stimulated new ideas in AI. I then consider how one of these ideas – Raubal’s [6, 7] notion of mental affordances in robotics – moves beyond the understanding of affordances offered by psychologists. Finally, I explore how the notion of mental affordances might be applied in human psychology, and how it might be further developed in AI.  2. APPLYING AFFORDANCE THEORY TO AI The concept of affordances was introduced by the ecological psychologist J.J. Gibson, and his most fully developed articulation of the concept can be found in his 1979 work The Ecological Approach to Visual Perception [2]. In that book, Gibson explains the concept of affordances as follows: The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The verb to afford is found in the dictionary, but the noun affordance is not. I have made it up. I mean by it something that refers to both the environment and the animal in a way that no existing term does. It implies the complementarity of the animal and the environment.’ [1, p.127] Classic cases of affordances are those pertaining to basic bodily actions such as walking, gripping or catching. A pathway might afford walking, a stick might afford gripping and a ball might afford catching. Whether something has these affordances depends on the body and the abilities of the agent: a ball that affords catching for one agent might not afford catching for another. At the heart of the concept of affordances is a specific understanding of the relationship between action and perception. Gibson’s key theoretical claim is that agents do not perceive an action-neutral environment then infer what actions are available to them in an environment of that description. Instead, agents can simply perceive opportunities for action.  For Gibson, this claim was part of a radical understanding of behaviour according to which internal processes are unnecessary for perception, or for the transition from perception to action. Agents can pick-up affordances by directly perceiving optical patterns in the environment, and these affordances can directly guide action without the need for mediating processes. Some in the ecological school of psychology have sought to retain this radical understanding of behaviour. However, the majority of those who have taken up Gibson’s concept of affordances have left these wider claims behind. AI is no exception to this. Horton et al [2] note that AI researchers understand affordance perception in terms of internal representations of opportunities for action. But if perceiving and acting on affordances is taken to require internal representation, what value is there in the concept? Horton et al outline the application of affordances to AI as follows: In designing artificial agents, several successful patterns for control and coordination of perception and action have emerged. Some of these approaches share an important characteristic - a clear emphasis on utilizing the environment, and the agent’s interaction with it, to reduce the complexity of representation and reasoning. This characteristic is founded on an ecological view of the agent - an entity embodied in a world rich with observable cues that can help guide the agent’s behavior. [2, p.71] By programming behaviour in a way that’s sensitive to environmental affordances, one can thus mimimize the need for internal representations. This is a valuable result even if the Gibsonian dream of eliminating internal processing entirely is deemed implausible [2, p.79]. An especially interesting consequence of affordance-based programming is that agents engage in exploratory behaviour. This behaviour is not directed toward any specific goal, but by interacting with items in the environment in a range of ways the agent discovers the opportunities for action presented by that object, and by other objects of the same kind. Stoytchev [3], for instance, offers a distinctive approach to tool-learning in robotics that involves the robot engaging in random ‘dabbling’ behaviour toward a presented tool. The robot performs a variety of random actions on the tools and learns the results of these actions. By engaging in this behaviour, the robot is then able to perform a tool-using task that they would have been unable to perform without the lessons acquired from their goal-independent exploration.  3. MENTAL AFFORDANCES IN AI The affordances discussed by Gibson (and by the vast majority of those who have picked up on his term) are affordances for bodily action. As mentioned above, classic affordances include affording walking, affording gripping and affording catching. The affordances explored in AI research are almost universally affordances for bodily action in the sense that they involve some kind of physical movement on the part of the artificial agent (whether it might be virtual movement in simulation or actual movement through an artificial body). Examples include affordances for poking, pushing, pulling, rotating and lifting [2, p.73]. However, in a small number of cases AI researchers talk about affordances for mental action. Consider the following passage from Raubal & Moratz: …a public transportation terminal affords for a person to enter different buses and trains. It also affords to buy tickets or make a phone call. A path affords remembering and selecting,  a  decision  point  affords  orienting  and  deciding,  etc. In general, such situations offer for the person the mental affordance of deciding which of the perceived affordances to utilize according to her goal. [4, p.3] Some of the affordances cited in this passage are affordances for bodily actions, such as the bodily act of getting on a specific bus. But the ‘mental affordances’ are affordances for mental action, such as the mental act of deciding what to do. Raubal & Moratz offer an affordance-oriented robot architecture that includes sensitivity to these mental affordances. They explain this architecture as follows: Mental affordances (Maff) arise for the agent when perceiving a set of physical and social-institutional affordances in an environment at a specific location and time. Affordances offer possibilities for action as well as possibilities for the agent to reason about them and decide whether to utilize them or not, i.e., mental affordances. The agent needs to perform an internal operation Op (Int) to utilize a mental affordance.  Internal operations are carried out on the agent’s beliefs (including its history and experiences) and lead to an internal outcome O (Int). In order to transfer such outcome to the world, the agent has to perform an external operation Op (Ext), which then leads to an external outcome O (Ext), i.e., some change of the external world. [4, 95-96] So besides being sensitive to specific affordances for physical action, the robot is sensitive to situations in which a decision is required [4, 5]. The opportunities for physical action can be understood as first-order affordances. The situations in which a decision is required can be understood as second-order affordances, as they are affordances to decide between first-order affordances. Raubal & Moratz argue that this architecture better enables robots to respond to a dynamic environment and allows them to communicate plans before they are acted upon. Although they don’t draw explicitly on Raubal & Moratz, Saratha & Scheutz have also recently argued that uptake of such second-order affordances enhances performance in various ways [6].  4. MENTAL AFFORDANCE IN PSYCHOLOGY Raubal & Moratz [4] emphasise that one of the advantages of their mental affordance-based architecture is that it better corresponds to the architecture of human behaviour. However, when we look at how the concept of affordances is used in the psychology literature, we find no reference to affordances for mental actions such as deliberating. Psychology did AI a service with the notion of affordances, and perhaps here AI can return the favour. I propose that the notion of mental affordances opens up a range of promising avenues of enquiry for the understanding of human behaviour. Raubal & Moratz’s [4] example of affording deliberation is an obvious initial target. Do human agents perceive opportunities to make a decision? Does the concept of affordances for deliberation allow us to offer better explanations of when and how humans engage in explicit decision making? It certainly seems to fit with our phenomenology that situations afford deliberation: just as we experience a single open path as demanding to be walked down, we experience a fork in the path as demanding an act of explicit deliberation about which path to take. Although affordance-based theories are ultimately answerable to the empirical data, their phenomenological plausibility is responsible for a lot of their appeal [7]. If the notion of mental affordances tallies with our phenomenology, this would be a point in its favour. Moving beyond affordances for deliberation, we can explore the possibility of other affordances for mental action. I make a case for a range of possible affordances for mental actions, starting with some relatively innocuous proposals then building up to some more dramatic suggestions. First, I suggest that stimuli can afford covert attention. Since covert attention is a mental act, to afford covert attention is to afford a mental act. I suggest this holds even if all such stimuli also afford the bodily act of overtly attending. Second, I consider the possibility of stimuli affording offline bodily acts. The act of mental self-rotation, for instance, is an off-line counterpart to the bodily of act of moving one’s body around. In situations where subjects need to assess how things appear from another agent’s perspective, it has been established that they perform this act of mental self-rotation [8]. I consider whether this kind of situation can appropriately be described as affording mental self-rotation. Third, I consider the possibility that the environment can afford the performance of mathematical operations such as counting. I propose that the role of counting in certain mental disorders – specifically utilization behaviour [9] and OCD [10] – might fruitfully be explained in terms of a failure to suppress afforded mathematical activities. For each of these proposals, I explain how they might be investigated empirically. If it transpired that there were affordances for mental action, what would that teach us about the architecture of human behaviour? To perceive an affordance is to perceive an opportunity for action, and to perceive an opportunity for action is to reduce the level of complexity required in the processes mediating perception and behaviour. The need to minimise cognitive demands is something that AI engineers share with human evolutionary history. As such, we shouldn’t be surprised that the cognitive-shortcuts found in AI are mirrored in nature. Having made some provisional suggestions about how mental affordances might figure in human psychology, I then discuss how a broader conception of mental affordances might feed back into AI. In particular, I focus on the connection between an affordance-based architecture and exploration-based learning. If goal-independent ‘dabbling’ with external objects allows agents to learn the affordances for physical action offered by external objects [3], perhaps goal-independent dabbling with internal states will allow agents to learn the affordances for mental action offered by their own internal architecture. By freely exploring the effects of various internal manipulations, artificial agents may be able to discover strategies for deploying their cognitive capacities more effectively: discoveries that would be unavailable in a rule-based meta-cognitive architecture. REFERENCES [1] Gibson, J. (1979). The Ecological Approach to Visual Perception. New York: Psychology Press. [2] Horton, T., Chakraborty, A., & St. Amant, R. (2012). Affordances for robots: a brief survey. Avant, 3(2), 70-86. [3] Stoytchev, A. Behavior-Grounded Representation of Tool Affordances Stoytchev, Alexander. Proceedings of the 2005 IEEE International Conference on Robotics and Automation. IEEE, 2005.  [4] Raubal, M., & Moratz, R. (2008). A Functional Model for Affordance Based Agents. In E. Rome, Affordance-based robotic control (pp. 91-105). Berlin: Springer. [5] Raubal, M. (2001). Human Wayfinding in Unfamiliar Buildings: A Simulation With a Cognizing Agent. Cognitive Processing, 363-388. [6] Sarathy, V., & Scheutz, M. (2016). Cognitive Affordance Representations in Uncertain Logic. Proceedings, Fifteenth International Conference on Principles of Knowledge Representation and Reasoning.  [7] eft, H. (2001). Ecological Psychology in Context: James Gibson, Roger Barker and the Legacy of William James's Radical Empiricism. New York: Psychology Press. [8] Kessler, K., & Thomsom, L. A. (2010). The Embodied Nature of Spatial Perspective Taking: Embodied Transformation versus Sensorimotor Interference. Cognition, 114(1), 72-88. [9] Brazzelli, M., & Spinnler, H. (1998). An Example of Lack of Frontal Inhibition: the 'Utilization Behaviour'. European Journal of Neurology(5), 347-353. [10] Pinto, A. et al. (2008) Further development of YBOCS dimensions in the OCD Collaborative Genetics study: symptoms vs. categories. Psychiatry research 160(1): 83-93. 

AI and affordances for mental action

McClelland, Tom

Warwick Research Archives Portal Repository

https://www.repository.cam.ac.uk/bitstream/1810/309000/1/AI%20and%20affordances%20for%20mental%20action.pdf

AI and affordances for mental action

Abstract

Similar works

Full text

Available Versions

Apollo (Cambridge)

Warwick Research Archives Portal Repository