Modeling the Violation of Reward Maximization and Invariance in Reinforcement Schedules

It is often assumed that animals and people adjust their behavior to maximize reward acquisition. In visually cued reinforcement schedules, monkeys make errors in trials that are not immediately rewarded, despite having to repeat error trials. Here we show that error rates are typically smaller in trials equally distant from reward but belonging to longer schedules (referred to as “schedule length effect”). This violates the principles of reward maximization and invariance and cannot be predicted by the standard methods of Reinforcement Learning, such as the method of temporal differences. We develop a heuristic model that accounts for all of the properties of the behavior in the reinforcement schedule task but whose predictions are not different from those of the standard temporal difference model in choice tasks. In the modification of temporal difference learning introduced here, the effect of schedule length emerges spontaneously from the sensitivity to the immediately preceding trial. We also introduce a policy for general Markov Decision Processes, where the decision made at each node is conditioned on the motivation to perform an instrumental action, and show that the application of our model to the reinforcement schedule task and the choice task are special cases of this general theoretical framework. Within this framework, Reinforcement Learning can approach contextual learning with the mixture of empirical findings and principled assumptions that seem to coexist in the best descriptions of animal behavior. As examples, we discuss two phenomena observed in humans that often derive from the violation of the principle of invariance: “framing,” wherein equivalent options are treated differently depending on the context in which they are presented, and the “sunk cost” effect, the greater tendency to continue an endeavor once an investment in money, effort, or time has been made. The schedule length effect might be a manifestation of these phenomena in monkeys

The human challenge in understanding animal cognition

Animal cognition studies have progressively moved themselves to an impasse due to an overemphasis on controlled experiments on captive animals that are completely detached from species-typical socio-ecologies. If progresses in studies on wild-living animals have provided a wealth of detailed observations on sophisticated cognitive achievements, captive experimental studies have concentrated on the “failure” of nonhuman species to demonstrate so-called uniquely human cognitive skills. In the present chapter, I stress the need to better understand what “cognition” is and to perform valid comparisons on chimpanzees if we want to understand the evolution of human cognitive abilities. Cognition is not just an innate property of a species, but an adaptation of individuals to their living conditions. As such, cognitive studies need ecological validity to explore the adaptations to the environments typical to the studied species. New understanding about brain plasticity and the effect of environmental enrichment in different species, including humans, confirm the importance of environment on the development of cognitive abilities. This invalidates the assumption of most experimental captive studies that one can generalize from such atypical conditions to the whole of the species. Furthermore, observations on wild chimpanzees stress the importance of population differences, thereby illustrating how cognition develops over the lifespan as individuals solve the daily challenges faced in their social and physical environment. Combining the information about brain plasticity, environmental validity, and population differences will permit cognitive studies to progress and finally contribute to our understanding of the evolution of human and human-like cognitive abilities

Diet and environment 1.2 million years ago revealed through analysis of dental calculus from Europe’s oldest hominin at Sima del Elefante, Spain

Sima del Elefante, Atapuerca, Spain contains one of the earliest hominin fragments yet known in Europe, dating to 1.2 Ma. Dental calculus from a hominin molar was removed, degraded and analysed to recover entrapped remains. Evidence for plant use at this time is very limited and this study has revealed the earliest direct evidence for foods consumed in the genus Homo. This comprises starchy carbohydrates from two plants, including a species of grass from the Triticeae or Bromideae tribe, meat and plant fibres. All food was eaten raw, and there is no evidence for processing of the starch granules which are intact and undamaged. Additional biographical detail includes fragments of non-edible wood found adjacent to an interproximal groove suggesting oral hygiene activities, while plant fibres may be linked to raw material processing. Environmental evidence comprises spores, insect fragments and conifer pollen grains which are consistent with a forested environment

Chimpanzees strategically manipulate what others can see

Humans often strategically manipulate the informational access of others to their own advantage. Although chimpanzees know what others can and cannot see, it is unclear whether they can strategically manipulate others’ visual access. In this study, chimpanzees were given the opportunity to save food for themselves by concealing it from a human competitor and also to get more food for themselves by revealing it to a human cooperator. When knowing that a competitor was approaching, chimpanzees kept more food hidden (left it covered) than when expecting a cooperator to approach. When the experimenter was already at the location of the hidden food, they actively revealed less food to the competitor than to the cooperator. They did not actively hide food (cover up food in the open) from the competitor, however. Chimpanzees thus strategically manipulated what another could see in order to maximize their payoffs and showed their ability to plan for future situations