Reinforcement Learning Explains Conditional Cooperation and Its Moody Cousin

Direct reciprocity, or repeated interaction, is a main mechanism to sustain cooperation under social dilemmas involving two individuals. For larger groups and networks, which are probably more relevant to understanding and engineering our society, experiments employing repeated multiplayer social dilemma games have suggested that humans often show conditional cooperation behavior and its moody variant. Mechanisms underlying these behaviors largely remain unclear. Here we provide a proximate account for this behavior by showing that individuals adopting a type of reinforcement learning, called aspiration learning, phenomenologically behave as conditional cooperator. By definition, individuals are satisfied if and only if the obtained payoff is larger than a fixed aspiration level. They reinforce actions that have resulted in satisfactory outcomes and anti-reinforce those yielding unsatisfactory outcomes. The results obtained in the present study are general in that they explain extant experimental results obtained for both so-called moody and non-moody conditional cooperation, prisoner's dilemma and public goods games, and well-mixed groups and networks. Different from the previous theory, individuals are assumed to have no access to information about what other individuals are doing such that they cannot explicitly use conditional cooperation rules. In this sense, myopic aspiration learning in which the unconditional propensity of cooperation is modulated in every discrete time step explains conditional behavior of humans. Aspiration learners showing (moody) conditional cooperation obeyed a noisy GRIM-like strategy. This is different from the Pavlov, a reinforcement learning strategy promoting mutual cooperation in two-player situations

Transient nature of cooperation by pay-it-forward reciprocity

Humans often forward kindness received from others to strangers, a phenomenon called the upstream or pay-it-forward indirect reciprocity. Some field observations and laboratory experiments found evidence of pay-it-forward reciprocity in which chains of cooperative acts persist in social dilemma situations. Theoretically, however, cooperation based on pay-it-forward reciprocity is not sustainable. We carried out laboratory experiments of a pay-it-forward indirect reciprocity game (i.e., chained gift-giving game) on a large scale in terms of group size and time. We found that cooperation consistent with pay-it-forward reciprocity occurred only in a first few decisions per participant and that cooperation originated from inherent pro-sociality of individuals. In contrast, the same groups of participants showed persisting chains of cooperation in a different indirect reciprocity game in which participants earned reputation by cooperating. Our experimental results suggest that pay-it-forward reciprocity is transient and disappears when a person makes decisions repeatedly, whereas the reputation-based reciprocity is stable in the same situation

Reinforcement learning accounts for moody conditional cooperation behavior:experimental results

In social dilemma games, human participants often show conditional cooperation (CC) behavior or its variant called moody conditional cooperation (MCC), with which they basically tend to cooperate when many other peers have previously cooperated. Recent computational studies showed that CC and MCC behavioral patterns could be explained by reinforcement learning. In the present study, we use a repeated multiplayer prisoner’s dilemma game and the repeated public goods game played by human participants to examine whether MCC is observed across different types of game and the possibility that reinforcement learning explains observed behavior. We observed MCC behavior in both games, but the MCC that we observed was different from that observed in the past experiments. In the present study, whether or not a focal participant cooperated previously affected the overall level of cooperation, instead of changing the tendency of cooperation in response to cooperation of other participants in the previous time step. We found that, across different conditions, reinforcement learning models were approximately as accurate as a MCC model in describing the experimental results. Consistent with the previous computational studies, the present results suggest that reinforcement learning may be a major proximate mechanism governing MCC behavior

Function of the web box as an anti-predator barrier in the spider mite, Schizotetranychus recki

Spider mites inhabiting Sasa bamboo show considerable variation in traits believed to be the result of coevolution between predator and prey. In Schizotetranychus recki Ehara inhabiting the hairy leaves of a dwarf bamboo, Sasa senanensis, all quiescent stages, including eggs, appear within web boxes in the leaf hairs of their host plant, and this habit is thought to be a trait involved in predator avoidance. To test this hypothesis, the survival rates of S. recki eggs inside a web box and those freed of a web box were assessed in relation to six predator species that co-occur with the spider mite in the field. The results clearly show that the webbing behavior (web box) of S. recki has a function in avoiding five predator species. However, one predator species, Agistemus summersi Ehara, preyed more on eggs protected by a web box. This suggests that this predator species has a special trait to overcome the nest barriers

Cultural Differences in Strength of Conformity Explained Through Pathogen Stress : A Statistical Test Using Hierarchical Bayesian Estimation

The severity of the environment has been found to have played a selective pressure in the development of human behavior and psychology, and the historical prevalence of pathogens relate to cultural differences in group-oriented psychological mechanisms, such as collectivism and conformity to the in-group. However, previous studies have also proposed that the effectiveness of institutions, rather than pathogen stress, can account for regional variation in group-oriented psychological mechanisms. Moreover, previous studies using nations as units of analysis may have suffered from a problem of statistical non-independence, namely, Galton's problem. The present study tested whether or not regional variation in pathogen stress, rather than government effectiveness, affected collectivism and conformity to social norms by adjusting the effect of global regions using hierarchical Bayesian estimation. We found that the overall effect of pathogen stress remained significant in only one out of the four indices of the regional variability of conformity, and the effects of the government effectiveness also disappeared. Instead, we found that significant effects of both pathogen stress and government effectiveness in specific regions of the world, but these effects were not stable across the measurements. These results indicate that both the effects of pathogen stress and government effectiveness need further reevaluation

Conjecturing harmful intent and preemptive strike in paranoia

Paranoia depicts a belief of others having harmful intent. Research using economic games has exhibited the correlation between paranoia and the propensity to characterize ambiguous intentions as harmful. Using a non-clinical sample recruited online from the United States (N = 290), we examined whether paranoid thoughts influence aggressive behavior against the subjective perception of harmful intent. We conducted a preemptive strike game wherein aggressive behavior was assumed to be guided by the fear of an opponent. The outcomes indicate that 1) individuals with high paranoia assume harmful intent of an opponent more than those with low paranoia, 2) conjecturing an opponent’s harmful intent predicted an increase in the probability of a preemptive strike, 3) paranoia did not have a statistically significant effect on encouraging a preemptive strike. Additionally, the exploratory analysis revealed that paranoia was related to participant’s aggressiveness and with suppositions of other’s self-interests and competitiveness. This study presents empirical evidence that paranoia is related to perception of social threats in an uncertain situation. We discuss the possibility that paranoid ideation can promote or inhibit a preemptive strike