Reinforcement Learning Explains Conditional Cooperation and Its Moody Cousin

Direct reciprocity, or repeated interaction, is a main mechanism to sustain cooperation under social dilemmas involving two individuals. For larger groups and networks, which are probably more relevant to understanding and engineering our society, experiments employing repeated multiplayer social dilemma games have suggested that humans often show conditional cooperation behavior and its moody variant. Mechanisms underlying these behaviors largely remain unclear. Here we provide a proximate account for this behavior by showing that individuals adopting a type of reinforcement learning, called aspiration learning, phenomenologically behave as conditional cooperator. By definition, individuals are satisfied if and only if the obtained payoff is larger than a fixed aspiration level. They reinforce actions that have resulted in satisfactory outcomes and anti-reinforce those yielding unsatisfactory outcomes. The results obtained in the present study are general in that they explain extant experimental results obtained for both so-called moody and non-moody conditional cooperation, prisoner's dilemma and public goods games, and well-mixed groups and networks. Different from the previous theory, individuals are assumed to have no access to information about what other individuals are doing such that they cannot explicitly use conditional cooperation rules. In this sense, myopic aspiration learning in which the unconditional propensity of cooperation is modulated in every discrete time step explains conditional behavior of humans. Aspiration learners showing (moody) conditional cooperation obeyed a noisy GRIM-like strategy. This is different from the Pavlov, a reinforcement learning strategy promoting mutual cooperation in two-player situations

Transient nature of cooperation by pay-it-forward reciprocity

Humans often forward kindness received from others to strangers, a phenomenon called the upstream or pay-it-forward indirect reciprocity. Some field observations and laboratory experiments found evidence of pay-it-forward reciprocity in which chains of cooperative acts persist in social dilemma situations. Theoretically, however, cooperation based on pay-it-forward reciprocity is not sustainable. We carried out laboratory experiments of a pay-it-forward indirect reciprocity game (i.e., chained gift-giving game) on a large scale in terms of group size and time. We found that cooperation consistent with pay-it-forward reciprocity occurred only in a first few decisions per participant and that cooperation originated from inherent pro-sociality of individuals. In contrast, the same groups of participants showed persisting chains of cooperation in a different indirect reciprocity game in which participants earned reputation by cooperating. Our experimental results suggest that pay-it-forward reciprocity is transient and disappears when a person makes decisions repeatedly, whereas the reputation-based reciprocity is stable in the same situation

Histological development of stapes footplate in human embryos.

Normal development of the human stapes footplate was investigated in serial sections by light microscopy. Materials were obtained from 35 Japanese embryos from the 6th to 32nd week of embryonal age. Eighteen embryos up to 16 weeks of age (3.5mm to 105mm in crown-rump length) were examined, focusing particularly on the lamina stapedialis of the otic capsule. The present study showed that primordial formation of the lamina stapedialis appeared in 16mm embryo and that the lamina was completely formed and fused to the base of the annular stapes in a 35mm embryo. In a 50mm embryo, the adult form of stapes was found with a rim and annular ligament. The results, therefore, seemed to essentially agree with the theory of dual origin and development of the footplate proposed by Cauldwell and Anson, and teratogenic agents might affect any stage of the process producing anomalies,</p

Reinforcement learning accounts for moody conditional cooperation behavior:experimental results

In social dilemma games, human participants often show conditional cooperation (CC) behavior or its variant called moody conditional cooperation (MCC), with which they basically tend to cooperate when many other peers have previously cooperated. Recent computational studies showed that CC and MCC behavioral patterns could be explained by reinforcement learning. In the present study, we use a repeated multiplayer prisoner’s dilemma game and the repeated public goods game played by human participants to examine whether MCC is observed across different types of game and the possibility that reinforcement learning explains observed behavior. We observed MCC behavior in both games, but the MCC that we observed was different from that observed in the past experiments. In the present study, whether or not a focal participant cooperated previously affected the overall level of cooperation, instead of changing the tendency of cooperation in response to cooperation of other participants in the previous time step. We found that, across different conditions, reinforcement learning models were approximately as accurate as a MCC model in describing the experimental results. Consistent with the previous computational studies, the present results suggest that reinforcement learning may be a major proximate mechanism governing MCC behavior

Isorhamnetin activates lysosomes in J774.1 macrophages.

Lysosome is the principal organelle for the ultimate degradation of cellular macromolecules, which are delivered through endocytosis, phagocytosis and autophagy. The lysosomal functions have been found to be impaired by fatty foods and aging, and more importantly, the lysosomal dysfunction in macrophages has been reported as a risk of atherosclerosis development. In this study, we searched for dietary polyphenols which possess the activity for enhancing the lysosomal degradation in J774.1, a murine macrophage-like cell line. Screening test utilizing DQ-BSA digestion identified isorhamnetin (3’-O-methylquercetin) as an active compound. Interestingly, structural comparison to inactive flavonols revealed that the chemical structure of the B-ring moiety in isorhamnetin is the primary determinant of its lysosome-enhancing activity. Unexpectedly isorhamnetin failed to inhibit mTORC1-TFEB signaling, a master regulator of lysosomal biogenesis and function. Our data suggested that the other molecular mechanism might be critical for the regulation of lysosomes in macrophages

Cellular biology of cryopreserved allograft valves

Although analyzing the precise mechanisms of cryopreserved allograft valve failure may be difficult due to a number of crucial reasons and the interrelationships between the overlapping mechanisms, there is some evidence that cryopreservation is currently the best method of storing allograft valves. The present review shows the basic cellular biology of cryopreserved allograft valves for long-term durability, particularly relevant to allograft valve cellular viability, the immune response mainly caused by viable donor cells, and the preservation and regeneration of the intrinsic extracellular matrix. The present findings are as follows. First, cryopreservation produces serious damage to cytosolic and mitochondrial functions of both endothelial cells and fibroblasts, which may cause valve failure after implantation. Second, although the collagen synthesis of cryopreserved valves was relatively maintained, total protein synthesis was highly diminished and the collagenolytic ability was activated immediately after the thawing process. These findings imply that the cryopreservation itself may cause the collagen metabolism to become degradable, which will lead to valve failure. Further examination of collagen metabolism and modulation of the collagenolytic activity will be necessary to improve the tissue preservation for improved clinical use

Alterations in erythrocyte membrane lipid and its fragility in a patient with familial lecithin : cholesterol acyltrasferase (LCAT) deficiency

Lecithin : cholesterol acyltrasferase (LCAT) plays a key role in the cholesterol metabolism-mediated esterification of free cholesterol into the cholesterol ester in normal plasma. Familial LCAT deficiency is frequently associated with anemia. Using biochemical and physiological techniques, the erythrocytes of this patient were investigated to gain an insight into the relationship between the abnormalities of lipid metabolism and erythrocyte membrane fragility. Abnormal erythrocytes, so-called Target cells and/or Knizocytes, were observed at 20% in our patient’s erythrocytes. Moreover, the mean corpuscular volume of the patient’s cells was 7% greater than that of a normal individual. In the membrane lipids of the patient’s erythrocytes, cholesterol and phosphatidylcholine increased, and phosphatidylethanolamine decreased. The electron spin resonance technique with a fatty acid spin probe showed that the membrane fluidity was more elevated than that of normal cells in spite of the increase in cholesterol content and the cholesterol/ phospholipid ratio of the membrane of patient’s erythrocytes. The patient’s abnormally shaped erythrocytes were less deformed than those of the normal individual under high shear stress. The partial depletion of membrane cholesterol from the patient’s erythrocytes was demonstrated by incubation with normal plasma with LCAT activity. The increment of transformed erythrocytes during the incubation could be prevented by cholesterol depletion from the patient’s erythrocyte membrane. These findings indicate that normochromic anemia of the patient might be caused by erythrocyte fragility resulting from decreased deformity and/or abnormal shape of the cells due to abnormal lipid composition in the membrane