Reinforcement Learning Explains Conditional Cooperation and Its Moody Cousin

Direct reciprocity, or repeated interaction, is a main mechanism to sustain cooperation under social dilemmas involving two individuals. For larger groups and networks, which are probably more relevant to understanding and engineering our society, experiments employing repeated multiplayer social dilemma games have suggested that humans often show conditional cooperation behavior and its moody variant. Mechanisms underlying these behaviors largely remain unclear. Here we provide a proximate account for this behavior by showing that individuals adopting a type of reinforcement learning, called aspiration learning, phenomenologically behave as conditional cooperator. By definition, individuals are satisfied if and only if the obtained payoff is larger than a fixed aspiration level. They reinforce actions that have resulted in satisfactory outcomes and anti-reinforce those yielding unsatisfactory outcomes. The results obtained in the present study are general in that they explain extant experimental results obtained for both so-called moody and non-moody conditional cooperation, prisoner's dilemma and public goods games, and well-mixed groups and networks. Different from the previous theory, individuals are assumed to have no access to information about what other individuals are doing such that they cannot explicitly use conditional cooperation rules. In this sense, myopic aspiration learning in which the unconditional propensity of cooperation is modulated in every discrete time step explains conditional behavior of humans. Aspiration learners showing (moody) conditional cooperation obeyed a noisy GRIM-like strategy. This is different from the Pavlov, a reinforcement learning strategy promoting mutual cooperation in two-player situations

Transient nature of cooperation by pay-it-forward reciprocity

Humans often forward kindness received from others to strangers, a phenomenon called the upstream or pay-it-forward indirect reciprocity. Some field observations and laboratory experiments found evidence of pay-it-forward reciprocity in which chains of cooperative acts persist in social dilemma situations. Theoretically, however, cooperation based on pay-it-forward reciprocity is not sustainable. We carried out laboratory experiments of a pay-it-forward indirect reciprocity game (i.e., chained gift-giving game) on a large scale in terms of group size and time. We found that cooperation consistent with pay-it-forward reciprocity occurred only in a first few decisions per participant and that cooperation originated from inherent pro-sociality of individuals. In contrast, the same groups of participants showed persisting chains of cooperation in a different indirect reciprocity game in which participants earned reputation by cooperating. Our experimental results suggest that pay-it-forward reciprocity is transient and disappears when a person makes decisions repeatedly, whereas the reputation-based reciprocity is stable in the same situation

Reinforcement learning accounts for moody conditional cooperation behavior:experimental results

In social dilemma games, human participants often show conditional cooperation (CC) behavior or its variant called moody conditional cooperation (MCC), with which they basically tend to cooperate when many other peers have previously cooperated. Recent computational studies showed that CC and MCC behavioral patterns could be explained by reinforcement learning. In the present study, we use a repeated multiplayer prisoner’s dilemma game and the repeated public goods game played by human participants to examine whether MCC is observed across different types of game and the possibility that reinforcement learning explains observed behavior. We observed MCC behavior in both games, but the MCC that we observed was different from that observed in the past experiments. In the present study, whether or not a focal participant cooperated previously affected the overall level of cooperation, instead of changing the tendency of cooperation in response to cooperation of other participants in the previous time step. We found that, across different conditions, reinforcement learning models were approximately as accurate as a MCC model in describing the experimental results. Consistent with the previous computational studies, the present results suggest that reinforcement learning may be a major proximate mechanism governing MCC behavior

Real-time nanodiamond thermometry probing in vivo thermogenic responses

蛍光ナノダイヤモンドを用いた量子温度計により動物個体の発熱を捉えることに成功. 京都大学プレスリリース. 2020-09-25.Get diamonds, take temperature. 京都大学プレスリリース. 2020-10-06.Real-time temperature monitoring inside living organisms provides a direct measure of their biological activities. However, it is challenging to reduce the size of biocompatible thermometers down to submicrometers, despite their potential applications for the thermal imaging of subtissue structures with single-cell resolution. Here, using quantum nanothermometers based on optically accessible electron spins in nanodiamonds, we demonstrate in vivo real-time temperature monitoring inside Caenorhabditis elegans worms. We developed a microscope system that integrates a quick-docking sample chamber, particle tracking, and an error correction filter for temperature monitoring of mobile nanodiamonds inside live adult worms with a precision of ±0.22°C. With this system, we determined temperature increases based on the worms’ thermogenic responses during the chemical stimuli of mitochondrial uncouplers. Our technique demonstrates the submicrometer localization of temperature information in living animals and direct identification of their pharmacological thermogenesis, which may allow for quantification of their biological activities based on temperature

Insights into Land Plant Evolution Garnered from the Marchantia polymorpha Genome.

The evolution of land flora transformed the terrestrial environment. Land plants evolved from an ancestral charophycean alga from which they inherited developmental, biochemical, and cell biological attributes. Additional biochemical and physiological adaptations to land, and a life cycle with an alternation between multicellular haploid and diploid generations that facilitated efficient dispersal of desiccation tolerant spores, evolved in the ancestral land plant. We analyzed the genome of the liverwort Marchantia polymorpha, a member of a basal land plant lineage. Relative to charophycean algae, land plant genomes are characterized by genes encoding novel biochemical pathways, new phytohormone signaling pathways (notably auxin), expanded repertoires of signaling pathways, and increased diversity in some transcription factor families. Compared with other sequenced land plants, M. polymorpha exhibits low genetic redundancy in most regulatory pathways, with this portion of its genome resembling that predicted for the ancestral land plant. PAPERCLIP

Cultural Differences in Strength of Conformity Explained Through Pathogen Stress : A Statistical Test Using Hierarchical Bayesian Estimation

The severity of the environment has been found to have played a selective pressure in the development of human behavior and psychology, and the historical prevalence of pathogens relate to cultural differences in group-oriented psychological mechanisms, such as collectivism and conformity to the in-group. However, previous studies have also proposed that the effectiveness of institutions, rather than pathogen stress, can account for regional variation in group-oriented psychological mechanisms. Moreover, previous studies using nations as units of analysis may have suffered from a problem of statistical non-independence, namely, Galton's problem. The present study tested whether or not regional variation in pathogen stress, rather than government effectiveness, affected collectivism and conformity to social norms by adjusting the effect of global regions using hierarchical Bayesian estimation. We found that the overall effect of pathogen stress remained significant in only one out of the four indices of the regional variability of conformity, and the effects of the government effectiveness also disappeared. Instead, we found that significant effects of both pathogen stress and government effectiveness in specific regions of the world, but these effects were not stable across the measurements. These results indicate that both the effects of pathogen stress and government effectiveness need further reevaluation

腎盂原発腺癌の1例

28歳の女性で腺癌細胞を腸水中に含む腎盂原発腺癌の1例を経験した.原発巣に対し根治的腎摘出術, 残存尿管摘出, 膀胱部分切除術を施行した.補助療法としてシスプラチン, アドリアマイシン, サイクロフォスフママイド(CAP)を用いた多剤併用化学療法を行った.腎摘出後, 約3年経過しているが再発の兆候はなく, CAP療法が有効であったと思われた.A 28-year-old female had adenocarcinoma arising from the renal pelvis with ascites containing adenocarcinoma cells. The primary site was treated with radical nephrectomy, resection of the remnant ureter with a bladder cuff. Combination chemotherapy with cisplatin, doxorubicin, and cyclophosphamide (CAP) was performed as an adjuvant therapy. Approximately 3 years after the nephrectomy, she is currently alive with no clinical evidence of recurrence. CAP seems to have been effective in the treatment of the disease

ループス膀胱炎の1例

42歳女。SLEで経過を観察中に食思不振となり当院内科を受診。精査中に水腎症を発見され泌尿器科紹介となった。IVPで両側水腎水尿管を認めた。CTでは腹水貯留と粘膜と漿膜に造影効果を有する腸管壁および膀胱壁の肥厚を認めた。ラッシクス負荷レノグラムは両側腎とも正常であった。膀胱生検では間質の浮腫, 繊維化と炎症細胞浸潤を認めた。膀胱内圧測定でほぼ正常であった。ループス膀胱炎の診断でプレドニソロン40mgを開始した。CT上, 膀胱壁, 腸管壁の肥厚も消失し貧血も改善した。左水腎症は改善したが右尿管狭窄が悪化したため尿管ステントを挿入しステロイド療法を継続しながら経過を観察しているA 42-year-old female with a history of systemic lupus erythematosus (SLE) visited the Department of Internal Medicine at our hospital complaining of anorexia. Hydronephrosis was diagnosed, and she was referred to our department for further evaluation. Intravenous pyelography (IVP) revealed bilateral hydronephrosis and hydroureters. Computerized tomography (CT) showed retention of ascitic fluid and thickening of the intestinal and bladder walls with contrast effects in the serosa and mucosa. Lasix-loaded renography showed that both kidneys were normal. Examination of a biopsy specimen revealed interstitial edema, fibrosis, and infiltration of inflammatory cells. Cystometry showed normal intravesical pressure. A diagnosis of lupus cystitis was made and administration of 40mg of prednisolone was started. The anemia was ameliorated, and thickening of the bladder and intestinal walls was no longer seen on CT. Hydronephrosis of the left kidney was less prominent. However, aggravation of the right ureteral stenosis was noted, and a stent was inserted in the stenosed ureter. Stenosis was successfully relieved by the stent, and internal drainage was established. The patient is still receiving steroids and is being followed at our department. Twenty-two reported cases of lupus cystitis in Japan are reviewed

ラットBBN膀胱腫瘍発生に対する抗腫瘍剤の効果について

F334ラット115匹に0.05%BBNを20週間経口投与し, 同時にADM, CDDP, 生食水の膀胱内注入療法を5週毎に5回連続群(type A, 計20回)週1回群(type B, 計20回)に分け施行した.コントロール群はBBN投与のみとした.40週後に屠殺し検討した.Type Aの生食群はコントロール群におよびtype Bの生食群に比較し有意に膀胱腫瘍重量が増加していた.Type AのADM, CDDP, 膀注群はtype Bの同群より膀胱腫瘍重量が増加していた.ADM, CDDP, 群は2つのtypeでコントロール群に比較し膀胱腫瘍重量の減少した例と増加した例が見られた.組織学的に各群間に差はなかったが, type AのADM群にhigh stage例が多く見られたWe examined the effects of intravesical instillation of adriamycin (ADM) and cis-diammine-dichloroplatinum (CDDP) on the development of bladder tumors induced by 0.05% N-butyl-N-(4-hydroxybutyl)nitrosamine in 115 rats. Intravesical instillation was performed in two ways; continuous administration (5 times a week for 5 weeks) and intermittent administration (once a week). Group 1 (control group) did not receive intravesical instillation. Group 2 I(ADM), group 3 (CDDP) and group 4 (physiological saline) were continuous type. Group 5 (ADM), group 6 (CDDP) and group 7 (physiological saline) received intermittent administration. Bladder weight was significantly higher in group 4 than in groups 1 or 7, and that in groups 2 and 3 than that in group 5 and 6. In groups 2, 3, 5 and 6 bladder weight was almost normal or higher than in the control group, and in group 3 histologically cancer was not seen in one rat. Physiological saline had promoting activity, and ADM and CDDP had both inhibitory and promoting activities. Also, intravesical instillation itself was suggested to promote tumor development under carcinogenic circumstances. We conclude that intermittent intravesical instillation should be performed to inhibit tumor recurrence and intravesical instillation therapy should not be performed clinically for a long period