Strategic farsighted learning in competitive multi-agent games

We describe a generalized Q-learning type algorithm for reinforcement learning in competitive multi-agent games. We make the observation that in a competitive setting with adaptive agents an agent's actions will (likely) result in changes in the opponents policies. In addition to accounting for the estimated policies of the opponents, our algorithm also adjusts these future opponent policies by incorporating estimates of how the opponents change their policy as a reaction to ones own actions. We present results showing that agents that learn with this algorithm can successfully achieve high reward in competitive multi-agent games where myopic self-interested behavior conflicts with the long term individual interests of the players.We show that this approach successfully scales for multi-agent games of various sizes, in particular to the social dilemma type problems: from the small iterated Prisoner's Dilemma, to larger settings akin to Harding's Tragedy of the Commons. Thus, our multi-agent reinforcement algorithm is foresighted enough to correctly anticipate future rewards in the important problem class of social dilemmas, without having to resort to negotiation-like protocols or precoded strategies

Strategic farsighted learning in competitive multi-agent games

We describe a generalized Q-learning type algorithm for reinforcement learning in competitive multi-agent games. We make the observation that in a competitive setting with adaptive agents an agent's actions will (likely) result in changes in the opponents policies. In addition to accounting for the estimated policies of the opponents, our algorithm also adjusts these future opponent policies by incorporating estimates of how the opponents change their policy as a reaction to ones own actions. We present results showing that agents that learn with this algorithm can successfully achieve high reward in competitive multi-agent games where myopic self-interested behavior conflicts with the long term individual interests of the players.We show that this approach successfully scales for multi-agent games of various sizes, in particular to the social dilemma type problems: from the small iterated Prisoner's Dilemma, to larger settings akin to Harding's Tragedy of the Commons. Thus, our multi-agent reinforcement algorithm is foresighted enough to correctly anticipate future rewards in the important problem class of social dilemmas, without having to resort to negotiation-like protocols or precoded strategies

Learning from induced changes in opponent (re)actions in multi-agent games

Multi-agent learning is a growing area of research. An important topic is to formulate how an agent can learn a good policy in the face of adaptive, competitive opponents. Most research has focused on extensions of single agent learning techniques originally designed for agents in more static environments. These techniques however fail to incorporate a notion of the effect of own previous actions on the development of the policy of the other agents in the system. We argue that incorporation of this property is beneficial in competitive settings. In this paper, we present a novel algorithm to capture this notion, and present experimental results to validate our claim

Foresighted policy gradient reinforcement learning: solving large-scale social dilemmas with rational altruistic punishment

Many important and difficult problems can be modeled as “social dilemmas”, like Hardin's Tragedy of the Commons or the classic iterated Prisoner's Dilemma. It is well known that in these problems, it can be rational for self-interested agents to promote and sustain cooperation by altruistically dispensing costly punishment to other agents, thus maximizing their own long-term reward. However, self-interested agents using most current multi-agent reinforcement learning algorithms will not sustain cooperation in social dilemmas: the algorithms do not sufficiently capture the consequences on the agent's reward of the interactions that it has with other agents. Recent more foresighted algorithms specifically account for such expected consequences, and have been shown to work well for the small-scale Prisoner's Dilemma. However, this approach quickly becomes intractable for larger social dilemmas. Here, we advance on this work and develop a “teach/learn” stateless foresighted policy gradient reinforcement learning algorithm that applies to Social Dilemma's with negative, unilateral side-payments, in the from of costly punishment. In this setting, the algorithm allows agents to learn the most rewarding actions to take with respect to both the dilemma (Cooperate/Defect) and the “teaching” of other agent's behavior through the dispensing of punishment. Unlike other algorithms, we show that this approach scales well to large settings like the Tragedy of the Commons. We show for a variety of settings that large groups of self-interested agents using this algorithm will robustly find and sustain cooperation in social dilemmas where adaptive agents can punish the behavior of other similarly adaptive agents

A decommitment strategy in a competitive multi-agent transportation setting

Decommitment is the action of foregoing of a contract for another (superior) offer. It has been shown that, using decommitment, agents can reach higher utility levels in case of negotiations with uncertainty about future prospects. In this paper, we study the decommitment concept for the novel setting of a large-scale logistics setting with multiple, competing companies. Orders for transportation of loads are acquired by agents of the (competing) companies by bidding in online auctions. We find significant increases in profit when the agents can decommit and postpone the transportation of a load to a more suitable time. Furthermore, we analyze the circumstances for which decommitment has a positive impact if agents are capable of handling multiple contracts simultaneously

Occupational exposure to gases/fumes and mineral dust affect DNA methylation levels of genes regulating expression

Many workers are daily exposed to occupational agents like gases/fumes, mineral dust or biological dust, which could induce adverse health effects. Epigenetic mechanisms, such as DNA methylation, have been suggested to play a role. We therefore aimed to identify differentially methylated regions (DMRs) upon occupational exposures in never-smokers and investigated if these DMRs associated with gene expression levels. To determine the effects of occupational exposures independent of smoking, 903 never-smokers of the LifeLines cohort study were included. We performed three genome-wide methylation analyses (Illumina 450 K), one per occupational exposure being gases/fumes, mineral dust and biological dust, using robust linear regression adjusted for appropriate confounders. DMRs were identified using comb-p in Python. Results were validated in the Rotterdam Study (233 never-smokers) and methylation-expression associations were assessed using Biobank-based Integrative Omics Study data (n = 2802). Of the total 21 significant DMRs, 14 DMRs were associated with gases/fumes and 7 with mineral dust. Three of these DMRs were associated with both exposures (RPLP1 and LINC02169 (2x)) and 11 DMRs were located within transcript start sites of gene expression regulating genes. We replicated two DMRs with gases/fumes (VTRNA2-1 and GNAS) and one with mineral dust (CCDC144NL). In addition, nine gases/fumes DMRs and six mineral dust DMRs significantly associated with gene expression levels. Our data suggest that occupational exposures may induce differential methylation of gene expression regulating genes and thereby may induce adverse health effects. Given the millions of workers that are exposed daily to occupational exposures, further studies on this epigenetic mechanism and health outcomes are warranted

Highlights of the São Paulo ISEV workshop on extracellular vesicles in cross-kingdom communication

In the past years, extracellular vesicles (EVs) have become an important field of research since EVs have been found to play a central role in biological processes. In pathogens, EVs are involved in several events during the host–pathogen interaction, including invasion, immunomodulation, and pathology as well as parasite–parasite communication. In this report, we summarised the role of EVs in infections caused by viruses, bacteria, fungi, protozoa, and helminths based on the talks and discussions carried out during the International Society for Extracellular Vesicles (ISEV) workshop held in São Paulo (November, 2016), Brazil, entitled Cross-organism Communication by Extracellular Vesicles: Hosts, Microbes and Parasites. © 2017 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.11Ysciescopu

Comprehensive Gene-Expression Survey Identifies Wif1 as a Modulator of Cardiomyocyte Differentiation

During chicken cardiac development the proepicardium (PE) forms the epicardium (Epi), which contributes to several non-myocardial lineages within the heart. In contrast to Epi-explant cultures, PE explants can differentiate into a cardiomyocyte phenotype. By temporal microarray expression profiles of PE-explant cultures and maturing Epi cells, we identified genes specifically associated with differentiation towards either of these lineages and genes that are associated with the Epi-lineage restriction. We found a central role for Wnt signaling in the determination of the different cell lineages. Immunofluorescent staining after recombinant-protein incubation in PE-explant cultures indicated that the early upregulated Wnt inhibitory factor-1 (Wif1), stimulates cardiomyocyte differentiation in a similar manner as Wnt stimulation. Concordingly, in the mouse pluripotent embryogenic carcinoma cell line p19cl6, early and late Wif1 exposure enhances and attenuates differentiation, respectively. In ovo exposure of the HH12 chicken embryonic heart to Wif1 increases the Tbx18-positive cardiac progenitor pool. These data indicate that Wif1 enhances cardiomyogenesis

Prognostic factors in left-sided endocarditis: results from the andalusian multicenter cohort

<p>Abstract</p> <p>Background</p> <p>Despite medical advances, mortality in infective endocarditis (IE) is still very high. Previous studies on prognosis in IE have observed conflicting results. The aim of this study was to identify predictors of in-hospital mortality in a large multicenter cohort of left-sided IE.</p> <p>Methods</p> <p>An observational multicenter study was conducted from January 1984 to December 2006 in seven hospitals in Andalusia, Spain. Seven hundred and five left-side IE patients were included. The main outcome measure was in-hospital mortality. Several prognostic factors were analysed by univariate tests and then by multilogistic regression model.</p> <p>Results</p> <p>The overall mortality was 29.5% (25.5% from 1984 to 1995 and 31.9% from 1996 to 2006; Odds Ratio 1.25; 95% Confidence Interval: 0.97-1.60; p = 0.07). In univariate analysis, age, comorbidity, especially chronic liver disease, prosthetic valve, virulent microorganism such as <it>Staphylococcus aureus</it>, <it>Streptococcus agalactiae </it>and fungi, and complications (septic shock, severe heart failure, renal insufficiency, neurologic manifestations and perivalvular extension) were related with higher mortality. Independent factors for mortality in multivariate analysis were: Charlson comorbidity score (OR: 1.2; 95% CI: 1.1-1.3), prosthetic endocarditis (OR: 1.9; CI: 1.2-3.1), <it>Staphylococcus aureus </it>aetiology (OR: 2.1; CI: 1.3-3.5), severe heart failure (OR: 5.4; CI: 3.3-8.8), neurologic manifestations (OR: 1.9; CI: 1.2-2.9), septic shock (OR: 4.2; CI: 2.3-7.7), perivalvular extension (OR: 2.4; CI: 1.3-4.5) and acute renal failure (OR: 1.69; CI: 1.0-2.6). Conversely, <it>Streptococcus viridans </it>group etiology (OR: 0.4; CI: 0.2-0.7) and surgical treatment (OR: 0.5; CI: 0.3-0.8) were protective factors.</p> <p>Conclusions</p> <p>Several characteristics of left-sided endocarditis enable selection of a patient group at higher risk of mortality. This group may benefit from more specialised attention in referral centers and should help to identify those patients who might benefit from more aggressive diagnostic and/or therapeutic procedures.</p

Sox4 mediates Tbx3 transcriptional regulation of the gap junction protein Cx43

Tbx3, a T-box transcription factor, regulates key steps in development of the heart and other organ systems. Here, we identify Sox4 as an interacting partner of Tbx3. Pull-down and nuclear retention assays verify this interaction and in situ hybridization reveals Tbx3 and Sox4 to co-localize extensively in the embryo including the atrioventricular and outflow tract cushion mesenchyme and a small area of interventricular myocardium. Tbx3, SOX4, and SOX2 ChIP data, identify a region in intron 1 of Gja1 bound by all tree proteins and subsequent ChIP experiments verify that this sequence is bound, in vivo, in the developing heart. In a luciferase reporter assay, this element displays a synergistic antagonistic response to co-transfection of Tbx3 and Sox4 and in vivo, in zebrafish, drives expression of a reporter in the heart, confirming its function as a cardiac enhancer. Mechanistically, we postulate that Sox4 is a mediator of Tbx3 transcriptional activity