Assessing the Potential of Classical Q-learning in General Game Playing

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee $\&$ Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex)\footnote{source code: https://github.com/wh1992v/ggp-rl}, to allow comparison to Banerjee et al.. We find that Q-learning converges to a high win rate in GGP. For the $\epsilon$-greedy strategy, we propose a first enhancement, the dynamic $\epsilon$ algorithm. In addition, inspired by (Gelly $\&$ Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.Comment: arXiv admin note: substantial text overlap with arXiv:1802.0594

Agent cognition through micro-simulations: Adaptive and tunable intelligence with NetLogo LevelSpace

We present a method of endowing agents in an agent-based model (ABM) with sophisticated cognitive capabilities and a naturally tunable level of intelligence. Often, ABMs use random behavior or greedy algorithms for maximizing objectives (such as a predator always chasing after the closest prey). However, random behavior is too simplistic in many circumstances and greedy algorithms, as well as classic AI planning techniques, can be brittle in the context of the unpredictable and emergent situations in which agents may find themselves. Our method, called agent-centric Monte Carlo cognition (ACMCC), centers around using a separate agent-based model to represent the agents' cognition. This model is then used by the agents in the primary model to predict the outcomes of their actions, and thus guide their behavior. To that end, we have implemented our method in the NetLogo agent-based modeling platform, using the recently released LevelSpace extension, which we developed to allow NetLogo models to interact with other NetLogo models. As an illustrative example, we extend the Wolf Sheep Predation model (included with NetLogo) by using ACMCC to guide animal behavior, and analyze the impact on agent performance and model dynamics. We find that ACMCC provides a reliable and understandable method of controlling agent intelligence, and has a large impact on agent performance and model dynamics even at low settings.Comment: Model source code available here: https://github.com/qiemem/Wolf-Sheep-Predation-Micro-Sims, In: Unifying Themes in Complex Systems IX. ICCS 2018. Springer Proceedings in Complexity. Springer, Cha

Preference-Based Monte Carlo Tree Search

Monte Carlo tree search (MCTS) is a popular choice for solving sequential anytime problems. However, it depends on a numeric feedback signal, which can be difficult to define. Real-time MCTS is a variant which may only rarely encounter states with an explicit, extrinsic reward. To deal with such cases, the experimenter has to supply an additional numeric feedback signal in the form of a heuristic, which intrinsically guides the agent. Recent work has shown evidence that in different areas the underlying structure is ordinal and not numerical. Hence erroneous and biased heuristics are inevitable, especially in such domains. In this paper, we propose a MCTS variant which only depends on qualitative feedback, and therefore opens up new applications for MCTS. We also find indications that translating absolute into ordinal feedback may be beneficial. Using a puzzle domain, we show that our preference-based MCTS variant, wich only receives qualitative feedback, is able to reach a performance level comparable to a regular MCTS baseline, which obtains quantitative feedback.Comment: To be publishe

Assessing the Potential of Classical Q-learning in General Game Playing

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee & Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex), to allow comparison to Banerjee et al. We find that Q-learning converges to a high win rate in GGP. For the ϵ" role="presentation" style="display: inline-table; line-height: normal; letter-spacing: normal; word-spacing: normal; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; position: relative;">ϵ-greedy strategy, we propose a first enhancement, the dynamic ϵ" role="presentation" style="display: inline-table; line-height: normal; letter-spacing: normal; word-spacing: normal; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; position: relative;">ϵ algorithm. In addition, inspired by (Gelly & Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.Computer Systems, Imagery and Medi

Telomerase activity in melanoma and non-melanoma skin cancer

Telomeres are specialized structures consisting of repeat arrays of TTAGGGn located at the ends of chromosomes. They are essential for chromosome stability and, in the majority of normal somatic cells, telomeres shorten with each cell division. Most immortalized cell lines and tumours reactivate telomerase to stabilize the shortening chromosomes. Telomerase activation is regarded as a central step in carcinogenesis and, here, we demonstrate telomerase activation in premalignant skin lesions and also in all forms of skin cancer. Telomerase activation in normal skin was a rare event, and among 16 samples of normal skin (one with a history of chronic sun exposure) 12.5% (2 out of 16) exhibited telomerase activity. One out of 16 (6.25%) benign proliferative lesions, including viral and seborrhoeic wart samples, had telomerase activity. In premalignant actinic keratoses and Bowen's disease, 42% (11 out of 26) of samples exhibited telomerase activity. In the basal cell carcinoma and cutaneous malignant melanoma (CMM) lesions, telomerase was activated in 77% (10 out of 13) and 69% (22 out of 32) respectively. However, only 25% (3 out of 12) of squamous cell carcinomas (SCC) had telomerase activity. With the exception of one SCC sample, telomerase activity in a positive control cell line derived from a fibrosarcoma (HT1080) was not inhibited when mixed with the telomerase-negative SCC or CMM extracts, indicating that, overall, Taq polymerase and telomerase inhibitors were not responsible for the negative results. Mean telomere hybridizing restriction fragment (TRF) analysis was performed in a number of telomerase-positive and -negative samples and, although a broad range of TRF sizes ranging from 3.6 to 17 kb was observed, a relationship between telomerase status and TRF size was not found

The ADAMTS (A Disintegrin and Metalloproteinase with Thrombospondin motifs) family

The ADAMTS (A Disintegrin and Metalloproteinase with Thrombospondin motifs) enzymes are secreted, multi-domain matrix-associated zinc metalloendopeptidases that have diverse roles in tissue morphogenesis and patho-physiological remodeling, in inflammation and in vascular biology. The human family includes 19 members that can be sub-grouped on the basis of their known substrates, namely the aggrecanases or proteoglycanases (ADAMTS1, 4, 5, 8, 9, 15 and 20), the procollagen N-propeptidases (ADAMTS2, 3 and 14), the cartilage oligomeric matrix protein-cleaving enzymes (ADAMTS7 and 12), the von-Willebrand Factor proteinase (ADAMTS13) and a group of orphan enzymes (ADAMTS6, 10, 16, 17, 18 and 19). Control of the structure and function of the extracellular matrix (ECM) is a central theme of the biology of the ADAMTS, as exemplified by the actions of the procollagen-N-propeptidases in collagen fibril assembly and of the aggrecanases in the cleavage or modification of ECM proteoglycans. Defects in certain family members give rise to inherited genetic disorders, while the aberrant expression or function of others is associated with arthritis, cancer and cardiovascular disease. In particular, ADAMTS4 and 5 have emerged as therapeutic targets in arthritis. Multiple ADAMTSs from different sub-groupings exert either positive or negative effects on tumorigenesis and metastasis, with both metalloproteinase-dependent and -independent actions known to occur. The basic ADAMTS structure comprises a metalloproteinase catalytic domain and a carboxy-terminal ancillary domain, the latter determining substrate specificity and the localization of the protease and its interaction partners; ancillary domains probably also have independent biological functions. Focusing primarily on the aggrecanases and proteoglycanases, this review provides a perspective on the evolution of the ADAMTS family, their links with developmental and disease mechanisms, and key questions for the future

Cross talk of signals between EGFR and IL-6R through JAK2/STAT3 mediate epithelial–mesenchymal transition in ovarian carcinomas

Epidermal growth factor receptor (EGFR) is overexpressed in ovarian carcinomas, with direct or indirect activation of EGFR able to trigger tumour growth. We demonstrate significant activation of both signal transducer and activator of transcription (STAT)3 and its upstream activator Janus kinase (JAK)2, in high-grade ovarian carcinomas compared with normal ovaries and benign tumours. The association between STAT3 activation and migratory phenotype of ovarian cancer cells was investigated by EGF-induced epithelial–mesenchymal transition (EMT) in OVCA 433 and SKOV3 ovarian cancer cell lines. Ligand activation of EGFR induced a fibroblast-like morphology and migratory phenotype, consistent with the upregulation of mesenchyme-associated N-cadherin, vimentin and nuclear translocation of β-catenin. This occurred concomitantly with activation of the downstream JAK2/STAT3 pathway. Both cell lines expressed interleukin-6 receptor (IL-6R), and treatment with EGF within 1 h resulted in a several-fold enhancement of mRNA expression of IL-6. Consistent with that, EGF treatment of both OVCA 433 and SKOV3 cell lines resulted in enhanced IL-6 production in the serum-free medium. Exogenous addition of IL-6 to OVCA 433 cells stimulated STAT3 activation and enhanced migration. Blocking antibodies against IL-6R inhibited IL-6 production and EGF- and IL-6-induced migration. Specific inhibition of STAT3 activation by JAK2-specific inhibitor AG490 blocked STAT3 phosphorylation, cell motility, induction of N-cadherin and vimentin expression and IL6 production. These data suggest that the activated status of STAT3 in high-grade ovarian carcinomas may occur directly through activation of EGFR or IL-6R or indirectly through induction of IL-6R signalling. Such activation of STAT3 suggests a rationale for a combination of anti-STAT3 and EGFR/IL-6R therapy to suppress the peritoneal spread of ovarian cancer

Exposure to the tsunami disaster, PTSD symptoms and increased substance use – an Internet based survey of male and female residents of Switzerland

BACKGROUND: After the tsunami disaster in the Indian Ocean basin an Internet based self-screening test was made available in order to facilitate contact with mental health services. Although primarily designed for surviving Swiss tourists as well as relatives and acquaintances of the victims, the screening instrument was open to anyone who felt psychologically affected by this disaster. The aim of this study was to evaluate the influences between self-declared increased substance use in the aftermath of the tsunami disaster, trauma exposure and current PTSD symptoms. METHODS: One section of the screening covered addiction related behavior. We analyzed the relationship between increased substance use, the level of PTSD symptoms and trauma exposure using multivariable logistic regression with substance use as the dependent variable. Included in the study were only subjects who reported being residents of Switzerland and the analyses were stratified by gender in order to control for possible socio-cultural or gender differences in the use of psychotropic substances. RESULTS: In women PTSD symptoms and degree of exposure enlarged the odds of increased alcohol, pharmaceuticals and cannabis use significantly. In men the relationship was more specific: PTSD symptoms and degree of exposure only enlarged the odds of increased pharmaceutical consumption significantly. Increases in alcohol, cannabis and tobacco use were only significantly associated with the degree of PTSD symptoms. CONCLUSION: The tsunami was associated with increased substance use. This study not only replicates earlier findings but also suggests for a gender specificity of post-traumatic substance use increase

Chondroitinase and Growth Factors Enhance Activation and Oligodendrocyte Differentiation of Endogenous Neural Precursor Cells after Spinal Cord Injury

The adult spinal cord harbours a population of multipotent neural precursor cells (NPCs) with the ability to replace oligodendrocytes. However, despite this capacity, proliferation and endogenous remyelination is severely limited after spinal cord injury (SCI). In the post-traumatic microenvironment following SCI, endogenous spinal NPCs mainly differentiate into astrocytes which could contribute to astrogliosis that exacerbate the outcomes of SCI. These findings emphasize a key role for the post-SCI niche in modulating the behaviour of spinal NPCs after SCI. We recently reported that chondroitin sulphate proteoglycans (CSPGs) in the glial scar restrict the outcomes of NPC transplantation in SCI by reducing the survival, migration and integration of engrafted NPCs within the injured spinal cord. These inhibitory effects were attenuated by administration of chondroitinase (ChABC) prior to NPC transplantation. Here, in a rat model of compressive SCI, we show that perturbing CSPGs by ChABC in combination with sustained infusion of growth factors (EGF, bFGF and PDGF-AA) optimize the activation and oligodendroglial differentiation of spinal NPCs after injury. Four days following SCI, we intrathecally delivered ChABC and/or GFs for seven days. We performed BrdU incorporation to label proliferating cells during the treatment period after SCI. This strategy increased the proliferation of spinal NPCs, reduced the generation of new astrocytes and promoted their differentiation along an oligodendroglial lineage, a prerequisite for remyelination. Furthermore, ChABC and GF treatments enhanced the response of non-neural cells by increasing the generation of new vascular endothelial cells and decreasing the number of proliferating macrophages/microglia after SCI. In conclusions, our data strongly suggest that optimization of the behaviour of endogenous spinal NPCs after SCI is critical not only to promote endogenous oligodendrocyte replacement, but also to reverse the otherwise detrimental effects of their activation into astrocytes which could negatively influence the repair process after SCI

Primary Human mDC1, mDC2, and pDC Dendritic Cells Are Differentially Infected and Activated by Respiratory Syncytial Virus

Respiratory syncytial virus (RSV) causes recurrent infections throughout life. Vaccine development may depend upon understanding the molecular basis for induction of ineffective immunity. Because dendritic cells (DCs) are critically involved in early responses to infection, their interaction with RSV may determine the immunological outcome of RSV infection. Therefore, we investigated the ability of RSV to infect and activate primary mDCs and pDCs using recombinant RSV expressing green fluorescent protein (GFP). At a multiplicity of infection of 5, initial studies demonstrated ∼6.8% of mDC1 and ∼0.9% pDCs were infected. We extended these studies to include CD1c−CD141+ mDC2, finding mDC2 infected at similar frequencies as mDC1. Both infected and uninfected cells upregulated phenotypic markers of maturation. Divalent cations were required for infection and maturation, but maturation did not require viral replication. There is evidence that attachment and entry/replication processes exert distinct effects on DC activation. Cell-specific patterns of RSV-induced maturation and cytokine production were detected in mDC1, mDC2, and pDC. We also demonstrate for the first time that RSV induces significant TIMP-2 production in all DC subsets. Defining the influence of RSV on the function of selected DC subsets may improve the likelihood of achieving protective vaccine-induced immunity