Fast Reinforcement Learning with Large Action Sets Using Error-Correcting Output Codes for MDP Factorization

International audienceThe use of Reinforcement Learning in real-world scenarios is strongly limited by issues of scale. Most RL learning algorithms are unable to deal with problems composed of hundreds or sometimes even dozens of possible actions, and therefore cannot be applied to many real-world problems. We consider the RL problem in the supervised classification framework where the optimal policy is obtained through a multiclass classifier, the set of classes being the set of actions of the problem. We introduce error-correcting output codes (ECOCs) in this setting and propose two new methods for reducing complexity when using rollouts-based approaches. The first method consists in using an ECOC-based classifier as the multiclass classifier, reducing the learning complexity from O(A2) to O(Alog(A)) . We then propose a novel method that profits from the ECOC's coding dictionary to split the initial MDP into O(log(A)) separate two-action MDPs. This second method reduces learning complexity even further, from O(A2) to O(log(A)) , thus rendering problems with large action sets tractable. We finish by experimentally demonstrating the advantages of our approach on a set of benchmark problems, both in speed and performance

Bayesian optimization for materials design

We introduce Bayesian optimization, a technique developed for optimizing time-consuming engineering simulations and for fitting machine learning models on large datasets. Bayesian optimization guides the choice of experiments during materials design and discovery to find good material designs in as few experiments as possible. We focus on the case when materials designs are parameterized by a low-dimensional vector. Bayesian optimization is built on a statistical technique called Gaussian process regression, which allows predicting the performance of a new design based on previously tested designs. After providing a detailed introduction to Gaussian process regression, we introduce two Bayesian optimization methods: expected improvement, for design problems with noise-free evaluations; and the knowledge-gradient method, which generalizes expected improvement and may be used in design problems with noisy evaluations. Both methods are derived using a value-of-information analysis, and enjoy one-step Bayes-optimality

Diabetic autonomic neuropathy: evidence for apoptosis in situ in the rat

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/75065/1/j.1365-2982.2004.00524.x.pd

Efficient Induction of Extrinsic Cell Death by Dandelion Root Extract in Human Chronic Myelomonocytic Leukemia (CMML) Cells

BACKGROUND: Chronic Myelomonocytic Leukemia (CMML) is a heterogeneous disease that is not only hard to diagnose and classify, but is also highly resistant to treatment. Available forms of therapy for this disease have not shown significant effects and patients rapidly develop resistance early on in therapy. These factors lead to the very poor prognosis observed with CMML patients, with median survival duration between 12 and 24 months after diagnosis. This study is therefore centered around evaluating the selective efficacy of a natural extract from dandelion roots, in inducing programmed cell death in aggressive and resistant CMML cell lines. METHODOLOGY/PRINCIPAL FINDINGS: To confirm the induction of programmed cell death in three human CMML cell lines, nuclear condensation and externalization of the phosphatidylserine, two main characteristics of apoptosis, were detected using Hoechst staining and annexin-V binding assay. The induction of another mode of cell death, autophagy, was determined using a monodansylcadaverine (MDC) stain, to detect the formation of autophagy vacuoles. The results from this study indicate that Dandelion Root Extract (DRE) is able to efficiently and selectively induce apoptosis and autophagy in these cell lines in a dose and time dependent manner, with no significant toxicity on non-cancerous peripheral blood mononuclear cells. More importantly, we observed early activation of initiator caspase-8, which led to mitochondrial destabilization and the induction of autophagy, suggesting that DRE acts through the extrinsic pathway of apoptosis. The inability of DRE to induce apoptosis in dominant-negative FADD cells, confirms the mechanism of action of DRE in in vitro models of CMML. CONCLUSION: The results from this study indicate that natural products, in particular Dandelion Root Extract, have great potential, as non-toxic and effective alternatives to conventional modes of chemotherapy available today

Overexpression of nucleoside diphosphate/kinase A/nm23-H1 protein in human lung tumors: association with tumor progression in squamous carcinoma.

International audienceLevels of nm23-H1/nucleoside diphosphate/kinase A expression have been reported to correlate inversely with metastatic potential in some tumors but not in others. To clarify the role of nm23 in lung carcinoma, the genetic abnormalities of nucleoside diphosphate/kinase A/nm23-H1 were investigated at the DNA and protein levels. A series of 104 human lung tumors (42 neuroendocrine (NE) and 62 non-NE tumors) was analyzed for nm23-H1 protein expression by immunohistochemistry using one polyclonal and two monoclonal Ab and for genomic alterations using Southern blotting and single-strand conformation polymorphism. Overexpression of the nm23-H1 protein relative to the normal lung epithelia (pneumocyte and bronchial epithelial cells) was observed in 83% (35/42) of NE carcinomas and in 89% (55/62) of non-NE carcinomas. Eight of nine carcinoids exhibited an increased expression of nm23-H1 protein, suggesting that this overexpression of the nm23 protein is necessary for proliferation in any tumors. No significant correlation was found between nm23 staining and any clinicopathologic parameters in NE carcinoma or in adenocarcinoma. In squamous carcinoma, high levels of nm23-H1 protein expression were associated with tumor stage (p = 0.0036). Allelic deletion or genetic amplification was never found. No altered mobility was detected using single-strand conformation polymorphism analysis. These data show that nm23-H1 protein is overexpressed in a large number of lung tumors of all histologic types, in association with advanced tumor stage in squamous carcinoma. They also suggest that nm23-H1 might play a role in the progression of lung tumors rather than in antimetastatic function

p53 mutant immunophenotype and deregulation of p53 transcription pathway (Bcl2, Bax, and Waf1) in precursor bronchial lesions of lung cancer.

International audienceLung cancer is the end result of a multistep process in which genetic and molecular changes accompany, in an unknown temporal sequence, histological precursor (preinvasive) bronchial lesions. Biomarkers allowing prediction of the rate of progression of precursor lesions at different locations in the anatomical field may be clinically useful. Toward this aim, we analyzed, using immunohistochemistry, the expression of the p53 gene and of its transcriptional target genes bax, bcl2, and waf1 in preinvasive bronchial lesions from 69 patients with lung cancer and in similar lesions from 20 patients with no cancer progression. p53 accumulation occurred with increasing frequency, from 19% in mild dysplasia to 36% in moderate dysplasia and 59% in carcinoma in situ, and was exclusively observed in patients with p53-positive carcinoma. The higher frequency of the p53-positive immunophenotype in lesions adjacent to the p53-positive carcinoma, as compared to lesions distant from it, suggests that p53 mutant preneoplastic lesions had a higher rate of progression to invasion than did p53-negative lesions. Similar lesions in patients with no progression to lung cancer were all p53 negative. Bcl2 overexpression and Bax down-regulation, as shown by immunostaining, occurred in preinvasive lesions and were mainly maintained during invasion. The expressions of bax, bcl2, and waf1 did not correlate with p53 status. We conclude that p53 stabilization in preinvasive lesions has high predictive value for progression to invasion and that Bax/Bcl2 imbalance contributes to the clonal expansion during premalignant states

Apoptosis-related factors p53, Bcl2, and Bax in neuroendocrine lung tumors.

Neuroendocrine (NE) lung tumors comprise four classes of progressive aggressiveness for which proliferation and apoptosis rates could both contribute to their distinctive behavior. As p53 mutations may favor escape from apoptosis through changes in Bcl2-Bax expression balance, which are survival and apoptotic genes, respectively, we studied 121 NE lung tumors (16 typical carcinoids (TC), 5 atypical carcinoids (AC), 29 large-cell NE carcinomas (LCNECs), and 71 small-cell lung carcinomas (SCLCs) using immunohistochemistry. We quantified apoptosis by terminal-deoxynucleotidyl-transferase-mediated deoxyuridine triphosphate nick end labeling (TUNEL) in 31 of these cases. There was a significant increase of p53 mutant immunophenotype (defined as immunoreactivity with at least two antibodies for at least 20% of tumor cells) between atypical/typical carcinoids group and the LCNEC/SCLC group (P = 0.0003). There was an inverse correlation (P < 0.0001) between the scores of Bax and Bcl2 expression in individual tumors and a significant inversion of the Bcl2. Bax ratio between low-grade (typical and atypical carcinoids) and high-grade (LCNECs and SCLCs) tumors with a predominant Bax expression in the first group and predominant Bcl2 expression in the second. Whereas carcinoids had variable apoptotic indexes, LCNECs had high indexes (1.3 to 6.8%), Bcl2 overexpression, Bax down-regulation, and Bcl2.Bax ratio > 1 correlated with lower apoptotic index in both LCNEC and the pool of LCNECs and SCLCs (P < 0.05) and a lower survival rate in the group of atypical and typical carcinoids and LCNECs (P < 0.002). The highest levels of Bcl2 expression and Bcl2.Bax ratios were associated with p53 mutant immunophenotype (P = 0.02). Our results suggest that aggressiveness in NE lung tumors could be linked, in addition to proliferation, to apoptosis-related factors