Model-Based Reinforcement Learning Exploiting State-Action Equivalence

International audienceLeveraging an equivalence property in the state-space of a Markov Decision Process (MDP) has been investigated in several studies. This paper studies equivalence structure in the reinforcement learning (RL) setup, where transition distributions are no longer assumed to be known. We present a notion of similarity between transition probabilities of various state-action pairs of an MDP, which naturally defines an equivalence structure in the state-action space. We present equivalence-aware confidence sets for the case where the learner knows the underlying structure in advance. These sets are provably smaller than their corresponding equivalence-oblivious counterparts. In the more challenging case of an unknown equivalence structure, we present an algorithm called ApproxEquivalence that seeks to find an (approximate) equivalence structure, and define confidence sets using the approximate equivalence. To illustrate the efficacy of the presented confidence sets, we present C-UCRL, as a natural modification of UCRL2 for RL in undiscounted MDPs. In the case of a known equivalence structure, we show that C-UCRL improves over UCRL2 in terms of regret by a factor of SA/C, in any communicating MDP with S states, A actions, and C classes, which corresponds to a massive improvement when C SA. To the best of our knowledge, this is the first work providing regret bounds for RL when an equivalence structure in the MDP is efficiently exploited. In the case of an unknown equivalence structure, we show through numerical experiments that C-UCRL combined with ApproxEquivalence outperforms UCRL2 in ergodic MDPs

Hemangioblastoma of the Central Nervous System: A Case Series of Patients Surgically Treated at

Objective: Hemangioblastoma refers to a benign vascular neoplasm that comprises stromal and capillary cells. Based on the classification of nervous system tumors proposed by the World Health Organization, hemangioblastomas are classified as Grade I meningeal tumors of uncertain origin. These tumors are found almost exclusively in the central nervous system (CNS) and account for 0.9 to 2.1% of all primary CNS tumors.Methods: In this descriptive retrospective study, the archives of pathology reports were reviewed in the department of pathology of Shohadaye Tajrish Hospital and patients with definite diagnosis of hemangioblastoma made through histpopathological examinations during 2004-2014 were identified. Age, gender and the location of tumor were extracted from the medical records and entered into SPSS statistical software v.22 for analysis.Results: A total of 30 patients including 16 males (53.3%) and 14 females (46.7%) were identified. The mean age of the patients was calculated to be 41.2±13.47 years, ranging from 19 to 62 years old. The majority of lesions had been found in the cerebellum of the patients (93.3%); only one had occurred in the cerebrum (3.3%) and another in the fourth ventricle (3.3%). Conclusion: Cerebellum is the most commonly affected location in patients with CNS hemangioblastomas, and a male preponderance is observed in these cases.

Model-Based Reinforcement Learning Exploiting State-Action Equivalence

International audienceLeveraging an equivalence property in the state-space of a Markov Decision Process (MDP) has been investigated in several studies. This paper studies equivalence structure in the reinforcement learning (RL) setup, where transition distributions are no longer assumed to be known. We present a notion of similarity between transition probabilities of various state-action pairs of an MDP, which naturally defines an equivalence structure in the state-action space. We present equivalence-aware confidence sets for the case where the learner knows the underlying structure in advance. These sets are provably smaller than their corresponding equivalence-oblivious counterparts. In the more challenging case of an unknown equivalence structure, we present an algorithm called ApproxEquivalence that seeks to find an (approximate) equivalence structure, and define confidence sets using the approximate equivalence. To illustrate the efficacy of the presented confidence sets, we present C-UCRL, as a natural modification of UCRL2 for RL in undiscounted MDPs. In the case of a known equivalence structure, we show that C-UCRL improves over UCRL2 in terms of regret by a factor of SA/C, in any communicating MDP with S states, A actions, and C classes, which corresponds to a massive improvement when C SA. To the best of our knowledge, this is the first work providing regret bounds for RL when an equivalence structure in the MDP is efficiently exploited. In the case of an unknown equivalence structure, we show through numerical experiments that C-UCRL combined with ApproxEquivalence outperforms UCRL2 in ergodic MDPs

Statin efficacy in the treatment of hepatitis C genotype I

Background: Lipid metabolism is one of the hepatitis C virus (HCV) life cycle steps. Statins can reduce cholesterol level and finally can decrease HCV replication. Thus, we assessed the effect of Statins in combination with standard antiviral treatment on hyperlipidemic genotype I HCV infected patients. Materials and Methods: This study was a prospective clinical trial. 40 patients were selected from those referred to educational and Therapeutic Centers of Isfahan University of Medical Sciences from 2009 to 2010 with confirmed HCV viremia. All patients received Peg-interferon-a2a and ribavirin. 20 hyperlipidemic Patients received 20 mg atorvastatin nightly for 3 months and placebo was prescribed for 20 normolipidemic HCV infected patients as a control group. Liver enzymes and complete blood count were checked monthly and thyroid stimulating hormone was checked every 3 months. We also performed quantitative HCV-ribonucleic acid (RNA) test in 12th week of therapy, at the end of treatment and 6 months after therapy for all samples. Results: We didn't find any significant differences in the mean of HCV-RNA numbers between statin and placebo groups in 12th week of treatment, in the end of treatment and 6 months after treatment (P > 0.05). Conclusion: Atorvastatin has no effect on the mean of HCV viral load when we added it to standard treatment for hepatitis C infection. Further studies are necessary to examine the possible antiviral properties of statins and their potential role as adjuncts to standard HCV therapy

Model-Based Reinforcement Learning Exploiting State-Action Equivalence

International audienceLeveraging an equivalence property in the state-space of a Markov Decision Process (MDP) has been investigated in several studies. This paper studies equivalence structure in the reinforcement learning (RL) setup, where transition distributions are no longer assumed to be known. We present a notion of similarity between transition probabilities of various state-action pairs of an MDP, which naturally defines an equivalence structure in the state-action space. We present equivalence-aware confidence sets for the case where the learner knows the underlying structure in advance. These sets are provably smaller than their corresponding equivalence-oblivious counterparts. In the more challenging case of an unknown equivalence structure, we present an algorithm called ApproxEquivalence that seeks to find an (approximate) equivalence structure, and define confidence sets using the approximate equivalence. To illustrate the efficacy of the presented confidence sets, we present C-UCRL, as a natural modification of UCRL2 for RL in undiscounted MDPs. In the case of a known equivalence structure, we show that C-UCRL improves over UCRL2 in terms of regret by a factor of SA/C, in any communicating MDP with S states, A actions, and C classes, which corresponds to a massive improvement when C SA. To the best of our knowledge, this is the first work providing regret bounds for RL when an equivalence structure in the MDP is efficiently exploited. In the case of an unknown equivalence structure, we show through numerical experiments that C-UCRL combined with ApproxEquivalence outperforms UCRL2 in ergodic MDPs

Biochemical Modulation of Lipid Pathway in Microalgae Dunaliella sp. for Biodiesel Production

Exploitation of renewable sources of energy such as algal biodiesel could turn energy supplies problem around. Studies on a locally isolated strain of Dunaliella sp. showed that the mean lipid content in cultures enriched by 200 mg L−1 myoinositol was raised by around 33% (1.5 times higher than the control). Similarly, higher lipid productivity values were achieved in cultures treated by 100 and 200 mg L−1 myoinositol. Fluorometry analyses (microplate fluorescence and flow cytometry) revealed increased oil accumulation in the Nile red-stained algal samples. Moreover, it was predicted that biodiesel produced from myoinositol-treated cells possessed improved oxidative stability, cetane number, and cloud point values. From the genomic point of view, real-time analyses revealed that myoinositol negatively influenced transcript abundance of AccD gene (one of the key genes involved in lipid production pathway) due to feedback inhibition and that its positive effect must have been exerted through other genes. The findings of the current research are not to interprete that myoinositol supplementation could answer all the challenges faced in microalgal biodiesel production but instead to show that “there is a there there” for biochemical modulation strategies, which we achieved, increased algal oil quantity and enhanced resultant biodiesel quality

Hoarseness as the Presenting Symptom of Visceral Leishmaniasis with Muco-Cutaneous Lesions: A Case Report

  Herein, a 28-year-old man with hoarseness, skin and oral lesions is presented. At the time of admission, the patient had an erythematous plaque on his chin near his lower lip and an erythematous-violaceous plaque on his palate near the open-ing of the pharynx and 20 kg weight lost in last one year. The biopsy of his skin lesions by hematoxylin and eosin staining revealed an infiltration of the dermis by lymphoplasma and histiocytic cells with a loose granuloma formation sugges-tive of leishmaniasis. Biopsy of mucosal lesions revealed Leishman bodies in dermis. PCR was performed on the specimens of skin, bone marrow, mucosa, and saliva, the results were positive. The pathogenic agent was identified as Leishmania major by the nested PCR. Serologic tests including direct agglutination test (DAT) and indirect immunofluorescence test (IFAT) were positive with high titers of anti-L. infantum antibodies (1:102400 versus 1:800, respectively), indicative of visceral involvement. The patient responded to a combination of miltefosine and meglumine antimoniate (Glucantime®). Visceral involvement due to L. major is rarely reported. To the best of our knowledge, probably hoarseness due to L. major has not been previously reported from Iran

Effective Connectivity Evaluation of Resting-State Brain Networks in Alzheimer’s Disease, Amnestic Mild Cognitive Impairment, and Normal Aging: An Exploratory Study

(1) Background: Alzheimer’s disease (AD) is a neurodegenerative disease with a high prevalence. Despite the cognitive tests to diagnose AD, there are pitfalls in early diagnosis. Brain deposition of pathological markers of AD can affect the direction and intensity of the signaling. The study of effective connectivity allows the evaluation of intensity flow and signaling pathways in functional regions, even in the early stage, known as amnestic mild cognitive impairment (aMCI). (2) Methods: 16 aMCI, 13 AD, and 14 normal subjects were scanned using resting-state fMRI and T1-weighted protocols. After data pre-processing, the signal of the predefined nodes was extracted, and spectral dynamic causal modeling analysis (spDCM) was constructed. Afterward, the mean and standard deviation of the Jacobin matrix of each subject describing effective connectivity was calculated and compared. (3) Results: The maps of effective connectivity in the brain networks of the three groups were different, and the direction and strength of the causal effect with the progression of the disease showed substantial changes. (4) Conclusions: Impaired information flow in the resting-state networks of the aMCI and AD groups was found versus normal groups. Effective connectivity can serve as a potential marker of Alzheimer’s pathophysiology, even in the early stages of the disease

Multifunctional Self-Assembled Peptide Hydrogels for Biomedical Applications

Self-assembly is a growth mechanism in nature to apply local interactions forming a minimum energy structure. Currently, self-assembled materials are considered for biomedical applications due to their pleasant features, including scalability, versatility, simplicity, and inexpensiveness. Self-assembled peptides can be applied to design and fabricate different structures, such as micelles, hydrogels, and vesicles, by diverse physical interactions between specific building blocks. Among them, bioactivity, biocompatibility, and biodegradability of peptide hydrogels have introduced them as versatile platforms in biomedical applications, such as drug delivery, tissue engineering, biosensing, and treating different diseases. Moreover, peptides are capable of mimicking the microenvironment of natural tissues and responding to internal and external stimuli for triggered drug release. In the current review, the unique characteristics of peptide hydrogels and recent advances in their design, fabrication, as well as chemical, physical, and biological properties are presented. Additionally, recent developments of these biomaterials are discussed with a particular focus on their biomedical applications in targeted drug delivery and gene delivery, stem cell therapy, cancer therapy and immune regulation, bioimaging, and regenerative medicine