Probabilistic inference for determining options in reinforcement learning

Tasks that require many sequential decisions or complex solutions are hard to solve using conventional reinforcement learning algorithms. Based on the semi Markov decision process setting (SMDP) and the option framework, we propose a model which aims to alleviate these concerns. Instead of learning a single monolithic policy, the agent learns a set of simpler sub-policies as well as the initiation and termination probabilities for each of those sub-policies. While existing option learning algorithms frequently require manual specification of components such as the sub-policies, we present an algorithm which infers all relevant components of the option framework from data. Furthermore, the proposed approach is based on parametric option representations and works well in combination with current policy search methods, which are particularly well suited for continuous real-world tasks. We present results on SMDPs with discrete as well as continuous state-action spaces. The results show that the presented algorithm can combine simple sub-policies to solve complex tasks and can improve learning performance on simpler tasks

Instantons and Killing spinors

We investigate instantons on manifolds with Killing spinors and their cones. Examples of manifolds with Killing spinors include nearly Kaehler 6-manifolds, nearly parallel G_2-manifolds in dimension 7, Sasaki-Einstein manifolds, and 3-Sasakian manifolds. We construct a connection on the tangent bundle over these manifolds which solves the instanton equation, and also show that the instanton equation implies the Yang-Mills equation, despite the presence of torsion. We then construct instantons on the cones over these manifolds, and lift them to solutions of heterotic supergravity. Amongst our solutions are new instantons on even-dimensional Euclidean spaces, as well as the well-known BPST, quaternionic and octonionic instantons.Comment: 40 pages, 2 figures v2: author email addresses and affiliations adde

When Does Reward Maximization Lead to Matching Law?

What kind of strategies subjects follow in various behavioral circumstances has been a central issue in decision making. In particular, which behavioral strategy, maximizing or matching, is more fundamental to animal's decision behavior has been a matter of debate. Here, we prove that any algorithm to achieve the stationary condition for maximizing the average reward should lead to matching when it ignores the dependence of the expected outcome on subject's past choices. We may term this strategy of partial reward maximization “matching strategy”. Then, this strategy is applied to the case where the subject's decision system updates the information for making a decision. Such information includes subject's past actions or sensory stimuli, and the internal storage of this information is often called “state variables”. We demonstrate that the matching strategy provides an easy way to maximize reward when combined with the exploration of the state variables that correctly represent the crucial information for reward maximization. Our results reveal for the first time how a strategy to achieve matching behavior is beneficial to reward maximization, achieving a novel insight into the relationship between maximizing and matching

Higher pre-infection vitamin E levels are associated with higher mortality in HIV-1-infected Kenyan women: a prospective study

Background: Low vitamin E levels are often found in HIV-1 infection, and studies have suggested that higher levels may decrease the risk of disease progression. However, vitamin E supplementation has also been reported to increase CCR5 expression, which could increase HIV- 1 replication. We hypothesized that vitamin E levels at HIV-1 acquisition may influence disease progression. Methods: Vitamin E status was measured in stored samples from the last pre-infection visit for 67 Kenyan women with reliably estimated dates of HIV-1 acquisition. Regression analyses were used to estimate associations between pre-infection vitamin E and plasma viral load, time to CD4 count less than 200 cells/[micro]L, and mortality. Results: After controlling for potential confounding factors, each 1 mg/L increase in pre-infection vitamin E was associated with 0.08 log[sub]10 copies/mL (95% CI -0.01 to +0.17) higher set point viral load and 1.58-fold higher risk of mortality (95% CI 1.15�2.16). The association between higher preinfection vitamin E and mortality persisted after adjustment for set point viral load (HR 1.55, 95% CI 1.13�2.13). Conclusion: Higher pre-infection vitamin E levels were associated with increased mortality. Further research is needed to elucidate the role vitamin E plays in HIV-1 pathogenesis.This research was supported by National Institutes of Health grants AI-43844 and AI-38518 (all authors), and Fogarty International Center grant D43 TW000007 (SMG)

Hypothesis: primary antiangiogenic method proposed to treat early stage breast cancer

<p>Abstract</p> <p>Background</p> <p>Women with Down syndrome very rarely develop breast cancer even though they now live to an age when it normally occurs. This may be related to the fact that Down syndrome persons have an additional copy of chromosome 21 where the gene that codes for the antiangiogenic protein Endostatin is located. Can this information lead to a primary antiangiogenic therapy for early stage breast cancer that indefinitely prolongs remission? A key question that arises is when is the initial angiogenic switch thrown in micrometastases? We have conjectured that avascular micrometastases are dormant and relatively stable if undisturbed but that for some patients angiogenesis is precipitated by surgery. We also proposed that angiogenesis of micrometastases very rarely occurs before surgical removal of the primary tumor. If that is so, it seems possible that we could suggest a primary antiangiogenic therapy but the problem then arises that starting a therapy before surgery would interfere with wound healing.</p> <p>Results</p> <p>The therapy must be initiated at least one day prior to surgical removal of the primary tumor and kept at a Down syndrome level perhaps indefinitely. That means the drug must have virtually no toxicity and not interfere meaningfully with wound healing. This specifically excludes drugs that significantly inhibit the VEGF pathway since that is important for wound healing and because these agents have some toxicity. Endostatin is apparently non-toxic and does not significantly interfere with wound healing since Down syndrome patients have no abnormal wound healing problems.</p> <p>Conclusion</p> <p>We propose a therapy for early stage breast cancer consisting of Endostatin at or above Down syndrome levels starting at least one day before surgery and continuing at that level. This should prevent micrometastatic angiogenesis resulting from surgery or at any time later. Adjuvant chemotherapy or hormone therapy should not be necessary. This can be continued indefinitely since there is no acquired resistance that develops, as happens in most cancer therapies.</p

Sub-Optimal Vitamin B-12 Levels among ART-Naïve HIV-Positive Individuals in an Urban Cohort in Uganda

Malnutrition is common among HIV-infected individuals and is often accompanied by low serum levels of micronutrients. Vitamin B-12 deficiency has been associated with various factors including faster HIV disease progression and CD4 depletion in resource-rich settings. To describe prevalence and factors associated with sub-optimal vitamin B-12 levels among HIV-infected antiretroviral therapy (ART) naïve adults in a resource-poor setting, we performed a cross-sectional study with a retrospective chart review among individuals attending either the Mulago-Mbarara teaching hospitals’ Joint AIDS Program (MJAP) or the Infectious Diseases Institute (IDI) clinics, in Kampala, Uganda. Logistic regression was used to determine factors associated with sub-optimal vitamin B-12. The mean vitamin B-12 level was 384 pg/ml, normal range (200–900). Sub-optimal vitamin B-12 levels (<300 pg/ml) were found in 75/204 (36.8%). Twenty-one of 204 (10.3%) had vitamin B-12 deficiency (<200 pg/ml) while 54/204 (26.5%) had marginal depletion (200–300 pg/ml). Irritable mood was observed more among individuals with sub-optimal vitamin B-12 levels (OR 2.5, 95% CI; 1.1–5.6, P = 0.03). Increasing MCV was associated with decreasing serum B-12 category; 86.9 fl (±5.1) vs. 83 fl (±8.4) vs. 82 fl (±8.4) for B-12 deficiency, marginal and normal B-12 categories respectively (test for trend, P = 0.017). Compared to normal B-12, individuals with vitamin B-12 deficiency had a longer known duration of HIV infection: 42.2 months (±27.1) vs. 29.4 months (±23.8; P = 0.02). Participants eligible for ART (CD4<350 cells/µl) with sub-optimal B-12 had a higher mean rate of CD4 decline compared to counterparts with normal B-12; 118 (±145) vs. 22 (±115) cells/µl/year, P = 0.01 respectively. The prevalence of a sub-optimal vitamin B-12 was high in this HIV-infected, ART-naïve adult clinic population in urban Uganda. We recommend prospective studies to further clarify the causal relationships of sub-optimal vitamin B-12, and explore the role of vitamin B-12 supplementation in immune recovery