Multi-Armed Bandits for Correlated Markovian Environments with Smoothed Reward Feedback

We study a multi-armed bandit problem in a dynamic environment where arm rewards evolve in a correlated fashion according to a Markov chain. Different than much of the work on related problems, in our formulation a learning algorithm does not have access to either a priori information or observations of the state of the Markov chain and only observes smoothed reward feedback following time intervals we refer to as epochs. We demonstrate that existing methods such as UCB and $\varepsilon$-greedy can suffer linear regret in such an environment. Employing mixing-time bounds on Markov chains, we develop algorithms called EpochUCB and EpochGreedy that draw inspiration from the aforementioned methods, yet which admit sublinear regret guarantees for the problem formulation. Our proposed algorithms proceed in epochs in which an arm is played repeatedly for a number of iterations that grows linearly as a function of the number of times an arm has been played in the past. We analyze these algorithms under two types of smoothed reward feedback at the end of each epoch: a reward that is the discount-average of the discounted rewards within an epoch, and a reward that is the time-average of the rewards within an epoch.Comment: Significant revision of prior version including deeper discussion of related work, gap-independent regret bounds, and regret bounds for discounted reward

Activation of FGF Signaling Mediates Proliferative and Osteogenic Differences between Neural Crest Derived Frontal and Mesoderm Parietal Derived Bone

BACKGROUND: As a culmination of efforts over the last years, our knowledge of the embryonic origins of the mammalian frontal and parietal cranial bones is unambiguous. Progenitor cells that subsequently give rise to frontal bone are of neural crest origin, while parietal bone progenitors arise from paraxial mesoderm. Given the unique qualities of neural crest cells and the clear delineation of the embryonic origins of the calvarial bones, we sought to determine whether mouse neural crest derived frontal bone differs in biology from mesoderm derived parietal bone. METHODS: BrdU incorporation, immunoblotting and osteogenic differentiation assays were performed to investigate the proliferative rate and osteogenic potential of embryonic and postnatal osteoblasts derived from mouse frontal and parietal bones. Co-culture experiments and treatment with conditioned medium harvested from both types of osteoblasts were performed to investigate potential interactions between the two different tissue origin osteoblasts. Immunoblotting techniques were used to investigate the endogenous level of FGF-2 and the activation of three major FGF signaling pathways. Knockdown of FGF Receptor 1 (FgfR1) was employed to inactivate the FGF signaling. RESULTS: Our results demonstrated that striking differences in cell proliferation and osteogenic differentiation between the frontal and parietal bone can be detected already at embryonic stages. The greater proliferation rate, as well as osteogenic capacity of frontal bone derived osteoblasts, were paralleled by an elevated level of FGF-2 protein synthesis. Moreover, an enhanced activation of FGF-signaling pathways was observed in frontal bone derived osteoblasts. Finally, the greater osteogenic potential of frontal derived osteoblasts was dramatically impaired by knocking down FgfR1. CONCLUSIONS: Osteoblasts from mouse neural crest derived frontal bone displayed a greater proliferative and osteogenic potential and endogenous enhanced activation of FGF signaling compared to osteoblasts from mesoderm derived parietal bone. FGF signaling plays a key role in determining biological differences between the two types of osteoblasts

GSK-3β Function in Bone Regulates Skeletal Development, Whole-Body Metabolism, and Male Life Span

Glycogen synthase kinase 3 β (GSK-3β) is an essential negative regulator or “brake” on many anabolic-signaling pathways including Wnt and insulin. Global deletion of GSK-3β results in perinatal lethality and various skeletal defects. The goal of our research was to determine GSK-3β cell-autonomous effects and postnatal roles in the skeleton. We used the 3.6-kb Col1a1 promoter to inactivate the Gsk3b gene (Col1a1-Gsk3b knockout) in skeletal cells. Mutant mice exhibit decreased body fat and postnatal bone growth, as well as delayed development of several skeletal elements. Surprisingly, the mutant mice display decreased circulating glucose and insulin levels despite normal expression of GSK-3β in metabolic tissues. We showed that these effects are due to an increase in global insulin sensitivity. Most of the male mutant mice died after weaning. Prior to death, blood glucose changed from low to high, suggesting a possible switch from insulin sensitivity to resistance. These male mice die with extremely large bladders that are preceded by damage to the urogenital tract, defects that are also seen type 2 diabetes. Our data suggest that skeletal-specific deletion of GSK-3β affects global metabolism and sensitizes male mice to developing type 2 diabetes. (Endocrinology 154: 3702–3718, 2013

Recommandations pour l’utilisation de la toxine botulinique de type A (Botox®) dans l’hyperactivité vésicale réfractaire idiopathique

RésuméObjectifsDéfinir des recommandations pour l’utilisation pratique de la toxine botulinique de type A (BoNTA) dans l’hyperactivité vésicale réfractaire idiopathique (HAVRI).MéthodeÉlaboration de recommandations de bonne pratique par consensus formalisé, validées par un groupe de 13 experts puis par un groupe de lecture indépendant.RésultatsEn cas d’infection urinaire celle-ci doit être traitée et l’injection reportée. Avant l’injection, il est recommandé de s’assurer de la faisabilité et de l’acceptabilité de l’auto-sondage. L’injection peut être réalisée après une anesthésie locale urétro-vésicale (lidocaïne), éventuellement complétée par l’inhalation de protoxyde d’azote et parfois sous anesthésie générale. L’injection sera réalisée au bloc opératoire ou en salle d’endoscopie. La vessie ne doit pas être trop remplie (risque de perforation). Le traitement doit être appliqué en 10 à 20 injections de 0,5 à 1mL réparties de manière homogène dans la vessie en restant à distance des méats urétéraux. Il n’est pas recommandé de laisser en place une sonde vésicale sauf en cas d’hématurie importante. Le patient doit être surveillé jusqu’à la reprise mictionnelle. Une note d’information sur les effets indésirables éventuels doit lui être remise à sa sortie. Une consultation doit être prévue 3 mois après la première injection (calendrier mictionnel, débitmétrie, résidu post-mictionnel et examen cytobactériologique des urines). Un résidu >200mL et/ou symptomatique doit faire discuter des auto-sondages. Une nouvelle injection pourra être envisagée lorsque le bénéfice clinique de la précédente s’estompe (entre 6 et 9 mois).ConclusionsLe respect de ces recommandations devrait permettre une utilisation optimale de la BoNTA.Niveau de preuve3.SummaryObjectivesProvide guidelines for practical usage of botulinum toxin type A (BoNTA) for refractory idiopathic Overactive Bladder management.Patients and methodsGuidelines using formalized consensus guidelines method. These guidelines have been validated by a group of 13 experts quoting proposals, subsequently reviewed by an independent group of experts.ResultsIn the case of patients with urinary tract infection, it must be treated and injection postponed. Before proposing an injection, it is recommended to ensure the feasibility and acceptability of self-catheterisation by patient. The injection can be performed after local anesthesia of the bladder and urethra (lidocaine), supplemented where necessary by nitrous oxide inhalation and sometimes under general anesthesia. Injection is performed in the operating room or endoscopy suite. The bladder should not be too filled (increased risk of perforation). Treatment should be applied in 10 to 20 injections of 0.5 to 1mL homogeneously distributed in the bladder at a distance from the urethral orifices. It is not recommended to leave a urinary catheter in place except in cases of severe hematuria. The patient should be monitored until resumption of micturition. After the first injection, an appointment must be scheduled within 3 months (micturition diary, uroflowmetry, measurement of residual urine and urine culture). Performance of self-catheterisation should be questioned in the case of a symptomatic post-void residual and/or a residue>200mL. A new injection may be considered when the clinical benefit of the previous injection diminishes (between 6 and 9 months). A period of three months must elapse between each injection.ConclusionsImplementation of these guidelines may promote best practice usage of BoNTA with optimal risk/benefit ratio

Endocrine Regulation of Male Fertility by the Skeleton

Although the endocrine capacity of bone is widely recognized, interactions between bone and the reproductive system have until now focused on the gonads as a regulator of bone remodeling. We now show that in males, bone acts as a regulator of fertility. Using co-culture assays, we demonstrate that osteoblasts are able to induce testosterone production by the testes, while they fail to influence estrogen production by the ovaries. Analyses of cell-specific loss- and gain-of-function models reveal that the osteoblast-derived hormone osteocalcin performs this endocrine function. By binding to a G-protein coupled receptor expressed in the Leydig cells of the testes, osteocalcin regulates in a CREB-dependent manner the expression of enzymes required for testosterone synthesis, promoting germ cell survival. This study expands the physiological repertoire of osteocalcin, and provides the first evidence that the skeleton is an endocrine regulator of reproduction

Effects of low-magnitude, high-frequency mechanical stimulation in the rat osteopenia model

In this study, short-term, whole-body vertical vibration at 90 Hz improved trabecular bone quality. There was an improvement of bone quality and density in both osteoporotic and control rats. This treatment may therefore be an attractive option for the treatment of osteoporosis. Aside from pharmacological treatment options, physical exercise is known to augment bone mass. In this study, the effects of whole-body vertical vibration (WBVV) on bone quality and density were evaluated using an osteoporotic rat model. Sixty female Sprague Dawley rats were ovariectomized (C) or sham (SHAM) operated at the age of 3 months. After 3 months, both groups were divided into two subgroups that received either WBVV at 90 Hz for 35 days or no treatment. After sacrificing the rats, we evaluated vertebral bone strength, histomorphometric parameters, and bone mineral density (BMD). Treatment with WBVV resulted in improved biomechanical properties. The yield load after WBVV was significantly enhanced. According to yield load and Young's modulus, the treated OVX rats reached the level of the untreated SHAM animals. In all measured histomorphometric parameters, WBVV significantly improved bone density. Treatment with WBVV demonstrated greater effects on the trabecular bone compared to the cortical bone. The ash-BMD index showed significant differences between treated and untreated rats. Using WBVV as a non-pharmacological supportive treatment option for osteoporosis demonstrated an enhancement of bone strength and bone mass. This procedure may be an attractive option for the treatment of osteoporosis

Myoblast sensitivity and fibroblast insensitivity to osteogenic conversion by BMP-2 correlates with the expression of Bmpr-1a

<p>Abstract</p> <p>Background</p> <p>Osteoblasts are considered to primarily arise from osseous progenitors within the periosteum or bone marrow. We have speculated that cells from local soft tissues may also take on an osteogenic phenotype. Myoblasts are known to adopt a bone gene program upon treatment with the osteogenic bone morphogenetic proteins (BMP-2,-4,-6,-7,-9), but their osteogenic capacity relative to other progenitor types is unclear. We further hypothesized that the sensitivity of cells to BMP-2 would correlate with BMP receptor expression.</p> <p>Methods</p> <p>We directly compared the BMP-2 sensitivity of myoblastic murine cell lines and primary cells with osteoprogenitors from osseous tissues and fibroblasts. Fibroblasts forced to undergo myogenic conversion by transduction with a MyoD-expressing lentiviral vector (LV-MyoD) were also examined. Outcome measures included alkaline phosphatase expression, matrix mineralization, and expression of osteogenic genes <it>(alkaline phosphatase, osteocalcin </it>and <it>bone morphogenetic protein receptor-1A) </it>as measured by quantitative PCR.</p> <p>Results</p> <p>BMP-2 induced a rapid and robust osteogenic response in myoblasts and osteoprogenitors, but not in fibroblasts. Myoblasts and osteoprogenitors grown in osteogenic media rapidly upregulated <it>Bmpr-1a </it>expression. Chronic BMP-2 treatment resulted in peak <it>Bmpr-1a </it>expression at day 6 before declining, suggestive of a negative feedback mechanism. In contrast, fibroblasts expressed low levels of <it>Bmpr-1a </it>that was only weakly up-regulated by BMP-2 treatment. Bioinformatics analysis confirmed the presence of myogenic responsive elements in the proximal promoter region of human and murine <it>BMPR-1A/Bmpr-1a</it>. Forced myogenic gene expression in fibroblasts was associated with a significant increase in <it>Bmpr-1a </it>expression and a synergistic increase in the osteogenic response to BMP-2.</p> <p>Conclusion</p> <p>These data demonstrate the osteogenic sensitivity of muscle progenitors and provide a mechanistic insight into the variable response of different cell lineages to BMP-2.</p

The economic case for prioritizing governance over financial incentives in REDD+

This article contributes to the ongoing debate on the role of public policies and financial incentives in Reducing Emissions from Deforestation and forest Degradation (REDD+). It argues that the subordination of policies to results-based payments for emission reductions causes severe economic inefficiencies affecting the opportunity cost, transaction cost and economic rent of the programme. Such problems can be addressed by establishing sound procedural, land and financial governance at the national level, before REDD+ economic incentives are delivered at scale. Consideration is given to each governance dimension, the entry points for policy intervention and the impact on costs. International support must consider the financial and political cost of governance reforms, and use a pay-for-results ethos based on output and outcome indicators. This can be done in the readiness process but only if the latter’s legal force, scope, magnitude and time horizon are adequately reconsidered. In sum, the paper provides ammunition for the institutionalist argument that UNFCCC Parties must prioritise governance reform between now and the entry into force of the new climate agreement in 2020, and specific recommendations about how this can be done: only by doing so will they create the basis for the programme’s financial sustainability