Thompson Sampling: An Asymptotically Optimal Finite Time Analysis

The question of the optimality of Thompson Sampling for solving the stochastic multi-armed bandit problem had been open since 1933. In this paper we answer it positively for the case of Bernoulli rewards by providing the first finite-time analysis that matches the asymptotic rate given in the Lai and Robbins lower bound for the cumulative regret. The proof is accompanied by a numerical comparison with other optimal policies, experiments that have been lacking in the literature until now for the Bernoulli case.Comment: 15 pages, 2 figures, submitted to ALT (Algorithmic Learning Theory

Functional Sequential Treatment Allocation

Consider a setting in which a policy maker assigns subjects to treatments, observing each outcome before the next subject arrives. Initially, it is unknown which treatment is best, but the sequential nature of the problem permits learning about the effectiveness of the treatments. While the multi-armed-bandit literature has shed much light on the situation when the policy maker compares the effectiveness of the treatments through their mean, much less is known about other targets. This is restrictive, because a cautious decision maker may prefer to target a robust location measure such as a quantile or a trimmed mean. Furthermore, socio-economic decision making often requires targeting purpose specific characteristics of the outcome distribution, such as its inherent degree of inequality, welfare or poverty. In the present paper we introduce and study sequential learning algorithms when the distributional characteristic of interest is a general functional of the outcome distribution. Minimax expected regret optimality results are obtained within the subclass of explore-then-commit policies, and for the unrestricted class of all policies

Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Bandits

International audienceIn this paper, we study the problem of estimating the mean values of all the arms uniformly well in the multi-armed bandit setting. If the variances of the arms were known, one could design an optimal sampling strategy by pulling the arms proportionally to their variances. However, since the distributions are not known in advance, we need to design adaptive sampling strategies to select an arm at each round based on the previous observed samples. We describe two strategies based on pulling the arms proportionally to an upper-bound on their variances and derive regret bounds for these strategies. %on the excess estimation error compared to the optimal allocation. We show that the performance of these allocation strategies depends not only on the variances of the arms but also on the full shape of their distributions

An efficient algorithm for learning with semi-bandit feedback

We consider the problem of online combinatorial optimization under semi-bandit feedback. The goal of the learner is to sequentially select its actions from a combinatorial decision set so as to minimize its cumulative loss. We propose a learning algorithm for this problem based on combining the Follow-the-Perturbed-Leader (FPL) prediction method with a novel loss estimation procedure called Geometric Resampling (GR). Contrary to previous solutions, the resulting algorithm can be efficiently implemented for any decision set where efficient offline combinatorial optimization is possible at all. Assuming that the elements of the decision set can be described with d-dimensional binary vectors with at most m non-zero entries, we show that the expected regret of our algorithm after T rounds is O(m sqrt(dT log d)). As a side result, we also improve the best known regret bounds for FPL in the full information setting to O(m^(3/2) sqrt(T log d)), gaining a factor of sqrt(d/m) over previous bounds for this algorithm.Comment: submitted to ALT 201

PAC-Bayesian Bounds for Randomized Empirical Risk Minimizers

The aim of this paper is to generalize the PAC-Bayesian theorems proved by Catoni in the classification setting to more general problems of statistical inference. We show how to control the deviations of the risk of randomized estimators. A particular attention is paid to randomized estimators drawn in a small neighborhood of classical estimators, whose study leads to control the risk of the latter. These results allow to bound the risk of very general estimation procedures, as well as to perform model selection

Inland valley rice production systems and malaria infection and disease in the forest region of western Côte d'Ivoire

Background: This study aimed to determine the epidemiological impact of rice cultivation in inland valleys on malaria in the forest region of western Côte d'Ivoire. The importance of malaria was compared in terms of prevalence and parasite density of infections and also in terms of clinical malaria incidence between three agro-ecosystems: (i) uncultivated inland valleys, (R0), (ii) inland valleys with one annual rice cultivation in the rainy season, (R1) and (iii) developed inland valleys with two annual rice cultivation cycles, (R2). Methods: Between May 1998 and March 1999, seven villages of each agro-ecosystem (R0, R1 and R2) were randomly selected among villages pooled by farming system. In these 21 villages, a total of 1,900 people of all age groups were randomly selected and clinically monitored during one year. Clinical and parasitological information was obtained by active case detection of malaria episodes carried out during eight periods of five consecutive days scheduled at six weekly intervals and by cross-sectional surveys. Results: Plasmodium falciparum was the principal parasite observed in the three agro-ecosystems. A level of holoendemicity of malaria was observed in the three agro-ecosystems with more than 75% of children less than 12 months old infected. Geometric mean parasite density in asymptomatic persons varied between 180 and 206 P. falciparum asexual forms per μL of blood and was associated with season and with age, but not with farming system. The mean annual malaria incidence rate reached 0.7 (95% IC 0.5-0.9) malaria episodes per person in R0, 0.7 (95% IC 0.6-0.9) in R1 and 0.6 (95% IC 0.5-0.7) in R2. The burden of malaria was the highest among children under two years of age, with at least four attacks by person-year. Then malaria incidence decreased by half in the two to four-year age group. From the age of five years, the incidence was lower than one attack by person-year. Malaria incidence varied with season with more cases in the rainy season than in the dry season but not with farming system. Conclusion: In the forest area of western Côte d'Ivoire, inland valley rice cultivation was not significantly associated with malaria burden

The key role of nitric oxide in hypoxia: hypoxic vasodilation and energy supply-demand matching

Significance: a mismatch between energy supply and demand induces tissue hypoxia with the potential to cause cell death and organ failure. Whenever arterial oxygen concentration is reduced, increases in blood flow - 'hypoxic vasodilation' - occur in an attempt to restore oxygen supply. Nitric oxide is a major signalling and effector molecule mediating the body's response to hypoxia, given its unique characteristics of vasodilation (improving blood flow and oxygen supply) and modulation of energetic metabolism (reducing oxygen consumption and promoting utilization of alternative pathways). Recent advances: this review covers the role of oxygen in metabolism and responses to hypoxia, the hemodynamic and metabolic effects of nitric oxide, and mechanisms underlying the involvement of nitric oxide in hypoxic vasodilation. Recent insights into nitric oxide metabolism will be discussed, including the role for dietary intake of nitrate, endogenous nitrite reductases, and release of nitric oxide from storage pools. The processes through which nitric oxide levels are elevated during hypoxia are presented, namely (i) increased synthesis from nitric oxide synthases, increased reduction of nitrite to nitric oxide by heme- or pterin-based enzymes and increased release from nitric oxide stores, and (ii) reduced deactivation by mitochondrial cytochrome c oxidase. Critical issues: several reviews covered modulation of energetic metabolism by nitric oxide, while here we highlight the crucial role NO plays in achieving cardiocirculatory homeostasis during acute hypoxia through both vasodilation and metabolic suppression Future directions: we identify a key position for nitric oxide in the body's adaptation to an acute energy supply-demand mismatc

Numerical analysis of the available power in an overtopping wave energy converter subjected to a sea state of the Coastal Region of Tramandaí, Brazil

The present work proposes a numerical study of an overtopping wave energy converter. The goal of this study is to evaluate the theoretical power that can be converted by an overtopping device subjected to sea waves in the coastal region of Tramandaí, Brazil. For this, realistic irregular waves were generated using theWaveMIMO methodology, which allows numerical simulation of sea waves through the imposition of transient discrete data as prescribed velocity. For the numerical analysis, a two-dimensional computational model was employed using Fluent, where the device was inserted into a wave channel. The volume of the fluid multiphase model was used for the treatment of the air–water interaction. The results indicated that the free surface elevation obtained using the WaveMIMO methodology, which converts a realistic sea state into a free surface elevation series, was adequately represented. The evaluation of the theoretical power of the overtopping device during around 45 min indicated that 471.28 W was obtained. In addition, a monthly generation projection showed that this device would supply 100% of the electricity demand of a school in the city of Tramandaí. These results demonstrated that the conversion of sea wave energy into electrical energy can contribute to supplying electricity demand, especially for coastal cities

ALMA captures feeding and feedback from the active galactic nucleus in NGC 613

We report ALMA observations of CO(3-2) emission in the Seyfert/nuclear starburst galaxy NGC 613, at a spatial resolution of 17 pc, as part of our NUclei of GAlaxies (NUGA) sample. Our aim is to investigate the morphology and dynamics of the gas inside the central kiloparsec, and to probe nuclear fueling and feedback phenomena. The morphology of CO(3-2) line emission reveals a two-arm trailing nuclear spiral at r≤ 100 pc and a circumnuclear ring at a radius of ∼350 pc that is coincident with the star-forming ring seen in the optical images. Also, we find evidence for a filamentary structure connecting the ring and the nuclear spiral. The ring reveals two breaks into two winding spiral arms corresponding to the dust lanes in the optical images. The molecular gas in the galaxy disk is in a remarkably regular rotation, however the kinematics in the nuclear region are very skewed. The nuclear spectrum of CO and dense gas tracers HCN(4-3), HCO+(4-3), and CS(7-6) show broad wings up to \ub1300 km s-1, associated with a molecular outflow emanating from the nucleus (r ∼ 25 pc). We derive a molecular outflow mass Mout=2 7 106 M⊙ and a mass outflow rate of M out = 27 M⊙ yr-1. The molecular outflow energetics exceed the values predicted by AGN feedback models: the kinetic power of the outflow corresponds to PK, out=20%LAGN and the momentum rate is M outv ∼400LAGN/c. The outflow is mainly boosted by the AGN through entrainment by the radio jet, but given the weak nuclear activity of NGC 613, we might be witnessing a fossil outflow resulting from a previously strong AGN that has now faded. Furthermore, the nuclear trailing spiral observed in CO emission is inside the inner Lindblad resonance ring of the bar. We compute the gravitational torques exerted in the gas to estimate the efficiency of the angular momentum exchange. The gravity torques are negative from 25 to 100 pc and the gas loses its angular momentum in a rotation period, providing evidence for a highly efficient inflow towards the center. This phenomenon shows that the massive central black hole has significant dynamical influence on the gas, triggering the inflowing of molecular gas to feed the black hole

An expression signature of the angiogenic response in gastrointestinal neuroendocrine tumours: correlation with tumour phenotype and survival outcomes.

BACKGROUND: Gastroenteropancreatic neuroendocrine tumours (GEP-NETs) are heterogeneous with respect to biological behaviour and prognosis. As angiogenesis is a renowned pathogenic hallmark as well as a therapeutic target, we aimed to investigate the prognostic and clinico-pathological role of tissue markers of hypoxia and angiogenesis in GEP-NETs. METHODS: Tissue microarray (TMA) blocks were constructed with 86 tumours diagnosed from 1988 to 2010. Tissue microarray sections were immunostained for hypoxia inducible factor 1α (Hif-1α), vascular endothelial growth factor-A (VEGF-A), carbonic anhydrase IX (Ca-IX) and somatostatin receptors (SSTR) 1–5, Ki-67 and CD31. Biomarker expression was correlated with clinico-pathological variables and tested for survival prediction using Kaplan–Meier and Cox regression methods. RESULTS: Eighty-six consecutive cases were included: 51% male, median age 51 (range 16–82), 68% presenting with a pancreatic primary, 95% well differentiated, 51% metastatic. Higher grading (P=0.03), advanced stage (P<0.001), high Hif-1α and low SSTR-2 expression (P=0.03) predicted for shorter overall survival (OS) on univariate analyses. Stage, SSTR-2 and Hif-1α expression were confirmed as multivariate predictors of OS. Median OS for patients with SSTR-2+/Hif-1α-tumours was not reached after median follow up of 8.8 years, whereas SSTR-2-/Hif-1α+ GEP-NETs had a median survival of only 4.2 years (P=0.006). CONCLUSION: We have identified a coherent expression signature by immunohistochemistry that can be used for patient stratification and to optimise treatment decisions in GEP-NETs independently from stage and grading. Tumours with preserved SSTR-2 and low Hif-1α expression have an indolent phenotype and may be offered less aggressive management and less stringent follow up