Multi-Armed Bandits for Correlated Markovian Environments with Smoothed Reward Feedback

We study a multi-armed bandit problem in a dynamic environment where arm rewards evolve in a correlated fashion according to a Markov chain. Different than much of the work on related problems, in our formulation a learning algorithm does not have access to either a priori information or observations of the state of the Markov chain and only observes smoothed reward feedback following time intervals we refer to as epochs. We demonstrate that existing methods such as UCB and $\varepsilon$-greedy can suffer linear regret in such an environment. Employing mixing-time bounds on Markov chains, we develop algorithms called EpochUCB and EpochGreedy that draw inspiration from the aforementioned methods, yet which admit sublinear regret guarantees for the problem formulation. Our proposed algorithms proceed in epochs in which an arm is played repeatedly for a number of iterations that grows linearly as a function of the number of times an arm has been played in the past. We analyze these algorithms under two types of smoothed reward feedback at the end of each epoch: a reward that is the discount-average of the discounted rewards within an epoch, and a reward that is the time-average of the rewards within an epoch.Comment: Significant revision of prior version including deeper discussion of related work, gap-independent regret bounds, and regret bounds for discounted reward

The geographic distribution of onchocerciasis in the 20 participating countries of the African Programme for Onchocerciasis Control:(2) pre-control endemicity levels and estimated number infected

BACKGROUND: The original aim of the African Programme for Onchocerciasis Control (APOC) was to control onchocerciasis as a public health problem in 20 African countries. In order to identify all high risk areas where ivermectin treatment was needed to achieve control, APOC used Rapid Epidemiological Mapping of Onchocerciasis (REMO). REMO involved spatial sampling of villages to be surveyed, and examination of 30 to 50 adults per village for palpable onchocercal nodules. REMO has now been virtually completed and we report the results in two articles. A companion article reports the delineation of high risk areas based on expert analysis. The present article reports the results of a geostatistical analysis of the REMO data to map endemicity levels and estimate the number infected. METHODS: A model-based geostatistical analysis of the REMO data was undertaken to generate high-resolution maps of the predicted prevalence of nodules and of the probability that the true nodule prevalence exceeds the high risk threshold of 20%. The number infected was estimated by converting nodule prevalence to microfilaria prevalence, and multiplying the predicted prevalence for each location with local data on population density. The geostatistical analysis included the nodule palpation data for 14,473 surveyed villages. RESULTS: The generated map of onchocerciasis endemicity levels, as reflected in the prevalence of nodules, is a significant advance with many new endemic areas identified. The prevalence of nodules was > 20% over an area of 2.5 million km2 with an estimated population of 62 million people. The results were consistent with the delineation of high risk areas of the expert analysis except for borderline areas where the prevalence fluctuated around 20%. It is estimated that 36 million people would have been infected in the APOC countries by 2011 if there had been no ivermectin treatment. CONCLUSIONS: The map of onchocerciasis endemicity levels has proven very valuable for onchocerciasis control in the APOC countries. Following the recent shift to onchocerciasis elimination, the map continues to play an important role in planning treatment, evaluating impact and predicting treatment end dates in relation to local endemicity levels

Progress towards onchocerciasis elimination in the participating countries of the African Programme for Onchocerciasis Control: Epidemiological evaluation results

Background: The African Programme for Onchocerciasis Control (APOC) was created in 1995 to establish community-directed treatment with ivermectin (CDTi) in order to control onchocerciasis as a public health problem in 20 African countries that had 80 % of the global disease burden. When research showed that CDTi may ultimately eliminate onchocerciasis infection, APOC was given in 2008 the additional objective to determine when and where treatment can be safely stopped. We report the results of epidemiological evaluations undertaken from 2008 to 2014 to assess progress towards elimination in CDTi areas with ≥6 years treatment. Methods: Skin snip surveys were undertaken in samples of first-line villages to determine the prevalence of O. volvulus microfilariae. There were two evaluation phases. The decline in prevalence was evaluated in phase 1A. Observed and model-predicted prevalences were compared after correcting for endemicity level and treatment coverage. Bayesian statistics and Monte Carlo simulation were used to classify the decline in prevalence as faster than predicted, on track or delayed. Where the prevalence approached elimination levels, phase 1B was launched to determine if treatment could be safely stopped. Village sampling was extended to the whole CDTi area. Survey data were analysed within a Bayesian framework to determine if stopping criteria (overall prevalence <1.4 % and maximum stratum prevalence <5 %) were met. Results: In phase 1A 127 665 people from 639 villages in 54 areas were examined. The prevalence had fallen dramatically. The decline in prevalence was faster than predicted in 23 areas, on track in another 23 and delayed in eight areas. In phase 1B 108 636 people in 392 villages were examined in 22 areas of which 13 met the epidemiological criteria for stopping treatment. Overall, 32 areas (25.4 million people) had reached or were close to elimination, 18 areas (17.4 million) were on track but required more years treatment, and in eight areas (10.4 million) progress was unsatisfactory. Conclusions: Onchocerciasis has been largely controlled as a public health problem. Great progress has been made towards elimination which already appears to have been achieved for millions of people. For most APOC countries, nationwide onchocerciasis elimination is within reach

African Program for Onchocerciasis Control 1995–2010: Impact of Annual Ivermectin Mass Treatment on Off-Target Infectious Diseases

Since its initiation in 1995, the African Program for Onchocerciasis Control (APOC) has had a substantial impact on the prevalence and burden of onchocerciasis through annual ivermectin mass treatment. Ivermectin is a broad-spectrum anti-parasitic agent that also has an impact on other co-endemic parasitic infections. In this study, we roughly assessed the additional impact of APOC activities on the burden of the most important off-target infections: soil-transmitted helminthiases (STH; ascariasis, trichuriasis, hookworm, and strongyloidiasis), lymphatic filariasis (LF), and scabies. Based on a literature review, we formulated assumptions about the impact of ivermectin treatment on the disease burden of these off-target infections. Using data on the number of ivermectin treatments in APOC regions and the latest estimates of the burden of disease, we then calculated the impact of APOC activities on off-target infections in terms of disability-adjusted life years (DALYs) averted. We conservatively estimated that between 1995 and 2010, annual ivermectin mass treatment has cumulatively averted about 500 thousand DALYs from co-endemic STH infections, LF, and scabies. This impact comprised approximately an additional 5.5% relative to the total burden averted from onchocerciasis (8.9 million DALYs) and indicates that the overall cost-effectiveness of APOC is even higher than previously reported

The Lactic Acid Bacterium Pediococcus acidilactici Suppresses Autoimmune Encephalomyelitis by Inducing IL-10-Producing Regulatory T Cells

BACKGROUND: Certain intestinal microflora are thought to regulate the systemic immune response. Lactic acid bacteria are one of the most studied bacteria in terms of their beneficial effects on health and autoimmune diseases; one of which is Multiple sclerosis (MS) which affects the central nervous system. We investigated whether the lactic acid bacterium Pediococcus acidilactici, which comprises human commensal bacteria, has beneficial effects on experimental autoimmune encephalomyelitis (EAE), an animal model of MS. METHODOLOGY/PRINCIPAL FINDINGS: P. acidilactici R037 was orally administered to EAE mice to investigate the effects of R037. R037 treatment suppressed clinical EAE severity as prophylaxis and therapy. The antigen-specific production of inflammatory cytokines was inhibited in R037-treated mice. A significant increase in the number of CD4(+) Interleukin (IL)-10-producing cells was observed in the mesenteric lymph nodes (MLNs) and spleens isolated from R037-treated naive mice, while no increase was observed in the number of these cells in the lamina propria. Because only a slight increase in the CD4(+)Foxp3(+) cells was observed in MLNs, R037 may primarily induce Foxp3(-) IL10-producing T regulatory type 1 (Tr1) cells in MLNs, which contribute to the beneficial effect of R037 on EAE. CONCLUSIONS/SIGNIFICANCE: An orally administered single strain of P. acidilactici R037 ameliorates EAE by inducing IL10-producing Tr1 cells. Our findings indicate the therapeutic potential of the oral administration of R037 for treating multiple sclerosis

Projected Number of People With Onchocerciasis-Loiasis Coinfection in Africa, 1995 to 2025

BACKGROUND: Onchocerciasis elimination through mass drug administration (MDA) is hampered by coendemicity of Loa loa, as people with high L. loa microfilariae (mf) density can develop serious adverse events (SAEs) after ivermectin treatment. We assessed the geographical overlap of onchocerciasis and loiasis prevalence and estimated the number of coinfected individuals at risk of post-ivermectin SAEs in West and Central Africa from 1995 to 2025. METHODS: Focusing on regions with suspected loiasis transmission in 14 countries, we overlaid precontrol maps of loiasis and onchocerciasis prevalence to calculate precontrol prevalence of coinfection by 5 km2 × 5 km2 pixel, distinguishing different categories of L. loa mf intensity. Using statistical and mathematical models, we predicted prevalence of both infections and coinfection for 2015 and 2025, accounting for the impact of MDA with ivermectin. RESULTS: The number of people infected with onchocerciasis was predicted to decline from almost 19 million in 1995 to 4 million in 2025. Of these, 137 000 people were estimated to also have L. loa hypermicrofilaremia (≥20 000 L. loa mf/mL) in 1995, declining to 31 000 in 2025. In 2025, 92.8% of coinfecte

Real-time control of distributed batteries with blockchain-enabled market export commitments

Recent years have seen a surge of interest in distributed residential batteries for households with renewable generation. Yet, assuring battery assets are profitable for their owners requires a complex optimisation of the battery asset and additional revenue sources, such as novel ways to access wholesale energy markets. In this paper, we propose a framework in which wholesale market bids are placed on forward energy markets by an aggregator of distributed residential batteries that are controlled in real time by a novel Home Energy Management System (HEMS) control algorithm to meet the market commitments, while maximising local self-consumption. The proposed framework consists of three stages. In the first stage, an optimal day-ahead or intra-day scheduling of the aggregated storage assets is computed centrally. For the second stage, a bidding strategy is developed for wholesale energy markets. Finally, in the third stage, a novel HEMS real-time control algorithm based on a smart contract allows coordination of residential batteries to meet the market commitments and maximise self-consumption of local production. Using a case study provided by a large UK-based energy demonstrator, we apply the framework to an aggregator with 70 residential batteries. Experimental analysis is done using real per minute data for demand and production. Results indicate that the proposed approach increases the aggregator’s revenues by 35% compared to a case without residential flexibility, and increases the self-consumption rate of the households by a factor of two. The robustness of the results to uncertainty, forecast errors and to communication latency is also demonstrated

Ising Universality in Three Dimensions: A Monte Carlo Study

We investigate three Ising models on the simple cubic lattice by means of Monte Carlo methods and finite-size scaling. These models are the spin-1/2 Ising model with nearest-neighbor interactions, a spin-1/2 model with nearest-neighbor and third-neighbor interactions, and a spin-1 model with nearest-neighbor interactions. The results are in accurate agreement with the hypothesis of universality. Analysis of the finite-size scaling behavior reveals corrections beyond those caused by the leading irrelevant scaling field. We find that the correction-to-scaling amplitudes are strongly dependent on the introduction of further-neighbor interactions or a third spin state. In a spin-1 Ising model, these corrections appear to be very small. This is very helpful for the determination of the universal constants of the Ising model. The renormalization exponents of the Ising model are determined as y_t = 1.587 (2), y_h = 2.4815 (15) and y_i = -0.82 (6). The universal ratio Q = ^2/ is equal to 0.6233 (4) for periodic systems with cubic symmetry. The critical point of the nearest-neighbor spin-1/2 model is K_c=0.2216546 (10).Comment: 25 pages, uuencoded compressed PostScript file (to appear in Journal of Physics A

How can onchocerciasis elimination in Africa be accelerated? Modelling the impact of increased ivermectin treatment frequency and complementary vector control

Background: Great strides have been made toward onchocerciasis elimination by mass drug administration (MDA) of ivermectin. Focusing on MDA-eligible areas, we investigated where the elimination goal can be achieved by 2025 by continuation of current practice (annual MDA with ivermectin) and where intensification or additional vector control is required. We did not consider areas hypoendemic for onchocerciasis with loiasis coendemicity where MDA is contraindicated. Methods: We used 2 previously published mathematical models, ONCHOSIM and EPIONCHO, to simulate future trends in microfilarial prevalence for 80 different settings (defined by precontrol endemicity and past MDA frequency and coverage) under different future treatment scenarios (annual, biannual, or quarterly MDA with different treatment coverage through 2025, with or without vector control strategies), assessing for each strategy whether it eventually leads to elimination. Results: Areas with 40%–50% precontrol microfilarial prevalence and ≥10 years of annual MDA may achieve elimination with a further 7 years of annual MDA, if not achieved already, according to both models. For most areas with 70%–80% precontrol prevalence, ONCHOSIM predicts that either annual or biannual MDA is sufficient to achieve elimination by 2025, whereas EPIONCHO predicts that elimination will not be achieved even with complementary vector control. Conclusions: Whether elimination will be reached by 2025 depends on precontrol endemicity, control history, and strategies chosen from now until 2025. Biannual or quarterly MDA will accelerate progress toward elimination but cannot guarantee it by 2025 in high-endemicity areas. Long-term concomitant MDA and vector control for high-endemicity areas might be useful