On the Whittle Index for Restless Multi-armed Hidden Markov Bandits

We consider a restless multi-armed bandit in which each arm can be in one of two states. When an arm is sampled, the state of the arm is not available to the sampler. Instead, a binary signal with a known randomness that depends on the state of the arm is available. No signal is available if the arm is not sampled. An arm-dependent reward is accrued from each sampling. In each time step, each arm changes state according to known transition probabilities which in turn depend on whether the arm is sampled or not sampled. Since the state of the arm is never visible and has to be inferred from the current belief and a possible binary signal, we call this the hidden Markov bandit. Our interest is in a policy to select the arm(s) in each time step that maximizes the infinite horizon discounted reward. Specifically, we seek the use of Whittle's index in selecting the arms. We first analyze the single-armed bandit and show that in general, it admits an approximate threshold-type optimal policy when there is a positive reward for the `no-sample' action. We also identify several special cases for which the threshold policy is indeed the optimal policy. Next, we show that such a single-armed bandit also satisfies an approximate-indexability property. For the case when the single-armed bandit admits a threshold-type optimal policy, we perform the calculation of the Whittle index for each arm. Numerical examples illustrate the analytical results.Comment: Revised version, corrected few typo

Scalable Text and Link Analysis with Mixed-Topic Link Models

Many data sets contain rich information about objects, as well as pairwise relations between them. For instance, in networks of websites, scientific papers, and other documents, each node has content consisting of a collection of words, as well as hyperlinks or citations to other nodes. In order to perform inference on such data sets, and make predictions and recommendations, it is useful to have models that are able to capture the processes which generate the text at each node and the links between them. In this paper, we combine classic ideas in topic modeling with a variant of the mixed-membership block model recently developed in the statistical physics community. The resulting model has the advantage that its parameters, including the mixture of topics of each document and the resulting overlapping communities, can be inferred with a simple and scalable expectation-maximization algorithm. We test our model on three data sets, performing unsupervised topic classification and link prediction. For both tasks, our model outperforms several existing state-of-the-art methods, achieving higher accuracy with significantly less computation, analyzing a data set with 1.3 million words and 44 thousand links in a few minutes.Comment: 11 pages, 4 figure

Spin Gaps in Coupled t-J Ladders

Spin gaps in coupled $t$-$J$ ladders are investigated by exact diagonalization of small clusters up to 4$\times$8 sites. At half-filling, the numerical results for the triplet excitation spectrum are in very good agreement with a second order perturbation expansion in term of small inter-ladder and intra-ladder exchange couplings between rungs ($J/J^\prime$$<$$0.25$). The band of local triplet excitations moving coherently along the ladder (with momenta close to $\pi$) is split by the inter-ladder coupling. For intermediate couplings finite size scaling is used to estimate the spin gap. In the isotropic infinite 4-chain system (two coupled ladders) we find a spin gap of $0.245 J$, roughly half of the single ladder spin gap. When the system is hole doped, bonding and anti-bonding bound pairs of holes can propagate coherently along the chains and the spin gap remains finite.Comment: 11 pages, 5 figures, uuencoded form of postscript files of figures and text, LPQTH-94/

Comparing the effectiveness of small-particle versus large-particle inhaled corticosteroid in COPD

Dirkje S Postma,1 Nicolas Roche,2 Gene Colice,3 Elliot Israel,4 Richard J Martin,5 Willem MC van Aalderen,6 Jonathan Grigg,7 Anne Burden,8 Elizabeth V Hillyer,8 Julie von Ziegenweidt,8 Gokul Gopalan,9 David Price8,10 1University of Groningen, Department of Pulmonary Medicine and Tuberculosis, University Medical Center Groningen, Groningen, the Netherlands; 2Respiratory and Intensive Care Medicine, Cochin Hospital Group, APHP, Paris-Descartes University (EA2511), Paris, France; 3Pulmonary, Critical Care and Respiratory Services, Washington Hospital Center and George Washington University School of Medicine, Washington DC, USA; 4Pulmonary and Critical Care Division, Brigham and Women&#39;s Hospital, Harvard Medical School, Boston, MA, USA; 5Department of Medicine, National Jewish Health, Denver, CO, USA; 6Dept of Pediatric Respiratory Medicine and Allergy, Emma Children&#39;s Hospital AMC, Amsterdam, the Netherlands; 7Blizard Institute, Queen Mary University London, London, UK; 8Research in Real Life, Ltd, Cambridge, UK; 9Respiratory, Global Scientific Affairs, Teva Pharmaceuticals, Frazer, PA, USA; 10Academic Primary Care, Division of Applied Health Sciences, University of Aberdeen, Aberdeen, UK Purpose: Small airway changes and dysfunction contribute importantly to airway obstruction in chronic obstructive pulmonary disease (COPD), which is currently treated with inhaled corticosteroids (ICS) and long-acting bronchodilators at Global initiative for Obstructive Lung Disease (GOLD) grades 2&ndash;4. This retrospective matched cohort analysis compared effectiveness of a representative small-particle ICS (extrafine beclomethasone) and larger-particle ICS (fluticasone) in primary care patients with COPD. Patients and methods: Smokers and ex-smokers with COPD &ge;40 years old initiating or stepping-up their dose of extrafine beclomethasone or fluticasone were matched 1:1 for demographic characteristics, index prescription year, concomitant therapies, and disease severity during 1 baseline year. During 2 subsequent years, we evaluated treatment change and COPD exacerbations, defined as emergency care/hospitalization for COPD, acute oral corticosteroids, or antibiotics for lower respiratory tract infection. Results: Mean patient age was 67 years, 57%&ndash;60% being male. For both initiation (n=334:334) and step-up (n=189:189) patients, exacerbation rates were comparable between extrafine beclomethasone and fluticasone cohorts during the 2 year outcome period. Odds of treatment stability (no exacerbation or treatment change) were significantly greater for patients initiating extrafine beclomethasone compared with fluticasone (adjusted odds ratio 2.50; 95% confidence interval, 1.32&ndash;4.73). Median ICS dose exposure during 2 outcome years was significantly lower (P&lt;0.001) for extrafine beclomethasone than fluticasone cohorts (315 &micro;g/day versus 436 &micro;g/day for initiation, 438 &micro;g/day versus 534 &micro;g/day for step-up patients). Conclusion: We observed that small-particle ICS at significantly lower doses had comparable effects on exacerbation rates as larger-particle ICS at higher doses, whereas initiation of small-particle ICS was associated with better odds of treatment stability during 2-years&#39; follow-up. Keywords: COPD exacerbation, extrafine particle, matched cohort analysis, real life, small airway