New prioritized value iteration for Markov decision processes

The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach. © Springer Science+Business Media B.V. 2011.García Hernández, MDG.; Ruiz Pinales, J.; Onaindia De La Rivaherrera, E.; Aviña Cervantes, JG.; Ledesma Orozco, S.; Alvarado Mendez, E.; Reyes Ballesteros, A. (2012). New prioritized value iteration for Markov decision processes. Artificial Intelligence Review. 37(2):157-167. doi:10.1007/s10462-011-9224-zS157167372Agrawal S, Roth D (2002) Learning a sparse representation for object detection. In: Proceedings of the 7th European conference on computer vision. Copenhagen, Denmark, pp 1–15Bellman RE (1954) The theory of dynamic programming. Bull Amer Math Soc 60: 503–516Bellman RE (1957) Dynamic programming. Princeton University Press, New JerseyBertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, MassachusettsBhuma K, Goldsmith J (2003) Bidirectional LAO* algorithm. In: Proceedings of indian international conferences on artificial intelligence. p 980–992Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36: 226–235Bonet B, Geffner H (2003a) Faster heuristic search algorithms for planning with uncertainty and full feedback. In: Proceedings of the 18th international joint conference on artificial intelligence. Morgan Kaufmann, Acapulco, México, pp 1233–1238Bonet B, Geffner H (2003b) Labeled RTDP: improving the convergence of real-time dynamic programming. In: Proceedings of the international conference on automated planning and scheduling. Trento, Italy, pp 12–21Bonet B, Geffner H (2006) Learning depth-first search: a unified approach to heuristic search in deterministic and non-deterministic settings and its application to MDP. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UKBoutilier C, Dean T, Hanks S (1999) Decision-theoretic planning: structural assumptions and computational leverage. J Artif Intell Res 11: 1–94Chang I, Soo H (2007) Simulation-based algorithms for Markov decision processes Communications and control engineering. Springer, LondonDai P, Goldsmith J (2007a) Faster dynamic programming for Markov decision processes. Technical report. Doctoral consortium, department of computer science and engineering. University of WashingtonDai P, Goldsmith J (2007b) Topological value iteration algorithm for Markov decision processes. In: Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 1860–1865Dai P, Hansen EA (2007c) Prioritizing bellman backups without a priority queue. In: Proceedings of the 17th international conference on automated planning and scheduling, association for the advancement of artificial intelligence. Rhode Island, USA, pp 113–119Dibangoye JS, Chaib-draa B, Mouaddib A (2008) A Novel prioritization technique for solving Markov decision processes. In: Proceedings of the 21st international FLAIRS (The Florida Artificial Intelligence Research Society) conference, association for the advancement of artificial intelligence. Florida, USAFerguson D, Stentz A (2004) Focused propagation of MDPs for path planning. In: Proceedings of the 16th IEEE international conference on tools with artificial intelligence. pp 310–317Hansen EA, Zilberstein S (2001) LAO: a heuristic search algorithm that finds solutions with loops. Artif Intell 129: 35–62Hinderer K, Waldmann KH (2003) The critical discount factor for finite Markovian decision processes with an absorbing set. Math Methods Oper Res 57: 1–19Li L (2009) A unifying framework for computational reinforcement learning theory. PhD Thesis. The state university of New Jersey, New Brunswick. NJLittman ML, Dean TL, Kaelbling LP (1995) On the complexity of solving Markov decision problems.In: Proceedings of the 11th international conference on uncertainty in artificial intelligence. Montreal, Quebec pp 394–402McMahan HB, Gordon G (2005a) Fast exact planning in Markov decision processes. In: Proceedings of the 15th international conference on automated planning and scheduling. Monterey, CA, USAMcMahan HB, Gordon G (2005b) Generalizing Dijkstra’s algorithm and gaussian elimination for solving MDPs. Technical report, Carnegie Mellon University, PittsburghMeuleau N, Brafman R, Benazera E (2006) Stochastic over-subscription planning using hierarchies of MDPs. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UK, pp 121–130Moore A, Atkeson C (1993) Prioritized sweeping: reinforcement learning with less data and less real time. Mach Learn 13: 103–130Puterman ML (1994) Markov decision processes. Wiley Editors, New YorkPuterman ML (2005) Markov decision processes. Wiley Inter Science Editors, New YorkRussell S (2005) Artificial intelligence: a modern approach. Making complex decisions (Ch-17), 2nd edn. Pearson Prentice Hill Ed., USAShani G, Brafman R, Shimony S (2008) Prioritizing point-based POMDP solvers. IEEE Trans Syst Man Cybern 38(6): 1592–1605Sniedovich M (2006) Dijkstra’s algorithm revisited: the dynamic programming connexion. Control Cybern 35: 599–620Sniedovich M (2010) Dynamic programming: foundations and principles, 2nd edn. Pure and Applied Mathematics Series, UKTijms HC (2003) A first course in stochastic models. Discrete-time Markov decision processes (Ch-6). Wiley Editors, UKVanderbei RJ (1996) Optimal sailing strategies. Statistics and operations research program, University of Princeton, USA ( http://www.orfe.princeton.edu/~rvdb/sail/sail.html )Vanderbei RJ (2008) Linear programming: foundations and extensions, 3rd edn. Springer, New YorkWingate D, Seppi KD (2005) Prioritization methods for accelerating MDP solvers. J Mach Learn Res 6: 851–88

Differential Effects of High-Carbohydrate and High-Fat Diet Composition on Metabolic Control and Insulin Resistance in Normal Rats

The macronutrient component of diets is critical for metabolic control and insulin action. The aim of this study was to compare the effects of high fat diets (HFDs) vs. high carbohydrate diets (HCDs) on metabolic control and insulin resistance in Wistar rats. Thirty animals divided into five groups (n = 6) were fed: (1) Control diet (CD); (2) High-saturated fat diet (HSFD); (3) High-unsaturated fat diet (HUFD); (4) High-digestible starch diet, (HDSD); and (5) High-resistant starch diet (HRSD) during eight weeks. HFDs and HCDs reduced weight gain in comparison with CD, however no statistical significance was reached. Calorie intake was similar in both HFDs and CD, but rats receiving HCDs showed higher calorie consumption than other groups, (p < 0.01). HRSD showed the lowest levels of serum and hepatic lipids. The HUFD induced the lowest fasting glycemia levels and HOMA-IR values. The HDSD group exhibited the highest insulin resistance and hepatic cholesterol content. In conclusion, HUFD exhibited the most beneficial effects on glycemic control meanwhile HRSD induced the highest reduction on lipid content and did not modify insulin sensitivity. In both groups, HFDs and HCDs, the diet constituents were more important factors than caloric intake for metabolic disturbance and insulin resistance

Height and timing of growth spurt during puberty in young people living with vertically acquired HIV in Europe and Thailand.

OBJECTIVE: The aim of this study was to describe growth during puberty in young people with vertically acquired HIV. DESIGN: Pooled data from 12 paediatric HIV cohorts in Europe and Thailand. METHODS: One thousand and ninety-four children initiating a nonnucleoside reverse transcriptase inhibitor or boosted protease inhibitor based regimen aged 1-10 years were included. Super Imposition by Translation And Rotation (SITAR) models described growth from age 8 years using three parameters (average height, timing and shape of the growth spurt), dependent on age and height-for-age z-score (HAZ) (WHO references) at antiretroviral therapy (ART) initiation. Multivariate regression explored characteristics associated with these three parameters. RESULTS: At ART initiation, median age and HAZ was 6.4 [interquartile range (IQR): 2.8, 9.0] years and -1.2 (IQR: -2.3 to -0.2), respectively. Median follow-up was 9.1 (IQR: 6.9, 11.4) years. In girls, older age and lower HAZ at ART initiation were independently associated with a growth spurt which occurred 0.41 (95% confidence interval 0.20-0.62) years later in children starting ART age 6 to 10 years compared with 1 to 2 years and 1.50 (1.21-1.78) years later in those starting with HAZ less than -3 compared with HAZ at least -1. Later growth spurts in girls resulted in continued height growth into later adolescence. In boys starting ART with HAZ less than -1, growth spurts were later in children starting ART in the oldest age group, but for HAZ at least -1, there was no association with age. Girls and boys who initiated ART with HAZ at least -1 maintained a similar height to the WHO reference mean. CONCLUSION: Stunting at ART initiation was associated with later growth spurts in girls. Children with HAZ at least -1 at ART initiation grew in height at the level expected in HIV negative children of a comparable age

Efficacy and safety of acupuncture for the treatment of non-specific acute low back pain: a randomised controlled multicentre trial protocol [ISRCTN65814467]

BACKGROUND: Low back pain and its associated incapacitating effects constitute an important healthcare and socioeconomic problem, as well as being one of the main causes of disability among adults of working age. The prevalence of non-specific low back pain is very high among the general population, and 60–70% of adults are believed to have suffered this problem at some time. Nevertheless, few randomised clinical trials have been made of the efficacy and efficiency of acupuncture with respect to acute low back pain. The present study is intended to assess the efficacy of acupuncture for acute low back pain in terms of the improvement reported on the Roland Morris Questionnaire (RMQ) on low back pain incapacity, to estimate the specific and non-specific effects produced by the technique, and to carry out a cost-effectiveness analysis. METHODS/DESIGN: Randomised four-branch controlled multicentre prospective study made to compare semi-standardised real acupuncture, sham acupuncture (acupuncture at non-specific points), placebo acupuncture and conventional treatment. The patients are blinded to the real, sham and placebo acupuncture treatments. Patients in the sample present symptoms of non specific acute low back pain, with a case history of 2 weeks or less, and will be selected from working-age patients, whether in paid employment or not, referred by General Practitioners from Primary Healthcare Clinics to the four clinics participating in this study. In order to assess the primary and secondary result measures, the patients will be requested to fill in a questionnaire before the randomisation and again at 3, 12 and 48 weeks after starting the treatment. The primary result measure will be the clinical relevant improvement (CRI) at 3 weeks after randomisation. We define CRI as a reduction of 35% or more in the RMQ results. DISCUSSION: This study is intended to obtain further evidence on the effectiveness of acupuncture on acute low back pain and to isolate the specific and non-specific effects of the treatment

Overview of recent TJ-II stellarator results

The main results obtained in the TJ-II stellarator in the last two years are reported. The most important topics investigated have been modelling and validation of impurity transport, validation of gyrokinetic simulations, turbulence characterisation, effect of magnetic configuration on transport, fuelling with pellet injection, fast particles and liquid metal plasma facing components. As regards impurity transport research, a number of working lines exploring several recently discovered effects have been developed: the effect of tangential drifts on stellarator neoclassical transport, the impurity flux driven by electric fields tangent to magnetic surfaces and attempts of experimental validation with Doppler reflectometry of the variation of the radial electric field on the flux surface. Concerning gyrokinetic simulations, two validation activities have been performed, the comparison with measurements of zonal flow relaxation in pellet-induced fast transients and the comparison with experimental poloidal variation of fluctuations amplitude. The impact of radial electric fields on turbulence spreading in the edge and scrape-off layer has been also experimentally characterized using a 2D Langmuir probe array. Another remarkable piece of work has been the investigation of the radial propagation of small temperature perturbations using transfer entropy. Research on the physics and modelling of plasma core fuelling with pellet and tracer-encapsulated solid-pellet injection has produced also relevant results. Neutral beam injection driven Alfvénic activity and its possible control by electron cyclotron current drive has been examined as well in TJ-II. Finally, recent results on alternative plasma facing components based on liquid metals are also presentedThis work has been carried out within the framework of the EUROfusion Consortium and has received funding from the Euratom research and training programme 2014–2018 under Grant Agreement No. 633053. It has been partially funded by the Ministerio de Ciencia, Inovación y Universidades of Spain under projects ENE2013-48109-P, ENE2015-70142-P and FIS2017-88892-P. It has also received funds from the Spanish Government via mobility grant PRX17/00425. The authors thankfully acknowledge the computer resources at MareNostrum and the technical support provided by the Barcelona S.C. It has been supported as well by The Science and Technology Center in Ukraine (STCU), Project P-507F

Phylogenetic history demonstrates two different lineages of dengue type 1 virus in Colombia

Background: Dengue Fever is one of the most important viral re-emergent diseases affecting about 50 million people around the world especially in tropical and sub-tropical countries. In Colombia, the virus was first detected in the earliest 70′s when the disease became a major public health concern. Since then, all four serotypes of the virus have been reported. Although most of the huge outbreaks reported in this country have involved dengue virus serotype 1 (DENV-1), there are not studies about its origin, genetic diversity and distribution. Results: We used 224 bp corresponding to the carboxyl terminus of envelope (E) gene from 74 Colombian isolates in order to reconstruct phylogenetic relationships and to estimate time divergences. Analyzed DENV-1 Colombian isolates belonged to the formerly defined genotype V. Only one virus isolate was clasified in the genotype I, likely representing a sole introduction that did not spread. The oldest strains were closely related to those detected for the first time in America in 1977 from the Caribbean and were detected for two years until their disappearance about six years later. Around 1987, a split up generated 2 lineages that have been evolving separately, although not major aminoacid changes in the analyzed region were found. Conclusion: DENV-1 has been circulating since 1978 in Colombia. Yet, the phylogenetic relationships between strains isolated along the covered period of time suggests that viral strains detected in some years, although belonging to the same genotype V, have different recent origins corresponding to multiple re-introduction events of viral strains that were circulating in neighbor countries. Viral strains used in the present study did not form a monophyletic group, which is evidence of a polyphyletic origin. We report the rapid spread patterns and high evolution rate of the different DENV-1 lineages

Prognostic implications of comorbidity patterns in critically ill COVID-19 patients: A multicenter, observational study

Background The clinical heterogeneity of COVID-19 suggests the existence of different phenotypes with prognostic implications. We aimed to analyze comorbidity patterns in critically ill COVID-19 patients and assess their impact on in-hospital outcomes, response to treatment and sequelae. Methods Multicenter prospective/retrospective observational study in intensive care units of 55 Spanish hospitals. 5866 PCR-confirmed COVID-19 patients had comorbidities recorded at hospital admission; clinical and biological parameters, in-hospital procedures and complications throughout the stay; and, clinical complications, persistent symptoms and sequelae at 3 and 6 months. Findings Latent class analysis identified 3 phenotypes using training and test subcohorts: low-morbidity (n=3385; 58%), younger and with few comorbidities; high-morbidity (n=2074; 35%), with high comorbid burden; and renal-morbidity (n=407; 7%), with chronic kidney disease (CKD), high comorbidity burden and the worst oxygenation profile. Renal-morbidity and high-morbidity had more in-hospital complications and higher mortality risk than low-morbidity (adjusted HR (95% CI): 1.57 (1.34-1.84) and 1.16 (1.05-1.28), respectively). Corticosteroids, but not tocilizumab, were associated with lower mortality risk (HR (95% CI) 0.76 (0.63-0.93)), especially in renal-morbidity and high-morbidity. Renal-morbidity and high-morbidity showed the worst lung function throughout the follow-up, with renal-morbidity having the highest risk of infectious complications (6%), emergency visits (29%) or hospital readmissions (14%) at 6 months (p<0.01). Interpretation Comorbidity-based phenotypes were identified and associated with different expression of in-hospital complications, mortality, treatment response, and sequelae, with CKD playing a major role. This could help clinicians in day-to-day decision making including the management of post-discharge COVID-19 sequelae. Copyright (C) 2022 The Author(s). Published by Elsevier Ltd

Segmented flow generator for serial crystallography at the European X-ray free electron laser

Serial femtosecond crystallography (SFX) with X-ray free electron lasers (XFELs) allows structure determination of membrane proteins and time-resolved crystallography. Common liquid sample delivery continuously jets the protein crystal suspension into the path of the XFEL, wasting a vast amount of sample due to the pulsed nature of all current XFEL sources. The European XFEL (EuXFEL) delivers femtosecond (fs) X-ray pulses in trains spaced 100 ms apart whereas pulses within trains are currently separated by 889 ns. Therefore, continuous sample delivery via fast jets wastes >99% of sample. Here, we introduce a microfluidic device delivering crystal laden droplets segmented with an immiscible oil reducing sample waste and demonstrate droplet injection at the EuXFEL compatible with high pressure liquid delivery of an SFX experiment. While achieving ~60% reduction in sample waste, we determine the structure of the enzyme 3-deoxy-D-manno-octulosonate-8-phosphate synthase from microcrystals delivered in droplets revealing distinct structural features not previously reported