Mixed acceleration techniques for solving quickly stochastic shortest-path markov decision processes

In this paper we propose the combination of accelerated variants of value iteration mixed with improved prioritized sweeping for the fast solution of stochastic shortest-path Markov decision processes. Value iteration is a classical algorithm for solving Markov decision processes, but this algorithm and its variants are quite slow for solving considerably large problems. In order to improve the solution time, acceleration techniques such as asynchronous updates, prioritization and prioritized sweeping have been explored in this paper. A topological reordering algorithm was also compared with static reordering. Experimental results obtained on finite state and action-space stochastic shortest-path problems show that our approach achieves a considerable reduction in the solution time with respect to the tested variants of value iteration. For instance, the experiments showed in one test a reduction of 5.7 times with respect to value iteration with asynchronous updates.En este documento proponemos la combinación de variantes aceleradas del algoritmo de iteración de valor combinadas con el algoritmo de barrido priorizado mejorado para la rápida solución de los procesos de decisión de Markov de ruta estocástica más corta. Iteración de valor es un algoritmo clásico para resolver a los procesos de decisión de Markov, pero este algoritmo y sus variantes son lentos para resolver problemas considerablemente grandes. Con el objeto de mejorar el tiempo de solución de este algoritmo, en este documento se han explorado técnicas de aceleración tales como actualizaciones asíncronas, priorización y barrido priorizado. Un algoritmo de reordenamiento topológico también fue comparado con uno de reordenamiento estático. Los resultados experimentales obtenidos en un problema de ruta estocástica más corta con espacios de estados-acciones finitos; muestran que nuestro enfoque logra una considerable reducción en el tiempo de solución con respecto a las variantes de iteración de valor probadas. Por ejemplo, los experimentos mostraron en una prueba una reducción de 5.7 veces con respecto a iteración de valor usando actualizaciones asíncronas.García Hernández, MDG.; Ruiz Pinales, J.; Onaindia De La Rivaherrera, E.; Ledesma-Orozco, S.; Aviña-Cervantes, J.; Alvarado-Méndez, E.; Reyes-Ballesteros, A. (2011). Mixed acceleration techniques for solving quickly stochastic shortest-path markov decision processes. Journal of Applied Research and Technology. 9(2):129-144. http://hdl.handle.net/10251/46761S1291449

New prioritized value iteration for Markov decision processes

The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach. © Springer Science+Business Media B.V. 2011.García Hernández, MDG.; Ruiz Pinales, J.; Onaindia De La Rivaherrera, E.; Aviña Cervantes, JG.; Ledesma Orozco, S.; Alvarado Mendez, E.; Reyes Ballesteros, A. (2012). New prioritized value iteration for Markov decision processes. Artificial Intelligence Review. 37(2):157-167. doi:10.1007/s10462-011-9224-zS157167372Agrawal S, Roth D (2002) Learning a sparse representation for object detection. In: Proceedings of the 7th European conference on computer vision. Copenhagen, Denmark, pp 1–15Bellman RE (1954) The theory of dynamic programming. Bull Amer Math Soc 60: 503–516Bellman RE (1957) Dynamic programming. Princeton University Press, New JerseyBertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, MassachusettsBhuma K, Goldsmith J (2003) Bidirectional LAO* algorithm. In: Proceedings of indian international conferences on artificial intelligence. p 980–992Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36: 226–235Bonet B, Geffner H (2003a) Faster heuristic search algorithms for planning with uncertainty and full feedback. In: Proceedings of the 18th international joint conference on artificial intelligence. Morgan Kaufmann, Acapulco, México, pp 1233–1238Bonet B, Geffner H (2003b) Labeled RTDP: improving the convergence of real-time dynamic programming. In: Proceedings of the international conference on automated planning and scheduling. Trento, Italy, pp 12–21Bonet B, Geffner H (2006) Learning depth-first search: a unified approach to heuristic search in deterministic and non-deterministic settings and its application to MDP. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UKBoutilier C, Dean T, Hanks S (1999) Decision-theoretic planning: structural assumptions and computational leverage. J Artif Intell Res 11: 1–94Chang I, Soo H (2007) Simulation-based algorithms for Markov decision processes Communications and control engineering. Springer, LondonDai P, Goldsmith J (2007a) Faster dynamic programming for Markov decision processes. Technical report. Doctoral consortium, department of computer science and engineering. University of WashingtonDai P, Goldsmith J (2007b) Topological value iteration algorithm for Markov decision processes. In: Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 1860–1865Dai P, Hansen EA (2007c) Prioritizing bellman backups without a priority queue. In: Proceedings of the 17th international conference on automated planning and scheduling, association for the advancement of artificial intelligence. Rhode Island, USA, pp 113–119Dibangoye JS, Chaib-draa B, Mouaddib A (2008) A Novel prioritization technique for solving Markov decision processes. In: Proceedings of the 21st international FLAIRS (The Florida Artificial Intelligence Research Society) conference, association for the advancement of artificial intelligence. Florida, USAFerguson D, Stentz A (2004) Focused propagation of MDPs for path planning. In: Proceedings of the 16th IEEE international conference on tools with artificial intelligence. pp 310–317Hansen EA, Zilberstein S (2001) LAO: a heuristic search algorithm that finds solutions with loops. Artif Intell 129: 35–62Hinderer K, Waldmann KH (2003) The critical discount factor for finite Markovian decision processes with an absorbing set. Math Methods Oper Res 57: 1–19Li L (2009) A unifying framework for computational reinforcement learning theory. PhD Thesis. The state university of New Jersey, New Brunswick. NJLittman ML, Dean TL, Kaelbling LP (1995) On the complexity of solving Markov decision problems.In: Proceedings of the 11th international conference on uncertainty in artificial intelligence. Montreal, Quebec pp 394–402McMahan HB, Gordon G (2005a) Fast exact planning in Markov decision processes. In: Proceedings of the 15th international conference on automated planning and scheduling. Monterey, CA, USAMcMahan HB, Gordon G (2005b) Generalizing Dijkstra’s algorithm and gaussian elimination for solving MDPs. Technical report, Carnegie Mellon University, PittsburghMeuleau N, Brafman R, Benazera E (2006) Stochastic over-subscription planning using hierarchies of MDPs. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UK, pp 121–130Moore A, Atkeson C (1993) Prioritized sweeping: reinforcement learning with less data and less real time. Mach Learn 13: 103–130Puterman ML (1994) Markov decision processes. Wiley Editors, New YorkPuterman ML (2005) Markov decision processes. Wiley Inter Science Editors, New YorkRussell S (2005) Artificial intelligence: a modern approach. Making complex decisions (Ch-17), 2nd edn. Pearson Prentice Hill Ed., USAShani G, Brafman R, Shimony S (2008) Prioritizing point-based POMDP solvers. IEEE Trans Syst Man Cybern 38(6): 1592–1605Sniedovich M (2006) Dijkstra’s algorithm revisited: the dynamic programming connexion. Control Cybern 35: 599–620Sniedovich M (2010) Dynamic programming: foundations and principles, 2nd edn. Pure and Applied Mathematics Series, UKTijms HC (2003) A first course in stochastic models. Discrete-time Markov decision processes (Ch-6). Wiley Editors, UKVanderbei RJ (1996) Optimal sailing strategies. Statistics and operations research program, University of Princeton, USA ( http://www.orfe.princeton.edu/~rvdb/sail/sail.html )Vanderbei RJ (2008) Linear programming: foundations and extensions, 3rd edn. Springer, New YorkWingate D, Seppi KD (2005) Prioritization methods for accelerating MDP solvers. J Mach Learn Res 6: 851–88

The Mexican consensus on the diagnosis, treatment, and prevention of NSAID-induced gastropathy and enteropathy

Más de 30 millones de personas consumen diariamente antiinflamatorios noesteroideos (AINE) en el mundo, y este consumo se ve incrementado anualmente. Aunque losAINE poseen propiedades analgésicas y antiinflamatorias, sus eventos adversos gastrointesti-nales son bien reconocidos. En nuestro país no existía un consenso respecto al diagnóstico,tratamiento y prevención de la gastropatía y la enteropatía por AINE, por lo que la AsociaciónMexicana de Gastroenterología reunió a un grupo de expertos para establecer recomendacionesde utilidad para la comunidad médica. En este consenso se emitieron 33 recomendaciones. Elconsenso destaca que el riesgo de toxicidad gastrointestinal de los AINE varía según el fármacoempleado y su farmacocinética, lo cual debe ser considerado al momento de su prescripción. Losfactores de riesgo de complicación gastroduodenal por AINE son: antecedente de úlcera pép-tica, edad mayor a 65 a˜nos, dosis altas del AINE, infección por Helicobacter pylori (H.pylori), ypresencia de comorbilidades graves. Los síntomas y el da˜no gastroduodenal inducido por AINEson variables ya que puede cursar asintomático o manifestarse como anemia por deficiencia dehierro, hemorragia, estenosis y perforación. La cápsula endoscópica y la enteroscopia son méto-dos diagnósticos directos en la enteropatía por AINE. Respecto a la prevención, se recomiendaprescribir la dosis mínima necesaria de un AINE para obtener el efecto deseado y durante elmenor tiempo. Finalmente, los inhibidores de la bomba de protones (IBP) representan el están-dar de oro para la profilaxis y tratamiento de los efectos gastroduodenales, mas no son útilesen la enteropatía

The 2017 May 20th^{\rm th} stellar occultation by the elongated centaur (95626) 2002 GZ32_{32}

We predicted a stellar occultation of the bright star Gaia DR1 4332852996360346368 (UCAC4 385-75921) (m$_{\rm V}$= 14.0 mag) by the centaur 2002 GZ$_{32}$ for 2017 May 20$^{\rm th}$. Our latest shadow path prediction was favourable to a large region in Europe. Observations were arranged in a broad region inside the nominal shadow path. Series of images were obtained with 29 telescopes throughout Europe and from six of them (five in Spain and one in Greece) we detected the occultation. This is the fourth centaur, besides Chariklo, Chiron and Bienor, for which a multi-chord stellar occultation is reported. By means of an elliptical fit to the occultation chords we obtained the limb of 2002 GZ$_{32}$ during the occultation, resulting in an ellipse with axes of 305 $\pm$ 17 km $\times$ 146 $\pm$ 8 km. From this limb, thanks to a rotational light curve obtained shortly after the occultation, we derived the geometric albedo of 2002 GZ$_{32}$ ($p_{\rm V}$ = 0.043 $\pm$ 0.007) and a 3-D ellipsoidal shape with axes 366 km $\times$ 306 km $\times$ 120 km. This shape is not fully consistent with a homogeneous body in hydrostatic equilibrium for the known rotation period of 2002 GZ$_{32}$. The size (albedo) obtained from the occultation is respectively smaller (greater) than that derived from the radiometric technique but compatible within error bars. No rings or debris around 2002 GZ$_{32}$ were detected from the occultation, but narrow and thin rings cannot be discarded.Comment: Accepted for publication in MNRAS (8-Dec.-2020), 15 pages, 9 figure

Free backbone carbonyls mediate rhodopsin activation

Conserved prolines in the transmembrane helices of G-protein-coupled receptors (GPCRs) are often considered to function as hinges that divide the helix into two segments capable of independent motion. Depending on their potential to hydrogen-bond, the free C=O groups associated with these prolines can facilitate conformational flexibility, conformational switching or stabilization of the receptor structure. To address the role of conserved prolines in family A GPCRs through solid-state NMR spectroscopy, we focus on bovine rhodopsin, a GPCR in the visual receptor subfamily. The free backbone C=O groups on helices H5 and H7 stabilize the inactive rhodopsin structure through hydrogen-bonds to residues on adjacent helices. In response to light-induced isomerization of the retinal chromophore, hydrogen-bonding interactions involving these C=O groups are released, thus facilitating repacking of H5 and H7 onto the transmembrane core of the receptor. These results provide insights into the multiple structural and functional roles of prolines in membrane proteins

A novel μCT analysis reveals different responses of bioerosion and secondary accretion to environmental variability

Corals build reefs through accretion of calcium carbonate (CaCO3) skeletons, but net reef growth also depends on bioerosion by grazers and borers and on secondary calcification by crustose coralline algae and other calcifying invertebrates. However, traditional field methods for quantifying secondary accretion and bioerosion confound both processes, do not measure them on the same time-scale, or are restricted to 2D methods. In a prior study, we compared multiple environmental drivers of net erosion using pre- and post-deployment micro-computed tomography scans (μCT; calculated as the % change in volume of experimental CaCO3 blocks) and found a shift from net accretion to net erosion with increasing ocean acidity. Here, we present a novel μCT method and detail a procedure that aligns and digitally subtracts pre- and post-deployment μCT scans and measures the simultaneous response of secondary accretion and bioerosion on blocks exposed to the same environmental variation over the same time-scale. We tested our method on a dataset from a prior study and show that it can be used to uncover information previously unattainable using traditional methods. We demonstrated that secondary accretion and bioerosion are driven by different environmental parameters, bioerosion is more sensitive to ocean acidity than secondary accretion, and net erosion is driven more by changes in bioerosion than secondary accretion

Luminescence Sensors Applied to Water Analysis of Organic Pollutants—An Update

The development of chemical sensors for environmental analysis based on fluorescence, phosphorescence and chemiluminescence signals continues to be a dynamic topic within the sensor field. This review covers the fundamentals of this type of sensors, and an update on recent works devoted to quantifying organic pollutants in environmental waters, focusing on advances since about 2005. Among the wide variety of these contaminants, special attention has been paid polycyclic aromatic hydrocarbons, pesticides, explosives and emerging organic pollutants. The potential of coupling optical sensors with multivariate calibration methods in order to improve the selectivity is also discussed

The Biodiversity of the Mediterranean Sea: Estimates, Patterns, and Threats

The Mediterranean Sea is a marine biodiversity hot spot. Here we combined an extensive literature analysis with expert opinions to update publicly available estimates of major taxa in this marine ecosystem and to revise and update several species lists. We also assessed overall spatial and temporal patterns of species diversity and identified major changes and threats. Our results listed approximately 17,000 marine species occurring in the Mediterranean Sea. However, our estimates of marine diversity are still incomplete as yet—undescribed species will be added in the future. Diversity for microbes is substantially underestimated, and the deep-sea areas and portions of the southern and eastern region are still poorly known. In addition, the invasion of alien species is a crucial factor that will continue to change the biodiversity of the Mediterranean, mainly in its eastern basin that can spread rapidly northwards and westwards due to the warming of the Mediterranean Sea. Spatial patterns showed a general decrease in biodiversity from northwestern to southeastern regions following a gradient of production, with some exceptions and caution due to gaps in our knowledge of the biota along the southern and eastern rims. Biodiversity was also generally higher in coastal areas and continental shelves, and decreases with depth. Temporal trends indicated that overexploitation and habitat loss have been the main human drivers of historical changes in biodiversity. At present, habitat loss and degradation, followed by fishing impacts, pollution, climate change, eutrophication, and the establishment of alien species are the most important threats and affect the greatest number of taxonomic groups. All these impacts are expected to grow in importance in the future, especially climate change and habitat degradation. The spatial identification of hot spots highlighted the ecological importance of most of the western Mediterranean shelves (and in particular, the Strait of Gibraltar and the adjacent Alboran Sea), western African coast, the Adriatic, and the Aegean Sea, which show high concentrations of endangered, threatened, or vulnerable species. The Levantine Basin, severely impacted by the invasion of species, is endangered as well

Co-infection and ICU-acquired infection in COIVD-19 ICU patients: a secondary analysis of the UNITE-COVID data set

Background: The COVID-19 pandemic presented major challenges for critical care facilities worldwide. Infections which develop alongside or subsequent to viral pneumonitis are a challenge under sporadic and pandemic conditions; however, data have suggested that patterns of these differ between COVID-19 and other viral pneumonitides. This secondary analysis aimed to explore patterns of co-infection and intensive care unit-acquired infections (ICU-AI) and the relationship to use of corticosteroids in a large, international cohort of critically ill COVID-19 patients.Methods: This is a multicenter, international, observational study, including adult patients with PCR-confirmed COVID-19 diagnosis admitted to ICUs at the peak of wave one of COVID-19 (February 15th to May 15th, 2020). Data collected included investigator-assessed co-infection at ICU admission, infection acquired in ICU, infection with multi-drug resistant organisms (MDRO) and antibiotic use. Frequencies were compared by Pearson's Chi-squared and continuous variables by Mann-Whitney U test. Propensity score matching for variables associated with ICU-acquired infection was undertaken using R library MatchIT using the "full" matching method.Results: Data were available from 4994 patients. Bacterial co-infection at admission was detected in 716 patients (14%), whilst 85% of patients received antibiotics at that stage. ICU-AI developed in 2715 (54%). The most common ICU-AI was bacterial pneumonia (44% of infections), whilst 9% of patients developed fungal pneumonia; 25% of infections involved MDRO. Patients developing infections in ICU had greater antimicrobial exposure than those without such infections. Incident density (ICU-AI per 1000 ICU days) was in considerable excess of reports from pre-pandemic surveillance. Corticosteroid use was heterogenous between ICUs. In univariate analysis, 58% of patients receiving corticosteroids and 43% of those not receiving steroids developed ICU-AI. Adjusting for potential confounders in the propensity-matched cohort, 71% of patients receiving corticosteroids developed ICU-AI vs 52% of those not receiving corticosteroids. Duration of corticosteroid therapy was also associated with development of ICU-AI and infection with an MDRO.Conclusions: In patients with severe COVID-19 in the first wave, co-infection at admission to ICU was relatively rare but antibiotic use was in substantial excess to that indication. ICU-AI were common and were significantly associated with use of corticosteroids

Clinical and organizational factors associated with mortality during the peak of first COVID-19 wave: the global UNITE-COVID study

Purpose: To accommodate the unprecedented number of critically ill patients with pneumonia caused by coronavirus disease 2019 (COVID-19) expansion of the capacity of intensive care unit (ICU) to clinical areas not previously used for critical care was necessary. We describe the global burden of COVID-19 admissions and the clinical and organizational characteristics associated with outcomes in critically ill COVID-19 patients. Methods: Multicenter, international, point prevalence study, including adult patients with SARS-CoV-2 infection confirmed by polymerase chain reaction (PCR) and a diagnosis of COVID-19 admitted to ICU between February 15th and May 15th, 2020. Results: 4994 patients from 280 ICUs in 46 countries were included. Included ICUs increased their total capacity from 4931 to 7630 beds, deploying personnel from other areas. Overall, 1986 (39.8%) patients were admitted to surge capacity beds. Invasive ventilation at admission was present in 2325 (46.5%) patients and was required during ICU stay in 85.8% of patients. 60-day mortality was 33.9% (IQR across units: 20%–50%) and ICU mortality 32.7%. Older age, invasive mechanical ventilation, and acute kidney injury (AKI) were associated with increased mortality. These associations were also confirmed specifically in mechanically ventilated patients. Admission to surge capacity beds was not associated with mortality, even after controlling for other factors. Conclusions: ICUs responded to the increase in COVID-19 patients by increasing bed availability and staff, admitting up to 40% of patients in surge capacity beds. Although mortality in this population was high, admission to a surge capacity bed was not associated with increased mortality. Older age, invasive mechanical ventilation, and AKI were identified as the strongest predictors of mortality