New prioritized value iteration for Markov decision processes

The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach. © Springer Science+Business Media B.V. 2011.García Hernández, MDG.; Ruiz Pinales, J.; Onaindia De La Rivaherrera, E.; Aviña Cervantes, JG.; Ledesma Orozco, S.; Alvarado Mendez, E.; Reyes Ballesteros, A. (2012). New prioritized value iteration for Markov decision processes. Artificial Intelligence Review. 37(2):157-167. doi:10.1007/s10462-011-9224-zS157167372Agrawal S, Roth D (2002) Learning a sparse representation for object detection. In: Proceedings of the 7th European conference on computer vision. Copenhagen, Denmark, pp 1–15Bellman RE (1954) The theory of dynamic programming. Bull Amer Math Soc 60: 503–516Bellman RE (1957) Dynamic programming. Princeton University Press, New JerseyBertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, MassachusettsBhuma K, Goldsmith J (2003) Bidirectional LAO* algorithm. In: Proceedings of indian international conferences on artificial intelligence. p 980–992Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36: 226–235Bonet B, Geffner H (2003a) Faster heuristic search algorithms for planning with uncertainty and full feedback. In: Proceedings of the 18th international joint conference on artificial intelligence. Morgan Kaufmann, Acapulco, México, pp 1233–1238Bonet B, Geffner H (2003b) Labeled RTDP: improving the convergence of real-time dynamic programming. In: Proceedings of the international conference on automated planning and scheduling. Trento, Italy, pp 12–21Bonet B, Geffner H (2006) Learning depth-first search: a unified approach to heuristic search in deterministic and non-deterministic settings and its application to MDP. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UKBoutilier C, Dean T, Hanks S (1999) Decision-theoretic planning: structural assumptions and computational leverage. J Artif Intell Res 11: 1–94Chang I, Soo H (2007) Simulation-based algorithms for Markov decision processes Communications and control engineering. Springer, LondonDai P, Goldsmith J (2007a) Faster dynamic programming for Markov decision processes. Technical report. Doctoral consortium, department of computer science and engineering. University of WashingtonDai P, Goldsmith J (2007b) Topological value iteration algorithm for Markov decision processes. In: Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 1860–1865Dai P, Hansen EA (2007c) Prioritizing bellman backups without a priority queue. In: Proceedings of the 17th international conference on automated planning and scheduling, association for the advancement of artificial intelligence. Rhode Island, USA, pp 113–119Dibangoye JS, Chaib-draa B, Mouaddib A (2008) A Novel prioritization technique for solving Markov decision processes. In: Proceedings of the 21st international FLAIRS (The Florida Artificial Intelligence Research Society) conference, association for the advancement of artificial intelligence. Florida, USAFerguson D, Stentz A (2004) Focused propagation of MDPs for path planning. In: Proceedings of the 16th IEEE international conference on tools with artificial intelligence. pp 310–317Hansen EA, Zilberstein S (2001) LAO: a heuristic search algorithm that finds solutions with loops. Artif Intell 129: 35–62Hinderer K, Waldmann KH (2003) The critical discount factor for finite Markovian decision processes with an absorbing set. Math Methods Oper Res 57: 1–19Li L (2009) A unifying framework for computational reinforcement learning theory. PhD Thesis. The state university of New Jersey, New Brunswick. NJLittman ML, Dean TL, Kaelbling LP (1995) On the complexity of solving Markov decision problems.In: Proceedings of the 11th international conference on uncertainty in artificial intelligence. Montreal, Quebec pp 394–402McMahan HB, Gordon G (2005a) Fast exact planning in Markov decision processes. In: Proceedings of the 15th international conference on automated planning and scheduling. Monterey, CA, USAMcMahan HB, Gordon G (2005b) Generalizing Dijkstra’s algorithm and gaussian elimination for solving MDPs. Technical report, Carnegie Mellon University, PittsburghMeuleau N, Brafman R, Benazera E (2006) Stochastic over-subscription planning using hierarchies of MDPs. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UK, pp 121–130Moore A, Atkeson C (1993) Prioritized sweeping: reinforcement learning with less data and less real time. Mach Learn 13: 103–130Puterman ML (1994) Markov decision processes. Wiley Editors, New YorkPuterman ML (2005) Markov decision processes. Wiley Inter Science Editors, New YorkRussell S (2005) Artificial intelligence: a modern approach. Making complex decisions (Ch-17), 2nd edn. Pearson Prentice Hill Ed., USAShani G, Brafman R, Shimony S (2008) Prioritizing point-based POMDP solvers. IEEE Trans Syst Man Cybern 38(6): 1592–1605Sniedovich M (2006) Dijkstra’s algorithm revisited: the dynamic programming connexion. Control Cybern 35: 599–620Sniedovich M (2010) Dynamic programming: foundations and principles, 2nd edn. Pure and Applied Mathematics Series, UKTijms HC (2003) A first course in stochastic models. Discrete-time Markov decision processes (Ch-6). Wiley Editors, UKVanderbei RJ (1996) Optimal sailing strategies. Statistics and operations research program, University of Princeton, USA ( http://www.orfe.princeton.edu/~rvdb/sail/sail.html )Vanderbei RJ (2008) Linear programming: foundations and extensions, 3rd edn. Springer, New YorkWingate D, Seppi KD (2005) Prioritization methods for accelerating MDP solvers. J Mach Learn Res 6: 851–88

Mortality from gastrointestinal congenital anomalies at 264 hospitals in 74 low-income, middle-income, and high-income countries: a multicentre, international, prospective cohort study

Summary Background Congenital anomalies are the fifth leading cause of mortality in children younger than 5 years globally. Many gastrointestinal congenital anomalies are fatal without timely access to neonatal surgical care, but few studies have been done on these conditions in low-income and middle-income countries (LMICs). We compared outcomes of the seven most common gastrointestinal congenital anomalies in low-income, middle-income, and high-income countries globally, and identified factors associated with mortality. Methods We did a multicentre, international prospective cohort study of patients younger than 16 years, presenting to hospital for the first time with oesophageal atresia, congenital diaphragmatic hernia, intestinal atresia, gastroschisis, exomphalos, anorectal malformation, and Hirschsprung’s disease. Recruitment was of consecutive patients for a minimum of 1 month between October, 2018, and April, 2019. We collected data on patient demographics, clinical status, interventions, and outcomes using the REDCap platform. Patients were followed up for 30 days after primary intervention, or 30 days after admission if they did not receive an intervention. The primary outcome was all-cause, in-hospital mortality for all conditions combined and each condition individually, stratified by country income status. We did a complete case analysis. Findings We included 3849 patients with 3975 study conditions (560 with oesophageal atresia, 448 with congenital diaphragmatic hernia, 681 with intestinal atresia, 453 with gastroschisis, 325 with exomphalos, 991 with anorectal malformation, and 517 with Hirschsprung’s disease) from 264 hospitals (89 in high-income countries, 166 in middleincome countries, and nine in low-income countries) in 74 countries. Of the 3849 patients, 2231 (58·0%) were male. Median gestational age at birth was 38 weeks (IQR 36–39) and median bodyweight at presentation was 2·8 kg (2·3–3·3). Mortality among all patients was 37 (39·8%) of 93 in low-income countries, 583 (20·4%) of 2860 in middle-income countries, and 50 (5·6%) of 896 in high-income countries (p<0·0001 between all country income groups). Gastroschisis had the greatest difference in mortality between country income strata (nine [90·0%] of ten in lowincome countries, 97 [31·9%] of 304 in middle-income countries, and two [1·4%] of 139 in high-income countries; p≤0·0001 between all country income groups). Factors significantly associated with higher mortality for all patients combined included country income status (low-income vs high-income countries, risk ratio 2·78 [95% CI 1·88–4·11], p<0·0001; middle-income vs high-income countries, 2·11 [1·59–2·79], p<0·0001), sepsis at presentation (1·20 [1·04–1·40], p=0·016), higher American Society of Anesthesiologists (ASA) score at primary intervention (ASA 4–5 vs ASA 1–2, 1·82 [1·40–2·35], p<0·0001; ASA 3 vs ASA 1–2, 1·58, [1·30–1·92], p<0·0001]), surgical safety checklist not used (1·39 [1·02–1·90], p=0·035), and ventilation or parenteral nutrition unavailable when needed (ventilation 1·96, [1·41–2·71], p=0·0001; parenteral nutrition 1·35, [1·05–1·74], p=0·018). Administration of parenteral nutrition (0·61, [0·47–0·79], p=0·0002) and use of a peripherally inserted central catheter (0·65 [0·50–0·86], p=0·0024) or percutaneous central line (0·69 [0·48–1·00], p=0·049) were associated with lower mortality. Interpretation Unacceptable differences in mortality exist for gastrointestinal congenital anomalies between lowincome, middle-income, and high-income countries. Improving access to quality neonatal surgical care in LMICs will be vital to achieve Sustainable Development Goal 3.2 of ending preventable deaths in neonates and children younger than 5 years by 2030

Gas sensor design using a Fabry-Perot interferometer and a pyroelectric detector

A gas sensor design based on cross correlation spectroscopy using a FPI with long cavity length (&gt;500 µm) as a modulator and with a pyroelectric detector is presented. This technique can be suitable to detect molecules with very well defined ro-vibrational lines such as CO, CO2 and CH4. To induce modulation one mirror of the FPI is scan over lambda/2, this will shift the spectral FPI transmission fringe pattern producing the cross correlation principle. In this design the FPI is illuminated with a converging beam and it is placed just in front of the pyroelectric detector, this configuration can be useful if the FPI and the detector are integrated within a MEMS. Furthermore here the FPI scan is driven using a triangular waveform. Therefore in this work we will analyze the effects of illuminate the FPI with a converging beam and their consequences in the overall sensor response. Moreover we will review some possible effects in the sensor response for driving the FPI mirror scan with a triangular waveform. This issue can be very important since it can change the harmonic composition of the modulated signal and consequently the overall sensor response since the pyroelectric detector responsivity is frequency dependant. Finally, simulation results and experimental measurements are provided

K∗(892)0\mathrm{K}^{*}(\mathrm{892})^{0} and ϕ(1020)\mathrm{\phi(1020)} production in p-Pb collisions at sNN\sqrt{s_{\rm NN}} = 8.16 TeV

The production of $\mathrm{K}^{*}(\mathrm{892})^{0}$ and $\mathrm{\phi(1020)}$ resonances has been measured in p-Pb collisions at $\sqrt{s_{\rm NN}}$ = 8.16 TeV using the ALICE detector. Resonances are reconstructed via their hadronic decay channels in the rapidity interval $-$0.5 $$ 8 GeV/$c$), the $R_{\rm pPb}$ values of all hadrons are consistent with unity within uncertainties. The $R_{\rm pPb}$ of $\mathrm{K}^{*}(\mathrm{892})^{0}$ and $\mathrm{\phi(1020)}$ at $\sqrt{s_{\rm NN}}$ = 8.16 and 5.02 TeV show no significant energy dependence

Charged-particle multiplicity fluctuations in Pb–Pb collisions at √ sNN = 2.76 TeV

Measurements of event-by-event fluctuations of charged-particle multiplicities in Pb–Pb collisionsat √sNN = 2.76 TeV using the ALICE detector at the CERN Large Hadron Collider (LHC) are presented in the pseudorapidity range |η| < 0.8 and transverse momentum 0.2 < pT < 2.0 GeV/c. The amplitude of the fluctuations is expressed in terms of the variance normalized by the mean of the multiplicity distribution. The η and pT dependences of the fluctuations and their evolution with respect to collision centrality are investigated. The multiplicity fluctuations tend to decrease from peripheral to central collisions. The results are compared to those obtained from HIJING and AMPT Monte Carlo event generators as well as to experimental data at lower collision energies. Additionally, the measured multiplicity fluctuations are discussed in the context of the isothermal compressibility of the high-density strongly-interacting system formed in central Pb–Pb collisions

Measurements of jet quenching using semi-inclusive hadron+jet distributions in pp and central Pb−-Pb collisions at sNN=5.02\sqrt{s_{\rm NN}}=5.02 TeV

The ALICE Collaboration reports measurements of the semi-inclusive distribution of charged-particle jets recoiling from a high transverse momentum (high $p_{\rm T}$) charged hadron, in pp and central Pb$-$Pb collisions at center of mass energy per nucleon$-$nucleon collision $\sqrt{s_{\rm NN}}=5.02$ TeV. The large uncorrelated background in central Pb$-$Pb collisions is corrected using a data-driven statistical approach, which enables precise measurement of recoil jet distributions over a broad range in $p_{\rm T,ch\,jet}$ and jet resolution parameter $R$. Recoil jet yields are reported for $R=0.2$, 0.4, and 0.5 in the range $7 < p_{\rm T,ch\, jet} < 140$ GeV$/c$ and $\pi/2<\Delta\varphi<\pi$, where $\Delta\varphi$ is the azimuthal angular separation between hadron trigger and recoil jet. The low $p_{\rm T,ch\,jet}$ reach of the measurement explores unique phase space for studying jet quenching, the interaction of jets with the quark-gluonnplasma generated in high-energy nuclear collisions. Comparison of $p_{\rm T,ch\,jet}$ distributions from pp and central Pb$-$Pb collisions probes medium-induced jet energy loss and intra-jet broadening, while comparison of their acoplanarity distributions explores in-medium jet scattering and medium response. The measurements are compared to theoretical calculations incorporating jet quenching.The ALICE Collaboration reports measurements of the semi-inclusive distribution of charged-particle jets recoiling from a high transverse momentum (high $p_{\rm T}$) charged hadron, in pp and central Pb$-$Pb collisions at center of mass energy per nucleon$-$nucleon collision $\sqrt{s_{\rm NN}}=5.02$ TeV. The large uncorrelated background in central Pb$-$Pb collisions is corrected using a data-driven statistical approach, which enables precise measurement of recoil jet distributions over a broad range in $p_{\rm T,ch\,jet}$ and jet resolution parameter $R$. Recoil jet yields are reported for $R=0.2$, 0.4, and 0.5 in the range $7 < p_{\rm T,ch\, jet} < 140$ GeV$/c$ and $\pi/2<\Delta\varphi<\pi$, where $\Delta\varphi$ is the azimuthal angular separation between hadron trigger and recoil jet. The low $p_{\rm T,ch\,jet}$ reach of the measurement explores unique phase space for studying jet quenching, the interaction of jets with the quark-gluonnplasma generated in high-energy nuclear collisions. Comparison of $p_{\rm T,ch\,jet}$ distributions from pp and central Pb$-$Pb collisions probes medium-induced jet energy loss and intra-jet broadening, while comparison of their acoplanarity distributions explores in-medium jet scattering and medium response. The measurements are compared to theoretical calculations incorporating jet quenching

Dielectron production in central Pb−-Pb collisions at sNN\sqrt{s_\mathrm{NN}} = 5.02 TeV

International audienceThe first measurement of the e$^+$e$^-$ pair production at midrapidity and low invariant mass in central Pb$-$Pb collisions at $\sqrt{s_{\mathrm{NN}}}=5.02$ TeV at the LHC is presented. The yield of e$^+$e$^-$ pairs is compared with a cocktail of expected hadronic decay contributions in the invariant mass ($m_{\rm ee}$) and pair transverse momentum ($p_{\rm T,ee}$) ranges $m_{\rm ee} < 3.5$ GeV$/c^2$ and $p_{\rm T,ee} < 8$ GeV$/c$. For $0.18 < m_{\rm ee} < 0.5$ GeV$/c^2$ the ratio of data to the cocktail of hadronic contributions without $\rho$ mesons amounts to $1.42 \pm 0.12 \ ({\rm stat.}) \pm 0.17 \ ({\rm syst.}) \pm 0.12 \ ({\rm cocktail})$ and $1.44 \pm 0.12 \ ({\rm stat.}) \pm 0.17 \ ({\rm syst.}) ^{+0.17}_{-0.21} \ ({\rm cocktail})$, including or not including medium effects in the estimation of the heavy-flavor background, respectively. It is consistent with predictions from two different models for an additional contribution of thermal e$^+$e$^-$ pairs from the hadronic and partonic phases. In the intermediate-mass range ($1.2 < m_{\rm ee} < 2.6$ GeV$/c^2$), the pair transverse impact parameter of the e$^+$e$^-$ pairs (DCA$_{\rm ee}$) is used for the first time in Pb$-$Pb collisions to separate displaced dielectrons from heavy-flavor hadron decays from a possible (thermal) contribution produced at the interaction point. The data are consistent with a suppression of e$^+$e$^-$ pairs from ${\rm c\overline{c}}$ and an additional prompt component. Finally, the first direct-photon measurement in the 10% most central Pb$-$Pb collisions at $\sqrt{s_{\mathrm{NN}}}=5.02$ TeV is reported via the study of virtual direct photons in the transverse momentum range $1 < p_{\rm T} < 5$ GeV$/c$. A model including prompt photons, as well as photons from the pre-equilibrium and fluid-dynamic phases, can reproduce the result, while being at the upper edge of the data uncertainties

System size dependence of hadronic rescattering effect at LHC energies

International audienceThe first measurements of $\mathrm{K^{*}(892)^{0}}$ resonance production as a function of charged-particle multiplicity in Xe$-$Xe collisions at $\sqrt{s_{\mathrm{NN}}}=$ 5.44 TeV and pp collisions at $\sqrt{s}=$ 5.02 TeV using the ALICE detector are presented. The resonance is reconstructed at midrapidity ($|y|< 0.5$) using the hadronic decay channel $\mathrm{K^{*0}} \rightarrow \mathrm{K^{\pm} \pi^{\mp}}$. Measurements of transverse-momentum integrated yield, mean transverse-momentum, nuclear modification factor of $\mathrm{K^{*0}}$, and yield ratios of resonance to stable hadron ($\mathrm{K^{*0}}$/K) are compared across different collision systems (pp, p$-$Pb, Xe$-$Xe, and Pb$-$Pb) at similar collision energies to investigate how the production of $\mathrm{K^{*0}}$ resonances depends on the size of the system formed in these collisions. The hadronic rescattering effect is found to be independent of the size of colliding systems and mainly driven by the produced charged-particle multiplicity, which is a proxy of the volume of produced matter at the chemical freeze-out. In addition, the production yields of $\mathrm{K^{*0}}$ in Xe$-$Xe collisions are utilized to constrain the dependence of the kinetic freeze-out temperature on the system size using HRG-PCE model