New prioritized value iteration for Markov decision processes

The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach. © Springer Science+Business Media B.V. 2011.García Hernández, MDG.; Ruiz Pinales, J.; Onaindia De La Rivaherrera, E.; Aviña Cervantes, JG.; Ledesma Orozco, S.; Alvarado Mendez, E.; Reyes Ballesteros, A. (2012). New prioritized value iteration for Markov decision processes. Artificial Intelligence Review. 37(2):157-167. doi:10.1007/s10462-011-9224-zS157167372Agrawal S, Roth D (2002) Learning a sparse representation for object detection. In: Proceedings of the 7th European conference on computer vision. Copenhagen, Denmark, pp 1–15Bellman RE (1954) The theory of dynamic programming. Bull Amer Math Soc 60: 503–516Bellman RE (1957) Dynamic programming. Princeton University Press, New JerseyBertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, MassachusettsBhuma K, Goldsmith J (2003) Bidirectional LAO* algorithm. In: Proceedings of indian international conferences on artificial intelligence. p 980–992Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36: 226–235Bonet B, Geffner H (2003a) Faster heuristic search algorithms for planning with uncertainty and full feedback. In: Proceedings of the 18th international joint conference on artificial intelligence. Morgan Kaufmann, Acapulco, México, pp 1233–1238Bonet B, Geffner H (2003b) Labeled RTDP: improving the convergence of real-time dynamic programming. In: Proceedings of the international conference on automated planning and scheduling. Trento, Italy, pp 12–21Bonet B, Geffner H (2006) Learning depth-first search: a unified approach to heuristic search in deterministic and non-deterministic settings and its application to MDP. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UKBoutilier C, Dean T, Hanks S (1999) Decision-theoretic planning: structural assumptions and computational leverage. J Artif Intell Res 11: 1–94Chang I, Soo H (2007) Simulation-based algorithms for Markov decision processes Communications and control engineering. Springer, LondonDai P, Goldsmith J (2007a) Faster dynamic programming for Markov decision processes. Technical report. Doctoral consortium, department of computer science and engineering. University of WashingtonDai P, Goldsmith J (2007b) Topological value iteration algorithm for Markov decision processes. In: Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 1860–1865Dai P, Hansen EA (2007c) Prioritizing bellman backups without a priority queue. In: Proceedings of the 17th international conference on automated planning and scheduling, association for the advancement of artificial intelligence. Rhode Island, USA, pp 113–119Dibangoye JS, Chaib-draa B, Mouaddib A (2008) A Novel prioritization technique for solving Markov decision processes. In: Proceedings of the 21st international FLAIRS (The Florida Artificial Intelligence Research Society) conference, association for the advancement of artificial intelligence. Florida, USAFerguson D, Stentz A (2004) Focused propagation of MDPs for path planning. In: Proceedings of the 16th IEEE international conference on tools with artificial intelligence. pp 310–317Hansen EA, Zilberstein S (2001) LAO: a heuristic search algorithm that finds solutions with loops. Artif Intell 129: 35–62Hinderer K, Waldmann KH (2003) The critical discount factor for finite Markovian decision processes with an absorbing set. Math Methods Oper Res 57: 1–19Li L (2009) A unifying framework for computational reinforcement learning theory. PhD Thesis. The state university of New Jersey, New Brunswick. NJLittman ML, Dean TL, Kaelbling LP (1995) On the complexity of solving Markov decision problems.In: Proceedings of the 11th international conference on uncertainty in artificial intelligence. Montreal, Quebec pp 394–402McMahan HB, Gordon G (2005a) Fast exact planning in Markov decision processes. In: Proceedings of the 15th international conference on automated planning and scheduling. Monterey, CA, USAMcMahan HB, Gordon G (2005b) Generalizing Dijkstra’s algorithm and gaussian elimination for solving MDPs. Technical report, Carnegie Mellon University, PittsburghMeuleau N, Brafman R, Benazera E (2006) Stochastic over-subscription planning using hierarchies of MDPs. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UK, pp 121–130Moore A, Atkeson C (1993) Prioritized sweeping: reinforcement learning with less data and less real time. Mach Learn 13: 103–130Puterman ML (1994) Markov decision processes. Wiley Editors, New YorkPuterman ML (2005) Markov decision processes. Wiley Inter Science Editors, New YorkRussell S (2005) Artificial intelligence: a modern approach. Making complex decisions (Ch-17), 2nd edn. Pearson Prentice Hill Ed., USAShani G, Brafman R, Shimony S (2008) Prioritizing point-based POMDP solvers. IEEE Trans Syst Man Cybern 38(6): 1592–1605Sniedovich M (2006) Dijkstra’s algorithm revisited: the dynamic programming connexion. Control Cybern 35: 599–620Sniedovich M (2010) Dynamic programming: foundations and principles, 2nd edn. Pure and Applied Mathematics Series, UKTijms HC (2003) A first course in stochastic models. Discrete-time Markov decision processes (Ch-6). Wiley Editors, UKVanderbei RJ (1996) Optimal sailing strategies. Statistics and operations research program, University of Princeton, USA ( http://www.orfe.princeton.edu/~rvdb/sail/sail.html )Vanderbei RJ (2008) Linear programming: foundations and extensions, 3rd edn. Springer, New YorkWingate D, Seppi KD (2005) Prioritization methods for accelerating MDP solvers. J Mach Learn Res 6: 851–88

Moonlighting Proteins Hal3 and Vhs3 Form a Heteromeric PPCDC with Ykl088w in Yeast CoA Biosynthesis

Premi a l'excel·lència investigadora. 2010Unlike most other organisms, the essential five-step Coenzyme A biosynthetic pathway has not been fully resolved in yeast. Specifically, the gene(s) encoding the phosphopantothenoylcysteine decarboxylase (PPCDC) activity still remains unidentified. Sequence homology analyses suggest three candidates, namely Ykl088w, Hal3 and Vhs3, as putative PPCDC enzymes in Saccharomyces cerevisiae. Interestingly, Hal3 and Vhs3 have been characterized as negative regulatory subunits of the Ppz1 protein phosphatase. Here we show that YKL088w does not encode a third Ppz1 regulatory subunit, and that the essential roles of Ykl088w and the Hal3/Vhs3 pair are complementary, cannot be interchanged and can be attributed to PPCDC-related functions. We demonstrate that while known eukaryotic PPCDCs are homotrimers, the active yeast enzyme is a heterotrimer which consists of Ykl088w and Hal3/Vhs3 monomers that separately provides two essential catalytic residues. Our results unveil Hal3/Vhs3 as moonlighting proteins, involved in both CoA biosynthesis and protein phosphatase regulation

Molecular behaviour of phenol in zeolite Beta catalysts as a function of acid site presence: a quasielastic neutron scattering and molecular dynamics simulation study

Quasielastic neutron scattering (QENS) experiments complemented by classical molecular dynamics (MD) simulations at 393–443 K were employed in a study of the mobility and interactions of phenol in acidic zeolite H-Beta, to understand systems relevant to potential routes for the depolymerization and hydrodeoxygenation of lignin. QENS experiments observed isotropic phenol rotation with a fraction of static molecules, yielding rotational diffusion coefficients between 2.60 × 1010 and 3.33 × 1010 s−1 and an activation energy of rotation of 7.2 kJ mol−1. The MD simulations of phenol in the acidic and all-silica zeolite corroborate the experimental results, where molecules strongly adsorbed to the acidic sites behave as an immobile fraction with minimal contribution to the rotational diffusion, and the mobile molecules yield similar rotational diffusion coefficients to experiment. Translational diffusion is too slow to be detected in the instrumental time window of the QENS experiments, which is supported by MD-calculated activation energies of translation larger than 25 kJ mol−1. The study illustrates the effect of active sites in potential catalyst structures on the dynamical behaviour of molecules relevant to biomass conversion

Differential modulation of the TRAIL receptors and the CD95 receptor in colon carcinoma cell lines

Tumour necrosis factor-related apoptosis-inducing ligand (TRAIL) and CD95 ligand (CD95L) are potent inducers of apoptosis in various tumour cell types. Death receptors DR4 and DR5 can induce and decoy receptors DcR1 and DcR2 can inhibit TRAIL-mediated apoptosis. The study aim was to investigate whether anticancer agents can modulate similarly TRAIL-receptor and CD95 membrane expression and TRAIL and CD95L sensitivity.Three colon carcinoma cell lines (Caco-2, Colo320 and SW948) were treated with 5-fluorouracil (5-FU), cisplatin or interferon-γ. TRAIL-receptor and CD95 membrane expression was determined flow cytometrically. Sensitivity to TRAIL or CD95L agonistic anti-CD95 antibody was determined with cytotoxicity and apoptosis assays. SW948 showed highest TRAIL sensitivity. The protein synthesis inhibitor cycloheximide decreased FLICE-like inhibitory protein levels in all cell lines, and the TRAIL-resistant cell lines Caco-2 and Colo320 became sensitive for TRAIL. Exposure of the cell lines to 5-FU, cisplatin and interferon-γ left TRAIL-receptor membrane expression and TRAIL sensitivity unaffected. CD95 membrane expression and anti-CD95 sensitivity was, however, modulated by the same drugs in all lines. Cisplatin and interferon-γ raised CD95 membrane levels 6–8-fold, interferon-γ also increased anti-CD95 sensitivity. These results indicate that the CD95 and TRAIL pathways use different mechanisms to respond to various anticancer agents. Induced CD95 membrane upregulation was associated with increased anti-CD95 sensitivity, whereas no upregulation of TRAIL-receptor membrane expression or TRAIL sensitisation could be established. For optimal use of TRAIL-mediated apoptosis for cancer therapy in certain tumours, downregulation of intracellular inhibiting factors may be required

Comparison of seven prognostic tools to identify low-risk pulmonary embolism in patients aged <50 years

publishersversionPeer reviewe

Observation of associated near-side and away-side long-range correlations in √sNN=5.02 TeV proton-lead collisions with the ATLAS detector

Two-particle correlations in relative azimuthal angle (Δϕ) and pseudorapidity (Δη) are measured in √sNN=5.02  TeV p+Pb collisions using the ATLAS detector at the LHC. The measurements are performed using approximately 1  μb-1 of data as a function of transverse momentum (pT) and the transverse energy (ΣETPb) summed over 3.1<η<4.9 in the direction of the Pb beam. The correlation function, constructed from charged particles, exhibits a long-range (2<|Δη|<5) “near-side” (Δϕ∼0) correlation that grows rapidly with increasing ΣETPb. A long-range “away-side” (Δϕ∼π) correlation, obtained by subtracting the expected contributions from recoiling dijets and other sources estimated using events with small ΣETPb, is found to match the near-side correlation in magnitude, shape (in Δη and Δϕ) and ΣETPb dependence. The resultant Δϕ correlation is approximately symmetric about π/2, and is consistent with a dominant cos⁡2Δϕ modulation for all ΣETPb ranges and particle pT

Performance of CMS muon reconstruction in pp collision events at sqrt(s) = 7 TeV

The performance of muon reconstruction, identification, and triggering in CMS has been studied using 40 inverse picobarns of data collected in pp collisions at sqrt(s) = 7 TeV at the LHC in 2010. A few benchmark sets of selection criteria covering a wide range of physics analysis needs have been examined. For all considered selections, the efficiency to reconstruct and identify a muon with a transverse momentum pT larger than a few GeV is above 95% over the whole region of pseudorapidity covered by the CMS muon system, abs(eta) < 2.4, while the probability to misidentify a hadron as a muon is well below 1%. The efficiency to trigger on single muons with pT above a few GeV is higher than 90% over the full eta range, and typically substantially better. The overall momentum scale is measured to a precision of 0.2% with muons from Z decays. The transverse momentum resolution varies from 1% to 6% depending on pseudorapidity for muons with pT below 100 GeV and, using cosmic rays, it is shown to be better than 10% in the central region up to pT = 1 TeV. Observed distributions of all quantities are well reproduced by the Monte Carlo simulation.Comment: Replaced with published version. Added journal reference and DO

Search for R-parity-violating supersymmetry in events with four or more leptons in sqrt(s) =7 TeV pp collisions with the ATLAS detector

A search for new phenomena in final states with four or more leptons (electrons or muons) is presented. The analysis is based on 4.7 fb−1 of $\sqrt{s}=7\;\mathrm{TeV}$ proton-proton collisions delivered by the Large Hadron Collider and recorded with the ATLAS detector. Observations are consistent with Standard Model expectations in two signal regions: one that requires moderate values of missing transverse momentum and another that requires large effective mass. The results are interpreted in a simplified model of R-parity-violating supersymmetry in which a 95% CL exclusion region is set for charged wino masses up to 540 GeV. In an R-parity-violating MSUGRA/CMSSM model, values of m 1/2 up to 820 GeV are excluded for 10 < tan β < 40

Search for high-mass resonances decaying to dilepton final states in pp collisions at s√=7 TeV with the ATLAS detector

The ATLAS detector at the Large Hadron Collider is used to search for high-mass resonances decaying to an electron-positron pair or a muon-antimuon pair. The search is sensitive to heavy neutral Z′ gauge bosons, Randall-Sundrum gravitons, Z * bosons, techni-mesons, Kaluza-Klein Z/γ bosons, and bosons predicted by Torsion models. Results are presented based on an analysis of pp collisions at a center-of-mass energy of 7 TeV corresponding to an integrated luminosity of 4.9 fb−1 in the e + e − channel and 5.0 fb−1 in the μ + μ −channel. A Z ′ boson with Standard Model-like couplings is excluded at 95 % confidence level for masses below 2.22 TeV. A Randall-Sundrum graviton with coupling k/MPl=0.1 is excluded at 95 % confidence level for masses below 2.16 TeV. Limits on the other models are also presented, including Technicolor and Minimal Z′ Models