10 research outputs found

    New prioritized value iteration for Markov decision processes

    Full text link
    The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach. © Springer Science+Business Media B.V. 2011.García Hernández, MDG.; Ruiz Pinales, J.; Onaindia De La Rivaherrera, E.; Aviña Cervantes, JG.; Ledesma Orozco, S.; Alvarado Mendez, E.; Reyes Ballesteros, A. (2012). New prioritized value iteration for Markov decision processes. Artificial Intelligence Review. 37(2):157-167. doi:10.1007/s10462-011-9224-zS157167372Agrawal S, Roth D (2002) Learning a sparse representation for object detection. In: Proceedings of the 7th European conference on computer vision. Copenhagen, Denmark, pp 1–15Bellman RE (1954) The theory of dynamic programming. Bull Amer Math Soc 60: 503–516Bellman RE (1957) Dynamic programming. Princeton University Press, New JerseyBertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, MassachusettsBhuma K, Goldsmith J (2003) Bidirectional LAO* algorithm. In: Proceedings of indian international conferences on artificial intelligence. p 980–992Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36: 226–235Bonet B, Geffner H (2003a) Faster heuristic search algorithms for planning with uncertainty and full feedback. In: Proceedings of the 18th international joint conference on artificial intelligence. Morgan Kaufmann, Acapulco, México, pp 1233–1238Bonet B, Geffner H (2003b) Labeled RTDP: improving the convergence of real-time dynamic programming. In: Proceedings of the international conference on automated planning and scheduling. Trento, Italy, pp 12–21Bonet B, Geffner H (2006) Learning depth-first search: a unified approach to heuristic search in deterministic and non-deterministic settings and its application to MDP. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UKBoutilier C, Dean T, Hanks S (1999) Decision-theoretic planning: structural assumptions and computational leverage. J Artif Intell Res 11: 1–94Chang I, Soo H (2007) Simulation-based algorithms for Markov decision processes Communications and control engineering. Springer, LondonDai P, Goldsmith J (2007a) Faster dynamic programming for Markov decision processes. Technical report. Doctoral consortium, department of computer science and engineering. University of WashingtonDai P, Goldsmith J (2007b) Topological value iteration algorithm for Markov decision processes. In: Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 1860–1865Dai P, Hansen EA (2007c) Prioritizing bellman backups without a priority queue. In: Proceedings of the 17th international conference on automated planning and scheduling, association for the advancement of artificial intelligence. Rhode Island, USA, pp 113–119Dibangoye JS, Chaib-draa B, Mouaddib A (2008) A Novel prioritization technique for solving Markov decision processes. In: Proceedings of the 21st international FLAIRS (The Florida Artificial Intelligence Research Society) conference, association for the advancement of artificial intelligence. Florida, USAFerguson D, Stentz A (2004) Focused propagation of MDPs for path planning. In: Proceedings of the 16th IEEE international conference on tools with artificial intelligence. pp 310–317Hansen EA, Zilberstein S (2001) LAO: a heuristic search algorithm that finds solutions with loops. Artif Intell 129: 35–62Hinderer K, Waldmann KH (2003) The critical discount factor for finite Markovian decision processes with an absorbing set. Math Methods Oper Res 57: 1–19Li L (2009) A unifying framework for computational reinforcement learning theory. PhD Thesis. The state university of New Jersey, New Brunswick. NJLittman ML, Dean TL, Kaelbling LP (1995) On the complexity of solving Markov decision problems.In: Proceedings of the 11th international conference on uncertainty in artificial intelligence. Montreal, Quebec pp 394–402McMahan HB, Gordon G (2005a) Fast exact planning in Markov decision processes. In: Proceedings of the 15th international conference on automated planning and scheduling. Monterey, CA, USAMcMahan HB, Gordon G (2005b) Generalizing Dijkstra’s algorithm and gaussian elimination for solving MDPs. Technical report, Carnegie Mellon University, PittsburghMeuleau N, Brafman R, Benazera E (2006) Stochastic over-subscription planning using hierarchies of MDPs. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UK, pp 121–130Moore A, Atkeson C (1993) Prioritized sweeping: reinforcement learning with less data and less real time. Mach Learn 13: 103–130Puterman ML (1994) Markov decision processes. Wiley Editors, New YorkPuterman ML (2005) Markov decision processes. Wiley Inter Science Editors, New YorkRussell S (2005) Artificial intelligence: a modern approach. Making complex decisions (Ch-17), 2nd edn. Pearson Prentice Hill Ed., USAShani G, Brafman R, Shimony S (2008) Prioritizing point-based POMDP solvers. IEEE Trans Syst Man Cybern 38(6): 1592–1605Sniedovich M (2006) Dijkstra’s algorithm revisited: the dynamic programming connexion. Control Cybern 35: 599–620Sniedovich M (2010) Dynamic programming: foundations and principles, 2nd edn. Pure and Applied Mathematics Series, UKTijms HC (2003) A first course in stochastic models. Discrete-time Markov decision processes (Ch-6). Wiley Editors, UKVanderbei RJ (1996) Optimal sailing strategies. Statistics and operations research program, University of Princeton, USA ( http://www.orfe.princeton.edu/~rvdb/sail/sail.html )Vanderbei RJ (2008) Linear programming: foundations and extensions, 3rd edn. Springer, New YorkWingate D, Seppi KD (2005) Prioritization methods for accelerating MDP solvers. J Mach Learn Res 6: 851–88

    Dutch Culture Wars: On the politics of gutting the arts

    Full text link
    “No one is safe.” With these words Halbe Zijlstra, the State Secretary of Education, Culture and Science, announced the slashing of the cultural budget on the Dutch national news in December 2010. Whereas cutbacks are generally accompanied by at least the pretension of reluctance or regret, Zijlstra delivered the message with a sardonic smile. It’s a rather uncommon spectacle: a State Secretary of Culture who publicly flaunts his disdain for culture. Zijlstra described artists as being on a “subsidy drip” and took care to present himself as an avowed fan of Dan Brown, Tom Clancy, McDonalds and Metallica. Known amongst artists as “Halbe the Wrecker,” he has become the embodiment of the anti-artistic and anti-intellectual sentiment in the Netherlands. Zijlstra became the figure of the philistine that the cultured classes love to hate. And he welcomes that hatred. The slashing of the cultural budget is a symbolical centrepiece of the Dutch culture wars, initiated during the recent rightward turn in Dutch politics. It is a conflict framed along similar coordinates as its American counterpart, where the conservative Right channels popular discontent in the direction of cultural elites, instead of the economic establishment. What distinguishes the Dutch culture wars from those on the other side of the Atlantic, is that conservative Christian values are largely absent from the debate. The American focus on religious values is replaced with a secular “Judeo-Christian” anti-Islamism and opposition to multiculturalism. These differences notwithstanding, the overall effect is similar: the egalitarian critique of culture, described by its right-wing populist detractors as a “left-wing hobby” or an “elitist plaything,” allows the Right to push an economic agenda that is decidedly less egalitarian. In this sense, the Dutch culture cuts illustrate the powerful appeal of what Wendy Brown has described as the contradictory convergence of neoliberalism and neo-conservatism.[1] Where the neoconservative attack on “liberal elites” allows for a popular appeal that neoliberalism would otherwise lack

    Type and timing of adverse childhood experiences differentially affect severity of PTSD, dissociative and depressive symptoms in adult inpatients

    Get PDF
    Background: A dose-dependent effect of Adverse Childhood Experiences (ACE) on the course and severity of psychiatric disorders has been frequently reported. Recent evidence indicates additional impact of type and timing of distinct ACE on symptom severity experienced in adulthood, in support of stress-sensitive periods in (brain) development. The present study seeks to clarify the impact of ACE on symptoms that are often comorbid across various diagnostic groups: symptoms of posttraumatic stress disorder (PTSD), shutdown dissociation and depression. A key aim was to determine and compare the importance of dose-dependent versus type and timing specific prediction of ACE on symptom levels. Methods: Exposure to ten types of maltreatment up to age 18 were retrospectively assessed in N = 129 psychiatric inpatients using the Maltreatment and Abuse Chronology of Exposure (MACE). Symptoms of PTSD, shutdown dissociation, and depression were related to type and timing of ACE. The predictive power of peak types and timings was compared to that of global MACE measures of duration, multiplicity and overall severity. Results: A dose-dependent effect (MACE duration, multiplicity and overall severity) on severity of all symptoms confirmed earlier findings. Conditioned random forest regression verified that PTSD symptoms were best predicted by overall ACE severity, whereas type and timing specific effects showed stronger prediction for symptoms of dissociation and depression. In particular, physical neglect at age 5 and emotional neglect at ages 4–5 were related to increased symptoms of dissociation, whereas the emotional neglect at age 8–9 enhanced symptoms of depression. Conclusion: In support of the sensitive period of exposure model, present results indicate augmented vulnerability by type x timing of ACE, in particular emphasizing pre-school (age 4–5) and pre-adolescent (8–9) periods as sensitive for the impact of physical and emotional neglect. PTSD, the most severe stress-related disorder, varies with the amount of adverse experiences irrespective of age of experience. Considering type and timing of ACE improves understanding of vulnerability, and should inform diagnostics of psychopathology like PTSD, dissociation and depression in adult psychiatric patients

    Simulation of Acoustic Wave Propagation in Aluminium Coatings for Material Characterization

    Full text link
    Aluminium coatings and their characterization are of great interest in many fields of application, ranging from aircraft industries to microelectronics. Here, we present the simulation of acoustic wave propagation in aluminium coatings via the elastodynamic finite integration technique (EFIT) in comparison to experimental results. The simulations of intensity (I)–defocus (z) curves, obtained by scanning acoustic microscopy (SAM), were first carried out on an aluminium bulk sample, and secondly on a 1 µm aluminium coating deposited on a silicon substrate. The I(z) curves were used to determine the Rayleigh wave velocity of the aluminium bulk sample and the aluminium coating. The results of the simulations with respect to the Rayleigh velocity were corroborated by non-destructive SAM measurements and laser ultrasonic measurements (LUS)

    Steps toward Maturation of Embryonic Stem Cell-Derived Cardiomyocytes by Defined Physical Signals

    Full text link
    Cardiovascular disease remains a leading cause of mortality and morbidity worldwide. Embryonic stem cell-derived cardiomyocytes (ESC-CMs) may offer significant advances in creating in vitro cardiac tissues for disease modeling, drug testing, and elucidating developmental processes; however, the induction of ESCs to a more adult-like CM phenotype remains challenging. In this study, we developed a bioreactor system to employ pulsatile flow (1.48 mL/min), cyclic strain (5%), and extended culture time to improve the maturation of murine and human ESC-CMs. Dynamically-cultured ESC-CMs showed an increased expression of cardiac-associated proteins and genes, cardiac ion channel genes, as well as increased SERCA activity and a Raman fingerprint with the presence of maturation-associated peaks similar to primary CMs. We present a bioreactor platform that can serve as a foundation for the development of human-based cardiac in vitro models to verify drug candidates, and facilitates the study of cardiovascular development and disease
    corecore