Scaling multiagent reinforcement learning

Reinforcement learning in real-world domains suffers from three curses of dimensionality: explosions in state and action spaces, and high stochasticity or "outcome space" explosion. Multiagent domains are particularly susceptible to these problems. This thesis describes ways to mitigate these curses in several different multiagent domains, including real-time delivery of products using multiple vehicles with stochastic demands, a multiagent predator-prey domain, and a domain based on a real-time strategy game. To mitigate the problem of state-space explosion, this thesis present several approaches that mitigate each of these curses. "Tabular linear functions" (TLFs) are introduced that generalize tile-coding and linear value functions and allow learning of complex nonlinear functions in high-dimensional state-spaces. It is also shown how to adapt TLFs to relational domains, creating a "lifted" version called relational templates. To mitigate the problem of action-space explosion, the replacement of complete joint action space search with a form of hill climbing is described. To mitigate the problem of outcome space explosion, a more efficient calculation of the expected value of the next state is shown, and two real-time dynamic programming algorithms based on afterstates, ASH-learning and ATR-learning, are introduced. Lastly, two approaches that scale by treating a multiagent domain as being formed of several coordinating agents are presented. "Multiagent H-learning" and "Multiagent ASH-learning" are described, where coordination is achieved through a method called "serial coordination". This technique has the benefit of addressing each of the three curses of dimensionality simultaneously by reducing the space of states and actions each local agent must consider. The second approach to multiagent coordination presented is "assignment-based decomposition", which divides the action selection step into an assignment phase and a primitive action selection step. Like the multiagent approach, assignment-based decomposition addresses all three curses of dimensionality simultaneously by reducing the space of states and actions each group of agents must consider. This method is capable of much more sophisticated coordination. Experimental results are presented which show successful application of all methods described. These results demonstrate that the scaling techniques described in this thesis can greatly mitigate the three curses of dimensionality and allow solutions for multiagent domains to scale to large numbers of agents, and complex state and outcome spaces

Economic Evaluation and Transferability of Physical Activity Programmes in Primary Prevention: A Systematic Review

This systematic review aims to assess the characteristics of, and the clinical and economic evidence provided by, economic evaluations of primary preventive physical exercise interventions, and to analyse their transferability to Germany using recommended checklists. Fifteen economic evaluations from seven different countries met eligibility criteria, with seven of the fifteen providing high economic evidence in the special country context. Most of the identified studies conclude that the investigated intervention provide good value for money compared with alternatives. However, this review shows a high variability of the costing methods between the studies, which limits comparability, generalisability and transferability of the results

Effects of work ability and health promoting interventions for women with musculoskeletal symptoms: A 9-month prospective study

<p>Abstract</p> <p>Background</p> <p>Women working in the public human service sector in 'overstrained' situations run the risk of musculoskeletal symptoms and long-term sick leave. In order to maintain the level of health and work ability and strengthen the potential resources for health, it is important that employees gain greater control over decisions and actions affecting their health – a process associated with the concept of self-efficacy. The aim of this study was to describe the effects of a self-efficacy intervention and an ergonomic education intervention for women with musculoskeletal symptoms, employed in the public sector.</p> <p>Methods</p> <p>The design of the study was a 9-month prospective study describing the effects of two interventions, a comprehensive self-efficacy intervention (<it>n </it>= 21) and an ergonomic education intervention (<it>n </it>= 21). Data were obtained by a self-report questionnaire on health- and work ability-related factors at baseline, and at ten weeks and nine months follow-up. Within-group differences over time were analysed.</p> <p>Results</p> <p>Over the time period studied there were small magnitudes of improvements within each group. Within the self-efficacy intervention group positive effects in perceived work ability were shown. The ergonomic education group showed increased positive beliefs about future work ability and a more frequent use of pain coping strategies.</p> <p>Conclusion</p> <p>Both interventions showed positive effects on women with musculoskeletal symptoms, but in different ways. Future research in this area should tailor interventions to participants' motivation and readiness to change.</p

Multiagent Transfer Learning via Assignment-based Decomposition

Abstract—We describe a system that successfully transfers value function knowledge across multiple subdomains of realtime strategy games in the context of multiagent reinforcement learning. First, we implement an assignment-based decomposition architecture, which decomposes the problem of coordinating multiple agents into the two levels of task assignment and task execution. Second, a hybrid model-based approach allows us to use simple deterministic action models while relying on sampling for the opponents ’ actions. Third, value functions based on parameterized relational templates enable transfer across sub-domains with different numbers of agents. Keywords-reinforcement learning; markov decision processes; assignment problem; coordination; transfer learning I