Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES

Towards a hemodynamic model to characterize inaccuracies in finger pulse oximetry

A pulse oximeter monitors a patient's functional arterial oxygen saturation (SpO2) by illuminating vascularized tissue. However, the optical signals measured, called photoplethysmograms (PPGs), are easily distorted by motion, leading to inaccurate SpO2 readings. Motion artifacts in PPGs are partly attributed to hemodynamic variations, though the exact mechanism is not understood. This paper introduces a model-based approach to improve insight in the effects of hemodynamic variations on SpO2. To make a first step towards an improved understanding of hemodynamic variations, a hemodynamic fingertip PPG model has been developed, including hydrostatic pressures. Measurements on a healthy male subject show that the PPG model can explain changes in PPG baseline and pulsatility in a limited range of arterial and venous pressures. The measurements moreover indicate that modeling of blood flow regulations is required to explain transients in PPGs and inaccurate SpO2 readings in more situations

Responding to the increased genetic risk associated with customary consanguineous marriage among minority ethnic populations: lessons from local innovations in England.

Populations practising customary consanguineous marriage have a higher incidence of autosomal recessive genetic disorders than those in which reproductive partners are usually unrelated. In the absence of any national-level response, English service developments to address the additional needs of families living with or at risk of such disorders have been locally led. These interventions remain in their infancy here, as elsewhere in Europe, and important questions remain regarding how appropriate, effective and sustainable responses can be operationalised in practice. This formative service review employed four local case studies together with wider consultation exercises over a 4-year period (2011-2015) to document recent responses to this area of need, issues arising and lessons to inform future work. Service components included the following: enhancements to genetic services to provide family-centred, culturally competent approaches to counselling and testing; community genetic literacy approaches; and capacity development among health professionals. Local approaches were, however, very varied in their detail, scope, level of investment and longevity. The provisions of culturally competent genetic counselling services and community-level genetic literacy interventions were generally well received by those who accessed them. Coordinated action across all service components appeared important for an effective service, but healthcare professionals, particularly general practitioners, were often difficult to engage in this agenda. An evaluative culture and engagement in a wider community of practice had supported service development across sites. However, sustaining investment was challenging, particularly where new services were not well integrated into core provision and where commissioning was driven by expectations of short-term reductions in infant mortality and disability

Supersymmetry Without Prejudice

We begin an exploration of the physics associated with the general CP-conserving MSSM with Minimal Flavor Violation, the pMSSM. The 19 soft SUSY breaking parameters in this scenario are chosen so as to satisfy all existing experimental and theoretical constraints assuming that the WIMP is a conventional thermal relic, ie, the lightest neutralino. We scan this parameter space twice using both flat and log priors for the soft SUSY breaking mass parameters and compare the results which yield similar conclusions. Detailed constraints from both LEP and the Tevatron searches play a particularly important role in obtaining our final model samples. We find that the pMSSM leads to a much broader set of predictions for the properties of the SUSY partners as well as for a number of experimental observables than those found in any of the conventional SUSY breaking scenarios such as mSUGRA. This set of models can easily lead to atypical expectations for SUSY signals at the LHC.Comment: 61 pages, 24 figs. Refs., figs, and text added, typos fixed; This version has reduced/bitmapped figs. For a version with better figs please go to http://www.slac.stanford.edu/~rizz

Advances in infrastructures and tools for multiagent systems

In the last few years, information system technologies have focused on solving challenges in order to develop distributed applications. Distributed systems can be viewed as collections of service-provider and ser vice-consumer components interlinked by dynamically defined workflows (Luck and McBurney 2008).Alberola Oltra, JM.; Botti Navarro, VJ.; Such Aparicio, JM. (2014). Advances in infrastructures and tools for multiagent systems. Information Systems Frontiers. 16:163-167. doi:10.1007/s10796-014-9493-6S16316716Alberola, J. M., Búrdalo, L., Julián, V., Terrasa, A., & García-Fornes, A. (2014). An adaptive framework for monitoring agent organizations. Information Systems Frontiers, 16(2). doi: 10.1007/s10796-013-9478-x .Alfonso, B., Botti, V., Garrido, A., & Giret, A. (2014). A MAS-based infrastructure for negotiation and its application to a water-right market. Information Systems Frontiers, 16(2). doi: 10.1007/s10796-013-9443-8 .Andrighetto, G., Castelfranchi, C., Mayor, E., McBreen, J., López-Sánchez, M., & Parsons, S. (2013). (Social) norm dynamics. In G. Andrighetto, G. Governatori, P. Noriega, & L. W. van der Torre (Eds.), Normative multi-agent systems (pp. 135–170). Dagstuhl: Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik.Baarslag, T., Fujita, K., Gerding, E. H., Hindriks, K., Ito, T., Jennings, N. R., et al. (2013). Evaluating practical negotiating agents: results and analysis of the 2011 international competition. Artificial Intelligence, 198, 73–103.Boissier, O., Bordini, R. H., Hübner, J. F., Ricci, A., & Santi, A. (2013). Multi-agent oriented programming with JaCaMo. Science of Computer Programming, 78(6), 747–761.Campos, J., Esteva, M., López-Sánchez, M., Morales, J., & Salamó, M. (2011). Organisational adaptation of multi-agent systems in a peer-to-peer scenario. Computing, 91(2), 169–215.Carrera, A., Iglesias, C. A., & Garijo, M. (2014). Beast methodology: an agile testing methodology for multi-agent systems based on behaviour driven development. Information Systems Frontiers, 16(2). doi: 10.1007/s10796-013-9438-5 .Criado, N., Such, J. M., & Botti, V. (2014). Norm reasoning services. Information Systems Frontiers, 16(2). doi: 10.1007/s10796-013-9444-7 .Del Val, E., Rebollo, M., & Botti, V. (2014). Enhancing decentralized service discovery in open service-oriented multi-agent systems. Journal of Autonomous Agents and Multi-Agent Systems, 28(1), 1–30.Denti, E., Omicini, A., & Ricci, A. (2002). Coordination tools for MAS development and deployment. Applied Artificial Intelligence, 16(9–10), 721–752.Dignum, V., & Dignum, F. (2012). A logic of agent organizations. Logic Journal of IGPL, 20(1), 283–316.Ferber, J., & Gutknecht, O. (1998). A meta-model for the analysis and design of organizations in multi-agent systems. In Multi agent systems. Proceedings. International Conference on (pp. 128–135). IEEE.Fogués, R. L., Such, J. M., Espinosa, A., & Garcia-Fornes, A. (2014). BFF: a tool for eliciting tie strength and user communities in social networking services. Information Systems Frontiers, 16(2). doi: 10.1007/s10796-013-9453-6 .Garcia, E., Giret, A., & Botti, V. (2011). Evaluating software engineering techniques for developing complex systems with multiagent approaches. Information and Software Technology, 53(5), 494–506.Garcia-Fornes, A., Hübner, J., Omicini, A., Rodriguez-Aguilar, J., & Botti, V. (2011). Infrastructures and tools for multiagent systems for the new generation of distributed systems. Engineering Applications of Articial Intelligence, 24(7), 1095–1097.Jennings, N., Faratin, P., Lomuscio, A., Parsons, S., Sierra, C., & Wooldridge, M. (2001). Automated negotiation: prospects, methods and challenges. International Journal of Group Decision and Negotiation, 10(2), 199–215.Jung, Y., Kim, M., Masoumzadeh, A., & Joshi, J. B. (2012). A survey of security issue in multi-agent systems. Artificial Intelligence Review, 37(3), 239–260.Kota, R., Gibbins, N., & Jennings, N. R. (2012). Decentralized approaches for self-adaptation in agent organizations. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 7(1), 1.Kraus, S. (1997). Negotiation and cooperation in multi-agent environments. Artificial Intelligence, 94(1), 79–97.Lin, Y. I., Chou, Y. W., Shiau, J. Y., & Chu, C. H. (2013). Multi-agent negotiation based on price schedules algorithm for distributed collaborative design. Journal of Intelligent Manufacturing, 24(3), 545–557.Luck, M., & McBurney, P. (2008). Computing as interaction: agent and agreement technologies.Luck, M., McBurney, P., Shehory, O., & Willmott, S. (2005). Agent technology: Computing as interaction (A roadmap for agent based computing). AgentLink.Ossowski, S., & Menezes, R. (2006). On coordination and its significance to distributed and multiagent systems. Concurrency and Computation: Practice and Experience, 18(4), 359–370.Ossowski, S., Sierra, C., & Botti. (2013). Agreement technologies: A computing perspective. In Agreement Technologies (pp. 3–16). Springer Netherlands.Pinyol, I., & Sabater-Mir, J. (2013). Computational trust and reputation models for open multi-agent systems: a review. Artificial Intelligence Review, 40(1), 1–25.Ricci, A., Piunti, M., & Viroli, M. (2011). Environment programming in multi-agent systems: an artifact-based perspective. Autonomous Agents and Multi-Agent Systems, 23(2), 158–192.Sierra, C., & Debenham, J. (2006). Trust and honour in information-based agency. In Proceedings of the 5th international conference on autonomous agents and multi agent systems, (p. 1225–1232). New York: ACM.Sierra, C., Botti, V., & Ossowski, S. (2011). Agreement computing. KI-Knstliche Intelligenz, 25(1), 57–61.Vasconcelos, W., García-Camino, A., Gaertner, D., Rodríguez-Aguilar, J. A., & Noriega, P. (2012). Distributed norm management for multi-agent systems. Expert Systems with Applications, 39(5), 5990–5999.Wooldridge, M. (2002). An introduction to multiagent systems. New York: Wiley.Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: theory and practice. Knowledge Engineering Review, 10(2), 115–152