Avoiding Wireheading with Value Reinforcement Learning

How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) is a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward -- the so-called wireheading problem. In this paper we suggest an alternative to RL called value reinforcement learning (VRL). In VRL, agents use the reward signal to learn a utility function. The VRL setup allows us to remove the incentive to wirehead by placing a constraint on the agent's actions. The constraint is defined in terms of the agent's belief distributions, and does not require an explicit specification of which actions constitute wireheading.Comment: Artificial General Intelligence (AGI) 201

Power laws, Pareto distributions and Zipf's law

When the probability of measuring a particular value of some quantity varies inversely as a power of that value, the quantity is said to follow a power law, also known variously as Zipf's law or the Pareto distribution. Power laws appear widely in physics, biology, earth and planetary sciences, economics and finance, computer science, demography and the social sciences. For instance, the distributions of the sizes of cities, earthquakes, solar flares, moon craters, wars and people's personal fortunes all appear to follow power laws. The origin of power-law behaviour has been a topic of debate in the scientific community for more than a century. Here we review some of the empirical evidence for the existence of power-law forms and the theories proposed to explain them.Comment: 28 pages, 16 figures, minor corrections and additions in this versio

Suggesting Cooking Recipes Through Simulation and Bayesian Optimization

Cooking typically involves a plethora of decisions about ingredients and tools that need to be chosen in order to write a good cooking recipe. Cooking can be modelled in an optimization framework, as it involves a search space of ingredients, kitchen tools, cooking times or temperatures. If we model as an objective function the quality of the recipe, several problems arise. No analytical expression can model all the recipes, so no gradients are available. The objective function is subjective, in other words, it contains noise. Moreover, evaluations are expensive both in time and human resources. Bayesian Optimization (BO) emerges as an ideal methodology to tackle problems with these characteristics. In this paper, we propose a methodology to suggest recipe recommendations based on a Machine Learning (ML) model that fits real and simulated data and BO. We provide empirical evidence with two experiments that support the adequacy of the methodology

'Special K' and a loss of cell-to-cell adhesion in proximal tubule-derived epithelial cells: modulation of the adherens junction complex by ketamine

Ketamine, a mild hallucinogenic class C drug, is the fastest growing ‘party drug’ used by 16–24 year olds in the UK. As the recreational use of Ketamine increases we are beginning to see the signs of major renal and bladder complications. To date however, we know nothing of a role for Ketamine in modulating both structure and function of the human renal proximal tubule. In the current study we have used an established model cell line for human epithelial cells of the proximal tubule (HK2) to demonstrate that Ketamine evokes early changes in expression of proteins central to the adherens junction complex. Furthermore we use AFM single-cell force spectroscopy to assess if these changes functionally uncouple cells of the proximal tubule ahead of any overt loss in epithelial cell function. Our data suggests that Ketamine (24–48 hrs) produces gross changes in cell morphology and cytoskeletal architecture towards a fibrotic phenotype. These physical changes matched the concentration-dependent (0.1–1 mg/mL) cytotoxic effect of Ketamine and reflect a loss in expression of the key adherens junction proteins epithelial (E)- and neural (N)-cadherin and β-catenin. Down-regulation of protein expression does not involve the pro-fibrotic cytokine TGFβ, nor is it regulated by the usual increase in expression of Slug or Snail, the transcriptional regulators for E-cadherin. However, the loss in E-cadherin can be partially rescued pharmacologically by blocking p38 MAPK using SB203580. These data provide compelling evidence that Ketamine alters epithelial cell-to-cell adhesion and cell-coupling in the proximal kidney via a non-classical pro-fibrotic mechanism and the data provides the first indication that this illicit substance can have major implications on renal function. Understanding Ketamine-induced renal pathology may identify targets for future therapeutic intervention

Informed citizen and empowered citizen in health: results from an European survey

Background: The knowledge about the relationship between health-related activities on the Internet (i.e. informed citizens) and individuals? control over their own experiences of health or illness (i.e. empowered citizens) is valuable but scarce. In this paper, we investigate the correlation between four ways of using the Internet for information on health or illness and citizens attitudes and behaviours toward health professionals and health systems and establish the profile of empowered eHealth citizens in Europe. Methods: Data was collected during April and May 2007 (N = 7022), through computer-assisted telephone interviews (CATI). Respondents from Denmark, Germany, Greece, Latvia, Norway, Poland and Portugal participated in the survey. The profiles were generated using logistic regressions and are based on: a) socio-demographic and health information, b) the level of use of health-related online services, c) the level of use of the Internet to get health information to decide whether to consult a health professional, prepare for a medical appointment and assess its outcome, and d) the impact of online health information on citizens? attitudes and behavior towards health professionals and health systems. Results: Citizens using the Internet to decide whether to consult a health professional or to get a second opinion are likely to be frequent visitors of health sites, active participants of online health forums and recurrent buyers of medicines and other health related products online, while only infrequent epatients, visiting doctors they have never met face-to-face. Participation in online health communities seems to be related with more inquisitive and autonomous patients. Conclusions: The profiles of empowered eHealth citizens in Europe are situational and country dependent. The number of Europeans using the Internet to get health information to help them deal with a consultation is raising and having access to online health information seems to be associated with growing number of inquisitive and self-reliant patients. Doctors are increasingly likely to experience consultations with knowledgeable and empowered patients, who will challenge them in various ways

The effect of a multi-component intervention on disrespect and abuse during childbirth in Kenya

Background Disrespect and abuse (D & A) during labor and delivery are important issues correlated with human rights, equity, and public health that also affect women’s decisions to deliver in facilities, which provide appropriate management of maternal and neonatal complications. Little is known about interventions aimed at lowering the frequency of disrespectful and abusive behaviors. Methods Between 2011 and 2014, a pre-and-post study measured D & A levels in a three-tiered intervention at 13 facilities in Kenya under the Heshima project. The intervention involved working with policymakers to encourage greater focus on D & A, training providers on respectful maternity care, and strengthening linkages between the facility and community for accountability and governance. At participating facilities, postpartum women were approached at discharge and asked to participate in the study; those who consented were administered a questionnaire on D & A in general as well as six typologies, including physical and verbal abuse, violations of confidentiality and privacy, detainment for non-payment, and abandonment. Observation of provider-patient interaction during labor was also conducted in the same facilities. In both exit interview and observational studies, multivariate analyses of risk factors for D & A controlled for differences in socio-demographic and facility characteristics between baseline and endline surveys. Results Overall D & A decreased from 20–13 % (p < 0.004) and among four of the six typologies D & A decreased from 40–50 %. Night shift deliveries were associated with greater verbal and physical abuse. Patient and infant detainment declined dramatically from 8.0–0.8 %, though this was partially attributable to the 2013 national free delivery care policy. Conclusion Although a number of contextual factors may have influenced these findings, the magnitude and consistency of the observed decreases suggest that the multi-component intervention may have the potential to reduce the frequency of D & A. Greater efforts are needed to develop stronger evaluation methods for assessing D & A in other settings

N.Georgescu-Roegen's production model for EROI evaluation. Case study: Electrolytic H2 production using solar energy

Nowadays there is a considerable interest in studying the direct and indirect energies involved in products and services. This is particularly critical when novel energy resources are exploited by complex technological chains and to determine if they can indeed guarantee an useful energy societal supply. Unfortunately, there is no universally accepted procedure for doing this. The present paper aims to suggest a new procedure to evaluate the EROI of technologies producing energy carriers based on the stocks/flows-funds/services production model of N.G. Roegen. The suggested method can uniquely identify the energy flows involved in the technology consistent with biophysical and anthropological boundaries. This analytical formulation can be used either for single technologies or combination of them in series or parallel using different energy resources. Specific recommendations in the use of the Cumulative Energy Demand and Global Energy Requirements in the Net Energy Analysis, as well as in the evaluation of both for an electrical system are reported. The approach is here applied to the analysis of electrolytic H2 production using electricity produced by a photovoltaic panel ("green hydrogen"). The resulting EROI = 0.97 means that the technology is not sustainable, requiring 3% energy from the anthropological sphere to support it. The paper is organized as follows: providing a narrative model for EROI evaluation consistent with anthropological and biophysical spheres; covering the definition of stocks/flows-funds/services model for EROI evaluation; analysing and suggesting uses of the model for energy technologies scoring and selection based on sustainability and presenting a numerical case study

Bayesian optimization of the PC algorithm for learning Gaussian Bayesian networks

The PC algorithm is a popular method for learning the structure of Gaussian Bayesian networks. It carries out statistical tests to determine absent edges in the network. It is hence governed by two parameters: (i) The type of test, and (ii) its significance level. These parameters are usually set to values recommended by an expert. Nevertheless, such an approach can suffer from human bias, leading to suboptimal reconstruction results. In this paper we consider a more principled approach for choosing these parameters in an automatic way. For this we optimize a reconstruction score evaluated on a set of different Gaussian Bayesian networks. This objective is expensive to evaluate and lacks a closed-form expression, which means that Bayesian optimization (BO) is a natural choice. BO methods use a model to guide the search and are hence able to exploit smoothness properties of the objective surface. We show that the parameters found by a BO method outperform those found by a random search strategy and the expert recommendation. Importantly, we have found that an often overlooked statistical test provides the best over-all reconstruction results

Underwater near infrared spectroscopy (uNIRS) can measure training adaptations in adolescent swimmers

The development of an underwater near-infrared spectroscopy (uNIRS) device has enabled previously unattainable measurements of peripheral muscle hemodynamics and oxygenation to be taken within the natural aquatic environment. The purposes of this study were (i) to trial the use of uNIRS, in a real world training study, and (ii) to monitor the effects of a swim training program upon muscle oxygenation status in short distance swimming. 14 junior club level swimmers completed a repeated swim sprint test before and after an eight week endurance training program. A waterproof, portable NIRS device was attached to the vastus lateralis. uNIRS successfully measured changes in muscle oxygenation and blood volume in all individuals; rapid sub-second time resolution of the device was able to demonstrate muscle oxygenation changes during the characteristic swim movements. Post training heart rate recovery and swim performance time were significantly improved. uNIRS data also showed significant changes. A larger rise in deoxyhemoglobin during individual sprints suggested training induced an increase in muscle oxygen extraction; a faster recovery time for muscle oxygenation suggested positive training induced changes and significant changes in muscle blood volume also occur. As a strong correlation was seen between an increased reoxygenation rate and an improved swim performance time, these findings support the use of uNIRS as a new performance analysis tool in swimming