Rule Value Reinforcement Learning for Cognitive Agents

RVRL (Rule Value Reinforcement Learning) is a new algorithm which extends an existing learning framework that models the environment of a situated agent using a probabilistic rule representation. The algorithm attaches values to learned rules by adapting reinforcement learning. Structure captured by the rules is used to form a policy. The resulting rule values represent the utility of taking an action if the rule`s conditions are present in the agent`s current percept. Advantages of the new framework are demonstrated, through examples in a predator-prey environment

Modelling Emotion Based Reward Valuation with Computational Reinforcement Learning

We show that computational reinforcement learning can model human decision making in the Iowa Gambling Task (IGT). The IGT is a card game, which tests decision making under uncertainty. In our experiments, we found that modulating learning rate decay in Q-learning, enables the approximation of both the behaviour of normal subjects and those who are emotionally impaired by ventromedial prefrontal lesions. Outcomes observed in impaired subjects are modeled by high learning rate decay, while low learning rate decay replicates healthy subjects under otherwise identical conditions. The ventromedial prefrontal cortex has been associated with emotion based reward valuation, and, the value function in reinforcement learning provides an analogous assessment mechanism. Thus reinforcement learning can provide a good model for the role of emotional reward as a modulator of the learning rate

Learning to Act with RVRL Agents

The use of reinforcement learning to guide action selection of cognitive agents has been shown to be a powerful technique for stochastic environments. Standard Reinforcement learning techniques used to provide decision theoretic policies rely, however, on explicit state-based computations of value for each state-action pair. This requires the computation of a number of values exponential to the number of state variables and actions in the system. This research extends existing work with an acquired probabilistic rule representation of an agent environment by developing an algorithm to apply reinforcement learning to values attached to the rules themselves. Structure captured by the rules is then used to learn a policy directly. The resulting value attached to each rule represents the utility of taking an action if the conditions of the rule are present in the agent’s current set of percepts. This has several advantages for planning purposes: generalization over many states and over unseen states; effective decisions can therefore be made with less training data than state based modelling systems (e.g. Dyna Q-Learning); and the problem of computation in an exponential state-action space is alleviated. The results of application of this algorithm to rules in a specific environment are presented, with comparison to standard reinforcement learning policies developed from related work

NPCs as People, Too: The Extreme AI Personality Engine

PK Dick once asked “Do Androids Dream of Electric Sheep?” In video games, a similar question could be asked of non-player characters: Do NPCs have dreams? Can they live and change as humans do? Can NPCs have personalities, and can these develop through interactions with players, other NPCs, and the world around them? Despite advances in personality AI for games, most NPCs are still undeveloped and undeveloping, reacting with flat affect and predictable routines that make them far less than human— in fact, they become little more than bits of the scenery that give out parcels of information. This need not be the case. Extreme AI, a psychology-based personality engine, creates adaptive NPC personalities. Originally developed as part of the thesis “NPCs as People: Using Databases and Behaviour Trees to Give Non-Player Characters Personality,” Extreme AI is now a fully functioning personality engine using all thirty facets of the Five Factor model of personality and an AI system that is live throughout gameplay. This paper discusses the research leading to Extreme AI; develops the ideas found in that thesis; discusses the development of other personality engines; and provides examples of Extreme AI’s use in two game demos

Power loss in open cavity diodes and a modified Child Langmuir Law

Diodes used in most high power devices are inherently open. It is shown that under such circumstances, there is a loss of electromagnetic radiation leading to a lower critical current as compared to closed diodes. The power loss can be incorporated in the standard Child-Langmuir framework by introducing an effective potential. The modified Child-Langmuir law can be used to predict the maximum power loss for a given plate separation and potential difference as well as the maximum transmitted current for this power loss. The effectiveness of the theory is tested numerically.Comment: revtex4, 11 figure

Development of a Virtual Laparoscopic Trainer using Accelerometer Augmented Tools to Assess Performance in Surgical training

Previous research suggests that virtual reality (VR) may supplement conventional training in laparoscopy. It may prove useful in the selection of surgical trainees in terms of their dexterity and spatial awareness skills in the near future. Current VR training solutions provide levels of realism and in some instances, haptic feedback, but they are cumbersome by being tethered and not ergonomically close to the actual surgical instruments for weight and freedom of use factors. In addition, they are expensive hence making them less accessible to departments than conventional box trainers. The box trainers in comparison, although more economical, lack tangible feedback and realism for handling delicate tissue structures. We have previously reported on the development of a modified digitally enhanced surgical instrument for laparoscopic training, named the Parkar Tool. This tool contains wireless accelerometer and gyroscopic sensors integrated into actual laparoscopic instruments. By design, it alleviates the need for both tethered and physically different shaped tools thereby enhancing the realism when performing surgical procedures. Additionally the software (Valhalla) has the ability to digitally record surgical motions, thereby enabling it to remotely capture surgical training data to analyse and objectively evaluate performance. We have adapted and further developed our initial single training tool method as used with a laparoscopic pyloromyotomy scenario, to an enhanced method using multiple Parkar wireless tools simultaneously, for use in several different case scenarios. This allows the use and measurement of right and left handed dexterity with the benefit of using several tasks of differing complexity. The development of a 3D tissue-surface deformations solution written in OpenGL gives us several different virtual surgical training scenario approximations to use with the instruments. The trainee can start with learning simple tasks e.g. incising tissue, grasping, squeezing and stretching tissue, to more complex procedures such as suturing, herniotomies, bowel anastomoses, as well as the original pyloromyotomy as used in the first model

Use of personal child health records in the UK: findings from the millennium cohort study.

OBJECTIVES: The personal child health record (PCHR) is a record of a child's growth, development, and uptake of preventive health services, designed to enhance communication between parents and health professionals. We examined its use throughout the United Kingdom with respect to recording children's weight and measures of social disadvantage and infant health. DESIGN: Cross sectional survey within a cohort study. SETTING: UK. PARTICIPANTS: Mothers of 18,503 children born between 2000 and 2002, living in the UK at 9 months of age. MAIN OUTCOME MEASURES: Proportion of mothers able to produce their child's PCHR; proportion of PCHRs consulted containing record of child's last weight; effective use of the PCHR (defined as production, consultation, and child's last weight recorded). RESULTS: In all, 16,917 (93%) mothers produced their child's PCHR and 15,138 (85%) mothers showed effective use of their child's PCHR. Last weight was recorded in 97% of PCHRs consulted. Effective use was less in children previously admitted to hospital, and, in association with factors reflecting social disadvantage, including residence in disadvantaged communities, young maternal age, large family size (four or more children; incidence rate ratio 0.87; 95% confidence interval 0.83 to 0.91), and lone parent status (0.88; 0.86 to 0.91). CONCLUSIONS: Use of the PCHR is lower by women living in disadvantaged circumstances, but overall the record is retained and used by a high proportion of all mothers throughout the UK in their child's first year of life. PCHR use is endorsed in the National Service Framework for Children and has potential benefits which extend beyond the direct care of individual children

Feasibility of collecting oral fluid samples in the home setting to determine seroprevalence of infections in a large-scale cohort of preschool-aged children

Oral fluid is a non-invasive biological sample, which can be returned by post, making it suitable for large-scale epidemiological studies in children. We report our experience of oral fluid collection from 14 373 preschool-aged children in the UK Millennium Cohort Study. Samples were collected by mothers in the home setting following the guidance of trained interviewers, and posted to the laboratory. Samples were received from 11698 children (81.4 %). Children whose mothers were of Black Caribbean ethnicity and who lived in non-English-speaking households were less likely to provide a sample, and those with a maternal history of asthma more likely to provide a sample [adjusted risk ratio (95 % CI) 0.85 (0.73-0.98), 0.87 (0.77-0.98) and 1.03 (1.00-1.05) respectively]. Collection of oral fluid samples is feasible and acceptable in large-scale child cohort studies. Formal interpreter support may be required to increase participation rates in surveys that collect biological samples from ethnic minorities

An ecological systems approach to examining risk factors for early childhood overweight: findings from the UK Millennium Cohort Study

Objective: To use an ecological systems approach to examine individual-, family-, community- and area-level risk factors for overweight (including obesity) in 3-year-old children. Methods: A prospective nationally representative cohort study conducted in England, Wales, Scotland, Northern Ireland. Participants included 13 188 singleton children aged 3 years in the Millennium Cohort Study, born between 2000 and 2002, who had complete height/weight data. The main outcome measure was childhood overweight (including obesity) defined by the International Obesity TaskForce cut-offs for body mass index. Results: 23.0% of 3-year-old children were overweight or obese. In the fully adjusted model, primarily individual- and family-level factors were associated with early childhood overweight: birthweight z-score (adjusted odds ratio, 1.36, 95% CI 1.30 to 1.42), black ethnicity (1.41, 1.11 to 1.80) (compared with white), introduction to solid foods or =21 hours/week (1.23, 1.10 to 1.37) (compared with never worked). Breastfeeding > or =4 months (0.86, 0.76 to 0.97) (compared with none) and Indian ethnicity (0.63, 0.42 to 0.94) were associated with a decreased risk of early childhood overweight. Children from Wales were also more likely to be overweight than children from England. Conclusions: Most risk factors for early childhood overweight are modifiable or would allow at-risk groups to be identified. Policies and interventions should focus on parents and providing them with an environment to support healthy behaviours for themselves and their children