Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks

Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics

Variational Recurrent Models for Solving Partially Observable Control Tasks

In partially observable (PO) environments, deep reinforcement learning (RL) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how to improve the policy. In this study, we propose an RL algorithm for solving PO tasks. Our method comprises two parts: a variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM. The proposed algorithm was tested in two types of PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization. Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner.Comment: Published as a conference paper at the Eighth International Conference on Learning Representations (ICLR 2020

トクシマ ダイガク ニオケル チイキ イリョウ ニ コウケン スル イシ ノ イクセイ

The shortage of medical doctors is now a severe social problem in Japan. Tokushima Prefecture has the third highest population of medical doctors in Japan in ２０１２, but the shortage of doctors in rural areas was severely seen by an uneven distribution. Primary care practice was started in the education of clinical clerkship for medical students of our university since July,２００８. They have rounded a variety of medical facilities during one week mainly in the south of Tokushima Prefecture. Since the practice increased the passion of medical students in working at community medicine and medicine in remote area, it is important to prepare more courses to learn primary care and general medicine in our clinical practice system to continue the interest and passion in community medicine. Moreover, it is important that we made the educational system for general medicine which connects before and after the graduation

チイキ イリョウ ノ ジュウジツ ト キソ イガク ケンキュウ ワ リョウリツ スルカ

The number of doctors per residents is different among４７Prefectures in Japan. Tokushima Prefecture has more doctors per 0.1 million residents（approximately 260）than average number in Japan（approximately 200）. However, Tokushima has severe problems in a shortage of the number of doctors as well as other Prefectures because of an uneven distribution of doctors in the Prefecture. A shortage of the number of doctors in community medicine induced the decreased number of clinical doctors in the University Hospital which resulted in a decrease of the number of researchers corresponding to basic research. To relieve a break-down of community medicine in Tokushima, we have being done various trials in education of medical students and research in community medicine. These trials will improve the situation of community medicine which may result in an increase of human resource not only in community medicine but also basic research

Habits and goals in synergy: a variational Bayesian framework for behavior

How to behave efficiently and flexibly is a central problem for understanding biological agents and creating intelligent embodied AI. It has been well known that behavior can be classified as two types: reward-maximizing habitual behavior, which is fast while inflexible; and goal-directed behavior, which is flexible while slow. Conventionally, habitual and goal-directed behaviors are considered handled by two distinct systems in the brain. Here, we propose to bridge the gap between the two behaviors, drawing on the principles of variational Bayesian theory. We incorporate both behaviors in one framework by introducing a Bayesian latent variable called "intention". The habitual behavior is generated by using prior distribution of intention, which is goal-less; and the goal-directed behavior is generated by the posterior distribution of intention, which is conditioned on the goal. Building on this idea, we present a novel Bayesian framework for modeling behaviors. Our proposed framework enables skill sharing between the two kinds of behaviors, and by leveraging the idea of predictive coding, it enables an agent to seamlessly generalize from habitual to goal-directed behavior without requiring additional training. The proposed framework suggests a fresh perspective for cognitive science and embodied AI, highlighting the potential for greater integration between habitual and goal-directed behaviors

Home caregiver anxiety and depression

In recent years, Japan has promoted home visits to support older adults, with caregivers playing an important role. However, caregivers generally experience a high frequency of anxiety and depression, and the factors associated with these conditions among home visit caregivers remain unknown. To identify the associated factors, we conducted a questionnaire-based cross-sectional study of home visit caregivers in Tokushima Prefecture, Japan. The survey included caregivers’ Hospital Anxiety and Depression Scale ; sociodemographic items of patients and caregivers ; and caregivers’ perceptions of the home care environment, patients, and themselves. The questionnaires were sent to 379 caregivers ; 203 responded (53.6% response rate), of which 173 were valid (85.2% valid response rate). The prevalence of anxiety and depression was 43.9% and 69.4%, respectively. Multiple logistic regression analysis of factors associated with anxiety and depression showed that stable family finances (OR : 0.69, 95% CI : 0.48-1.00, p = 0.049) and stable caregiver health (OR : 0.45, 95% CI : 0.30-0.68, p < 0.001) were associated with anxiety. Further, stable family finances (OR : 0.60, 95% CI : 0.38-0.93, p = 0.022), stable caregiver health (OR : 0.49, 95% CI : 0.30-0.81, p = 0.005), and stable patient condition (OR : 0.51, 95% CI : 0.29-0.92, p = 0.025) were associated with depression. These findings demonstrate that caregiver wellbeing is essential in home care settings

イガクセイ ニ タイスル チイキ イリョウ キョウイク ノ ジッセン ト ソノ ヒョウカ

The shortage of doctors has been a major problem in community medicine in Japan. Tokushima Prefecture has more doctors than an average of Japan, but the shortage is seen in parts of the south and west in the Prefecture due to an uneven distribution of doctors. Therefore, lectures and practices to learn community medicine and general medicine have been newly started in a curriculum of Tokushima University since 2008. In this review, we showed results of a questionnaire for medical students about community medicine and general medicine which were obtained just before the beginning of the education, and discussed how to evaluate the system of education

アタラシイ リウマチ チリョウホウ

Rheumatoid arthritis (RA) is a chronic inflammatory disease characterized by the progressive destruction of joints. Cytokines such as tumor necrosis factor (TNF)-α, interleukin (IL)-1, IL-6 and IL-8 have a critical role in the pathogenesis of RA. Recently, etanercept and infliximab, which block the action of TNF-α, can reduce the disease activity of RA. They act more rapidly to decrease symptoms and slow joint damage in patients with early active RA than methotrexate. This article describes recent studies on anti-cytokine therapy in RA