Accelerating Reinforcement Learning through the Discovery of Useful Subgoals

An ability to adjust to changing environments and unforeseen circumstances is likely to be an important component of a successful autonomous space robot. This paper shows how to augment reinforcement learning algorithms with a method for automatically discovering certain types of subgoals online. By creating useful new subgoals while learning, the agent is able to accelerate learning on a current task and to transfer its expertise to related tasks through the reuse of its ability to attain subgoals. Subgoals are created based on commonalities across multiple paths to a solution. We cast the task of finding these commonalities as a multiple-instance learning problem and use the concept of diverse density to find solutions. We introduced this approach in [10] and here we present additional results for a simulated mobile robot task

Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density

This paper presents a method by which a reinforcement learning agent can automatically discover certain types of subgoals online. By creating useful new subgoals while learning, the agent is able to accelerate learning on the current task and to transfer its expertise to other, related tasks through the reuse of its ability to attain subgoals. The agent discovers subgoals based on commonalities across multiple paths to a solution. We cast the task of finding these commonalities as a multiple-instance learning problem and use the concept of diverse density to find solutions. We illustrate this approach using several gridworld tasks

Characterizing the role of disparity information in alleviating visual crowding

The ability to identify a target is reduced by the presence of nearby objects, a phenomenon known as visual crowding. The extent to which crowding impairs our perception is generally governed by the degree of similarity between a target stimulus and its surrounding flankers. Here we investigated the influence of disparity differences between target and flankers on crowding. Orientation discrimination thresholds for a parafoveal target were first measured when the target and flankers were presented at the same depth to establish a flanker separation that induced a significant elevation in threshold for each individual. Flankers were subsequently fixed at this spatial separation while the disparity of the flankers relative to the target was altered. For all participants, thresholds showed a systematic decrease as flanker-target disparity increased. The resulting tuning function was asymmetric: Crowding was lower when the target was perceived to be in front of the flankers rather than behind. A series of control experiments confirmed that these effects were driven by disparity, as opposed to other factors such as flanker-target separation in three-dimensional (3-D) space or monocular positional offsets used to create disparity. When flankers were distributed over a range of crossed and uncrossed disparities, such that the mean was in the plane of the target, there was an equivalent or greater release of crowding compared to when all flankers were presented at the maximum disparity of that range. Overall, our results suggest that depth cues can reduce the effects of visual crowding, and that this reduction is unlikely to be caused by grouping of flankers or positional shifts in the monocular image

Romantic and sexual behavior in young adolescents : repeated surveys in a population-based cohort

Adverse outcomes of teenage sexual activity are common in the United Kingdom. We used a computer-assisted interview to ask young adolescents aged 11-12 years (N = 6856) and 12 -13 years (N = 6801) who were part of the Avon Longitudinal Study of Parents and Children about romantic and intimate behaviors. 24% of 11-12 year olds and 41% of 12-13 year olds reported holding hands and 17% and 32% respectively reported having been kissed on the mouth. A minority of 12-13 year olds reported sexual behavior including intercourse. The majority of participants who reported intimate sexual behavior did not regret it. Boys reported all activities more than girls and were less likely to express regret. Our findings suggest that the timing and content of sex education should take into account the frequency with which young people are experimenting with sexual behaviors

Counting atoms in a deep optical microtrap

We demonstrate a method to count small numbers of atoms held in a deep, microscopic optical dipole trap by collecting fluorescence from atoms exposed to a standing wave of light that is blue detuned from resonance. While scattering photons, the atoms are also cooled by a Sisyphus mechanism that results from the spatial variation in light intensity. The use of a small blue detuning limits the losses due to light assisted collisions, thereby making the method suitable for counting several atoms in a microscopic volume

Systematic review of adherence rates by medication class in type 2 diabetes: a study protocol

Introduction: Treatment options for type 2 diabetes are becoming increasingly complex with people often prescribed multiple medications, and may include both oral and injectable therapies. There is ongoing debate about which drug classes provide the optimum second-line and third-line treatment options. In the real world, patient adherence and persistence determines medication effectiveness. A better understanding of adherence may help inform the choice of second-line and third-line drug classes. Methods and analysis: This systematic review will compare adherence and persistence rates across the different classes of medication available to people with type 2 diabetes. It will include all identified studies comparing medication adherence or persistence between two or more glucose-lowering medications in people with type 2 diabetes. Research databases (MEDLINE, EMBASE, The Cochrane Library, The Register of Controlled Trials, PsychINFO and CINAHL) will be searched for relevant articles, using a comprehensive search strategy. All identified medication trials and observational studies will be included which compare adherence or persistence across classes of diabetes medication. The characteristics and outcomes of all the included studies will be reported along with a study quality grade, assessed using the Cochrane Risk Assessment Tool. The quality of adjustment for confounders of adherence or persistence will be reported for each study. Where multiple (n ≥3) studies provide compare adherence or persistence across the same 2 medication classes, a meta-analysis will be performed. Ethics and dissemination: No ethics approval is required. This review and meta-analysis (where possible) will provide important information on the relative patient adherence and persistence, with the different classes of diabetes therapies. Once complete, the results will be made available by peer-reviewed publication. Trial registration number: CRD42015027865

Evolution of local scour around a collared monopile through tidal cycles

This paper presents the results of an experiment designed to assess the time-development of scour around an offshore wind turbine collared monopile over a number of tidal cycles. One collar shape and location was investigated. The scour developed more slowly and the scour depth was shallower than for the case of a smooth monopile throughout the majority of the first half-cycle. This difference reduced quite rapidly during the second half-cycle and the scour depth at the end of two tidal cycles was essentially the same as for the smooth monopile. The time development of the scour was compared with results from existing empirical models for the time-development of scour under unidirectional flow. As expected, these models give a much smoother evolution of scour and different scour rates than those measured. Time variation in scour depth was better reproduced with a simplified approach for prediction of the time-varying development of scour. This also highlighted a problem with estimation of the time scale for the development of the equilibrium scour depth. Further investigations are needed before this alternative scour protection is completely rejected

CLIMB: Curriculum Learning for Infant-inspired Model Building

We describe our team's contribution to the STRICT-SMALL track of the BabyLM Challenge. The challenge requires training a language model from scratch using only a relatively small training dataset of ten million words. We experiment with three variants of cognitively-motivated curriculum learning and analyze their effect on the performance of the model on linguistic evaluation tasks. In the vocabulary curriculum, we analyze methods for constraining the vocabulary in the early stages of training to simulate cognitively more plausible learning curves. In the data curriculum experiments, we vary the order of the training instances based on i) infant-inspired expectations and ii) the learning behavior of the model. In the objective curriculum, we explore different variations of combining the conventional masked language modeling task with a more coarse-grained word class prediction task to reinforce linguistic generalization capabilities. Our results did not yield consistent improvements over our own non-curriculum learning baseline across a range of linguistic benchmarks; however, we do find marginal gains on select tasks. Our analysis highlights key takeaways for specific combinations of tasks and settings which benefit from our proposed curricula. We moreover determine that careful selection of model architecture, and training hyper-parameters yield substantial improvements over the default baselines provided by the BabyLM challenge

CLIMB: Curriculum Learning for Infant-inspired Model Building

We describe our team's contribution to the STRICT-SMALL track of the BabyLM Challenge. The challenge requires training a language model from scratch using only a relatively small training dataset of ten million words. We experiment with three variants of cognitively-motivated curriculum learning and analyze their effect on the performance of the model on linguistic evaluation tasks. In the vocabulary curriculum, we analyze methods for constraining the vocabulary in the early stages of training to simulate cognitively more plausible learning curves. In the data curriculum experiments, we vary the order of the training instances based on i) infant-inspired expectations and ii) the learning behavior of the model. In the objective curriculum, we explore different variations of combining the conventional masked language modeling task with a more coarse-grained word class prediction task to reinforce linguistic generalization capabilities. Our results did not yield consistent improvements over our own non-curriculum learning baseline across a range of linguistic benchmarks; however, we do find marginal gains on select tasks. Our analysis highlights key takeaways for specific combinations of tasks and settings which benefit from our proposed curricula. We moreover determine that careful selection of model architecture, and training hyper-parameters yield substantial improvements over the default baselines provided by the BabyLM challenge

The Impact of Depression on Patient Outcomes in Hip Arthroscopic Surgery.

Background: Mental health impairments have been shown to negatively affect preoperative self-reported function in patients with various musculoskeletal disorders, including those with femoroacetabular impingement. Hypothesis: Those with symptoms of depression will have lower self-reported function, more pain, and less satisfaction on initial assessment and at 2-year follow-up than those without symptoms of depression. Study Design: Cohort study; Level of evidence, 3. Methods: Patients who were enrolled in a multicenter hip arthroscopic surgery registry and had 2-year outcome data available were included in the study. Patients completed the 12-item International Hip Outcome Tool (iHOT-12), visual analog scale (VAS) for pain, and 12-item Short-Form Health Survey (SF-12) when consenting for surgery. At 2-year follow-up, patients were emailed the iHOT, the VAS, and a rating scale of surgical satisfaction. Initial SF-12 mental component summary (MCS) scores Results: A total of 781 patients achieved the approximate 2-year milestone (mean follow-up, 735 ± 68 days), with 651 (83%) having 2-year outcome data available. There were 434 (67%) female and 217 (33%) male patients, with a mean age of 35.8 ± 13.0 years and a mean body mass index of 25.4 ± 8.8 kg/m Conclusion: A large number of patients who underwent hip arthroscopic surgery presented with symptoms of depression, which negatively affected self-reported function, pain levels, and satisfaction on initial assessment and at 2-year follow-up. Surgeons who perform hip arthroscopic surgery may need to identify the symptoms of depression and be aware of the impact that depression can have on surgical outcomes