Count-Based Exploration with the Successor Representation

In this paper we introduce a simple approach for exploration in reinforcement learning (RL) that allows us to develop theoretically justified algorithms in the tabular case but that is also extendable to settings where function approximation is required. Our approach is based on the successor representation (SR), which was originally introduced as a representation defining state generalization by the similarity of successor states. Here we show that the norm of the SR, while it is being learned, can be used as a reward bonus to incentivize exploration. In order to better understand this transient behavior of the norm of the SR we introduce the substochastic successor representation (SSR) and we show that it implicitly counts the number of times each state (or feature) has been observed. We use this result to introduce an algorithm that performs as well as some theoretically sample-efficient approaches. Finally, we extend these ideas to a deep RL algorithm and show that it achieves state-of-the-art performance in Atari 2600 games when in a low sample-complexity regime.Comment: This paper appears in the Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI 2020

Assessing filtering of mountaintop CO2 mole fractions for application to inverse models of biosphere-atmosphere carbon exchange

pre-printThere is a widely recognized need to improve our understanding of biosphere-atmosphere carbon exchanges in areas of complex terrain including the United States Mountain West. CO2 fluxes over mountainous terrain are often difficult to measure due to unusual and complicated influences associated with atmospheric transport. Consequently, deriving regional fluxes in mountain regions with carbon cycle inversion of atmospheric CO2 mole fraction is sensitive to filtering of observations to those that can be represented at the transport model resolution. Using five years of CO2 mole fraction observations from the Regional Atmospheric Continuous CO2 Network in the Rocky Mountains (Rocky RACCOON), five statistical filters are used to investigate a range of approaches for identifying regionally representative CO2 mole fractions. Test results from three filters indicate that subsets based on short-term variance and local CO2 gradients across tower inlet heights retain nine-tenths of the total observations and are able to define representative diel variability and seasonal cycles even for difficult-to-model sites where the influence of local fluxes is much larger than regional mole fraction variations. Test results from two other filters that consider measurements from previous and following days using spline fitting or sliding windows are overly selective. Case study examples showed that these windowing-filters rejected measurements representing synoptic changes in CO2, which suggests that they are not well suited to filtering continental CO2 measurements. We present a novel CO2 lapse rate filter that uses CO2 differences between levels in the model atmosphere to select subsets of site measurements that are representative on model scales. Our new filtering techniques provide guidance for novel approaches to assimilating mountain-top CO2 mole fractions in carbon cycle inverse models

Agreement between methods of measurement with multiple observations per individual

Limits of agreement provide a straightforward and intuitive approach to agreement between different methods for measuring the same quantity. When pairs of observations using the two methods are independent, i.e., on different subjects, the calculations are very simple and straightforward. Some authors collect repeated data, either as repeated pairs of measurements on the same subject, whose true value of the measured quantity may be changing, or more than one measurement by one or both methods of an unchanging underlying quantity. In this paper we describe methods for analysing such clustered observations, both when the underlying quantity is assumed to be changing and when it is not

Assessing satisfaction with social care services among black and minority ethnic and white British carers of stroke survivors in England

Overall satisfaction levels with social care are usually high but lower levels have been reported among black and minority ethnic (BME) service users in England. Reasons for this are poorly understood. This qualitative study therefore explored satisfaction with services among informal carer participants from five different ethnic groups. Fifty-seven carers (black Caribbean, black African, Asian Indian, Asian Pakistani and white British) were recruited from voluntary sector organisations and a local hospital in England, and took part in semi-structured interviews using cognitive interviewing and the critical incident technique. Interviews took place from summer 2013 to spring 2014. Thematic analysis of the interviews showed that participants often struggled to identify specific ‘incidents’, especially satisfactory ones. When describing satisfactory services, participants talked mostly about specific individuals and relationships. Unsatisfactory experiences centred on services overall. When rating services using cognitive interviewing, explicit comparisons with expectations or experiences with other services were common. Highest satisfaction ratings tended to be justified by positive personal characteristics among practitioners, trust and relationships. Lower level ratings were mostly explained by inconsistency in services, insufficient or poor care. Lowest level ratings were rare. Overall, few differences between ethnic groups were identified, although white British participants rated services higher overall giving more top ratings. White British participants also frequently took a more overall view of services, highlighting some concerns but still giving top ratings, while South Asian carers in particular focused on negative aspects of services. Together these methods provide insight into what participants mean by satisfactory and unsatisfactory services. Cognitive interviewing was more challenging for some BME participants, possibly a reflection of the meaningfulness of the concept of service satisfaction to them. Future research should include comparisons between BME and white participants’ understanding of the most positive parts of satisfaction scales and should focus on dissatisfied participants