Performance Dynamics and Termination Errors in Reinforcement Learning: A Unifying Perspective

In reinforcement learning, a decision needs to be made at some point as to whether it is worthwhile to carry on with the learning process or to terminate it. In many such situations, stochastic elements are often present which govern the occurrence of rewards, with the sequential occurrences of positive rewards randomly interleaved with negative rewards. For most practical learners, the learning is considered useful if the number of positive rewards always exceeds the negative ones. A situation that often calls for learning termination is when the number of negative rewards exceeds the number of positive rewards. However, while this seems reasonable, the error of premature termination, whereby termination is enacted along with the conclusion of learning failure despite the positive rewards eventually far outnumber the negative ones, can be significant. In this paper, using combinatorial analysis we study the error probability in wrongly terminating a reinforcement learning activity which undermines the effectiveness of an optimal policy, and we show that the resultant error can be quite high. Whilst we demonstrate mathematically that such errors can never be eliminated, we propose some practical mechanisms that can effectively reduce such errors. Simulation experiments have been carried out, the results of which are in close agreement with our theoretical findings.Comment: Short Paper in AIKE 201

Determination of the relative favourability of sites in the Arctic by topoclimatic modelling.

Dept. of Geography. Paper copy at Leddy Library: Theses & Major Papers - Basement, West Bldg. / Call Number: Thesis1978 .L486. Source: Masters Abstracts International, Volume: 40-07, page: . Thesis (M.A.)--University of Windsor (Canada), 1978

Stochastic Reinforcement Learning

In reinforcement learning episodes, the rewards and punishments are often non-deterministic, and there are invariably stochastic elements governing the underlying situation. Such stochastic elements are often numerous and cannot be known in advance, and they have a tendency to obscure the underlying rewards and punishments patterns. Indeed, if stochastic elements were absent, the same outcome would occur every time and the learning problems involved could be greatly simplified. In addition, in most practical situations, the cost of an observation to receive either a reward or punishment can be significant, and one would wish to arrive at the correct learning conclusion by incurring minimum cost. In this paper, we present a stochastic approach to reinforcement learning which explicitly models the variability present in the learning environment and the cost of observation. Criteria and rules for learning success are quantitatively analyzed, and probabilities of exceeding the observation cost bounds are also obtained.Comment: AIKE 201

Extreme low-lying carotid bifurcations

A 52-year-old male with no past medical history was referred to the transient ischaemic attack (TIA) clinic following an event at home. The transient symptoms were of an inability to move his left arm and leg for a period of approximately 15 minutes. The patient denied any numbness of the face, headaches or blurring of vision. A careful history revealed two previous transient attacks of blurred vision approximately 1–2 years prior to this presentation. He had no other co-morbidities or associated syndromes. Given this history, suggestive of TIA(s) in the right anterior circulation, an ultrasound examination of the carotid vessels was performed to include or exclude an atherosclerotic source of embolus. The ultrasound scan demonstrated an extremely short common carotid artery (CCA) of just 2.5 cm on the right, with apparently normal flows and velocities in both the external and internal carotids (ECA and ICA) but poor views of the bifurcation. Similarly, the flows and velocities in the left ICA and ECA were also normal, with the carotid bulb lying low in the base of the neck, so further imaging with a magnetic resonance angiogram (MRA) was performed (Panel A) to clarify the anatomic and ultrasound findings. This confirmed extremely low-lying bilateral carotid bifurcations (highlighted in Panel A). The short, right-side CCA bifurcates at the level of C7/T1 (Panel B) and the left carotid bifurcation is at the level of C6/C7 (Panel C)

A randomised clinical trial of ultrasound guided cannulation of difficult fistulae for dialysis access

Background: Arteriovenous fistulae (AVF) are preferred for dialysis access but require accurate cannulation for effective dialysis. Evidence supports improvements in cannulation and complication rates using ultrasound guidance (USG) in cannulating other sites. This mixed methods, randomised controlled trial aimed to assess effects of USG during AVF cannulation.Methods: Participants with difficult to cannulate AVF had each cannulation event randomised to USG or standard technique (no USG). The primary outcome was the incidence and number of additional needle passes. Secondary outcomes included: the incidence and number of additional skin punctures; time to achieve two needle cannulation; pain associated with cannulation; local complications. Qualitative outcomes were assessed using patient and staff questionnaires.Results: 32 participants had 346 cannulation events randomised (170 to USG and 176 to standard cannulation). USG resulted in a significant reduction in additional needle passes (72 vs 99 p=0.007) and additional skin punctures (10 vs 25 p=0.016.) but prolonged time to cannulation (p>0.001). There was no difference in pain score (p=0.705) or complications between groups. Questionnaires demonstrated that USG cannulation is acceptable to patients and staff. Conclusion: USG cannulation of AVF is more accurate and no more painful than non-image guided cannulation, but prolonged time to cannulation. Some of the excess time involved may be due to the trial being performed early in cannulating staff’s learning curve with the USG technique. Further work to elucidate which patients gain most benefit from USG cannulation and the effect of USG on cannulation complications and AVF patency is warranted

Prevalence and incidence of diagnosed hypertension in Alberta, Canada

Introduction The prevalence of diagnosed hypertension in Canada is projected to increase despite the incidence rate decreasing. Previous work around the world has utilized survey data to provide estimates of prevalence and incidence. Administrative data is population-level, and may provide more reliable estimates of provincial prevalence and incidence than could be achieved using survey data.  Objectives and Approach • To produce age and sex-specific prevalence and incidence estimates of diagnosed hypertension in Alberta from 2007 to 2015, • To project estimates to the fiscal year of 2019/2020. Data from the Discharge Abstract Database, physician claims database, National Ambulatory Care Reporting System, and provincial health insurance registry will be linked using unique anonymous personal identifier and gender. A validated case definition of diagnosed hypertension for use in administrative datasets will be used to identify annual prevalent and incident cases from claims data. Obstetric cases will be excluded. The provincial health insurance registry will be used to estimate denominator values. Results Results of this analysis are not available for the time of abstract submission as the timeline for this analysis projects completion in April 2018. Conclusion/Implications Maintained surveillance of diagnosed hypertension is important to inform health policy and spending decisions, to monitor efficacy of public health interventions, and to inform patient care. Furthermore, diagnosis guidelines have been updated since 2017. Providing estimates for the prevalence of diagnosed hypertension in Alberta five years into the future to compare to actual prevalence estimates may indicate whether changes in prevalence are due to actual changes in health status or to changes in diagnosis guidelines

Invasion success of a global avian invader is explained by within-taxon niche structure and association with humans in the native range

Aim To mitigate the threat invasive species pose to ecosystem functioning, reli- able risk assessment is paramount. Spatially explicit predictions of invasion risk obtained through bioclimatic envelope models calibrated with native species distribution data can play a critical role in invasive species management. Fore- casts of invasion risk to novel environments, however, remain controversial. Here, we assess how species’ association with human-modified habitats in the native range and within-taxon niche structure shape the distribution of invasive populations at biogeographical scales and influence the reliability of predictions of invasion risk. Location Africa, Asia and Europe. Methods We use ~1200 native and invasive ring-necked parakeet (Psittacula krameri) occurrences and associated data on establishment success in combi- nation with mtDNA-based phylogeographic structure to assess niche dynam- ics during biological invasion and to generate predictions of invasion risk. Niche dynamics were quantified in a gridded environmental space while bioclimatic models were created using the biomod2 ensemble modelling framework. Results Ring-necked parakeets show considerable niche expansion into climates colder than their native range. Only when incorporating a measure of human modification of habitats within the native range do bioclimatic envelope mod- els yield credible predictions of invasion risk for parakeets across Europe. Inva- sion risk derived from models that account for differing niche requirements of phylogeographic lineages and those that do not achieve similar statistical accu- racy, but there are pronounced differences in areas predicted to be susceptible for invasion. Main conclusions Information on within-taxon niche structure and especially association with humans in the native range can substantially improve predic- tive models of invasion risk. To provide policymakers with robust predictions of invasion risk, including these factors into bioclimatic envelope models is recommended

SNP rs1533428 at 2p16.3 as a marker for late-onset primary open-angle glaucoma

Purpose: To investigate the associations between gene variants in cholesterol 24S-hydroxylase (CYP46A1), LIM homeobox transcription factor 1-beta (LMX1B), plexin domain containing 2 (PLXDC2), toll-like receptor 4 (TLR4), transmembrane and tetratricopeptide repeat containing 2 (TMTC2), zona pellucida glycoprotein 4 (ZP4), chromosome 2p16.3, and primary open-angle glaucoma (POAG). Methods: We studied 462 POAG patients and 577 controls from three cohorts (Hong Kong, Shantou, and Beijing, China). Twelve single-nucleotide polymorphisms (SNPs) were genotyped in the Hong Kong cohort using TaqMan genotyping assay. Significant associations were validated in the Shantou and Beijing cohorts. Results: Association of POAG with TLR4 rs7037117, in a recessive model, was identified in the Hong Kong and Shantou cohorts (both southern Chinese, $p_{rec}$=0.0019) but not the Beijing cohort (northern Chinese). rs1533428 at chromosome 2p16.3 showed a consistent trend of age-specific association in all three cohorts. Genotypes TT + CT conferred a 2.16 fold of significantly increased risk to late-onset POAG ($p_{dom}$=0.00025), but no significant risk to POAG of younger ages of onset in the combined cohort. A joint effect was found between rs7037117 and rs1533428, with carriers of both higher-risk genotypes having a 4.53 fold of increased disease risk (p=0.00028). Conclusions: Our study reveals discrepant association patterns of 12 candidate SNPs in 7 genes/loci with POAG in Chinese, provides positive replications for POAG markers rs1533428 at 2p16.3 and TLR4 rs7037117, and suggests that rs1533428 is a putative risk variant for late-onset POAG. The identification of an age-specific association between rs1533428 and late-onset POAG highlights a new genotype-phenotype association in POAG. Further studies are warranted to confirm the age-specific association