Influencing Exploration in Actor-Critic Reinforcement Learning Algorithms

Reinforcement Learning (RL) is a subset of machine learning primarily concerned with goal-directed learning and optimal decision making. RL agents learn based on a reward signal discovered from trial and error in complex, uncertain environments with the goal of maximizing positive reward signals. RL approaches need to scale up as they are applied to more complex environments with extremely large state spaces. Inefficient exploration methods cannot sufficiently explore complex environments in a reasonable amount of time, and optimal policies will be unrealized resulting in RL agents failing to solve an environment. This thesis proposes a novel variant of the Actor-Advantage Critic (A2C) algorithm. The variant is validated against two state-of-the-art RL algorithms, Deep Q-Network (DQN) and A2C, across six Atari 2600 games of varying difficulty. The experimental results are competitive with state-of-the-art and achieve lower variance and quicker learning speed. Additionally, the thesis introduces a metric to objectively quantify the difficulty of any Markovian environment with respect to the exploratory capacity of RL agents

A novel culture system for modulating single cell geometry in 3D

Dedifferentiation of chondrocytes during in vitro expansion remains an unsolved challenge for repairing serious articular cartilage defects. In this study, a novel culture system was developed to modulate single cell geometry in 3D and investigate its effects on the chondrocyte phenotype. The approach uses 2D micropatterns followed by in situ hydrogel formation to constrain single cell shape and spreading. This enables independent control of cell geometry and extracellular matrix. Using collagen I matrix, we demonstrated the formation of a biomimetic collagenous “basket” enveloping individual chondrocytes cells. By quantitatively monitoring the production by single cells of chondrogenic matrix (e.g. collagen II and aggrecan) during 21-day cultures, we found that if the cell’s volume decreases, then so does its cell resistance to dedifferentiation (even if the cells remain spherical). Conversely, if the volume of spherical cells remains constant (after an initial decrease), then not only do the cells retain their differentiated status, but previously de-differentiated redifferentiate and regain a chondrocyte phenotype. The approach described here can be readily applied to pluripotent cells, offering a versatile platform in the search for niches toward either self-renewal or targeted differentiation

Getting back to basics : bimanual interaction on mobile touch screen devices

The availability, and popularity, of touch screen tablets is drastically increasing with over 30% of internet users now owning one. However the lack of bimanual interaction in touch screen tablets is presenting product designers with serious challenges. Several attempts have been made to facilitate bimanual interaction in such products but results are not comparable to that of their non-mobile cousins, e.g. laptops. This paper presents the finding of a group collaboration aimed at prototyping a mobile touch screen device which supports bimanual interaction during internet browser navigation through rear mounted inputs. The researchers found it problematic to add basic bimanual interactions for internet browser navigation to the rear of a prototype mobile touch screen device due to issues regarding grip type, finger movement and hand position. This paper concludes that in order to achieve bimanual interaction researchers need to return to basics and consider how to free the hand and fingers from current constraints

Physical interaction mappings : utilizing cognitive load theory in order to enhance physical product interaction

Learning to physically operate traditional products can be viewed as a learning process akin to any other. As such, many of today's products, such as cars, boats, and planes, which have traditional controls that predate modern user-centered design techniques may be imposing irrelevant or unrelated cognitive loads on their operators. The availability of working memory has long been identified as a critical aspect of an instructional design. Many conventional instructional procedures impose irrelevant or unrelated cognitive loads on the learner due to the fact that they were created without contemplation, or understanding, of cognitive work load. The goal of the research was to investigate the fundamental relationships between physical inputs, resulting actions, and learnability. The results showed that individuals can quickly adapt to input/output reversals across dimensions, however, individuals struggle to cope with the input/output when the dimensions are rotated due to the resulting increase in cognitive load

Questioning conventions : are product conventions trading off the usability of products for short term user satisfaction

Mapping conventions are a key aspect of user centered design as they present users with familiar interactions in unfamiliar products. Conventions evolve over time and are slow to be adopted, requiring a high percentage of acceptance within a society, ensuring that conventions exhibit a sufficient level of usability. However this paper argues that while usability is a necessary condition for good interactions it is not a sufficient one. Therefore user centered design which accents individuals bias towards conventions my in fact be hindering the innovation of product interactions. This paper argues that a cognitive approach should be adopted in order understand and reassess product interactions. An experiment was carried out that demonstrates the influence that simple mappings can have on cognitive load. The results showed that basic mappings of the types that are found throughout product conventions can have a substantial impact on mental load and subsequently product interaction

Earth–Air Heat Exchanger Geo-Climatic Suitability for Projected Climate Change Scenarios in the Americas

Earth–air heat exchangers (EAHE) provide heating and cooling that is intrinsically tied to the climate of the surrounding environment. A climate-based approach was applied to 273 sites for both historical and projected climate conditions, with the latter being defined by three different Representative Concentration Pathways (RCPs) from the CMIP5 collection of Global Circulation Models (GCMs). Changes to heating and cooling degree hours as well as heating and cooling capacity were estimated and used to classify geo-climatic suitability. The analysis revealed cooler climates will retain their ability to provide cooling despite increasing cooling needs driven by warming temperatures. On the other hand, warmer, more tropical, climates will observe reduced suitability as cooling demand grows. The magnitude and variability of the changes in EAHE potential were greatest for the RCP8.5 scenario during the 2061–2090 time period, particularly for regions with a comparable mix of heating and cooling needs. Ultimately, the results demonstrate that future EAHE suitability is climate dependent, with cooler climates being relatively resistant to changes when compared to warmer climates. The results can be used by stakeholders to find useful climate analogs for their sites of interest to consider the potential impact of global climate change on EAHE usability

GALA: an international multicentre randomised trial comparing general anaesthesia versus local anaesthesia for carotid surgery

Background: Patients who have severe narrowing at or near the origin of the internal carotid artery as a result of atherosclerosis have a high risk of ischaemic stroke ipsilateral to the arterial lesion. Previous trials have shown that carotid endarterectomy improves long-term outcomes, particularly when performed soon after a prior transient ischaemic attack or mild ischaemic stroke. However, complications may occur during or soon after surgery, the most serious of which is stroke, which can be fatal. It has been suggested that performing the operation under local anaesthesia, rather than general anaesthesia, may be safer. Therefore, a prospective, randomised trial of local versus general anaesthesia for carotid endarterectomy was proposed to determine whether type of anaesthesia influences peri-operative morbidity and mortality, quality of life and longer term outcome in terms of stroke-free survival. Methods/design: A two-arm, parallel group, multicentre randomised controlled trial with a recruitment target of 5000 patients. For entry into the study, in the opinion of the responsible clinician, the patient requiring an endarterectomy must be suitable for either local or general anaesthesia, and have no clear indication for either type. All patients with symptomatic or asymptomatic internal carotid stenosis for whom open surgery is advised are eligible. There is no upper age limit. Exclusion criteria are: no informed consent; definite preference for local or general anaesthetic by the clinician or patient; patient unlikely to be able to co-operate with awake testing during local anaesthesia; patient requiring simultaneous bilateral carotid endarterectomy; carotid endarterectomy combined with another operation such as coronary bypass surgery; and, the patient has been randomised into the trial previously. Patients are randomised to local or general anaesthesia by the central trial office. The primary outcome is the proportion of patients alive, stroke free ( including retinal infarction) and without myocardial infarction 30 days post-surgery. Secondary outcomes include the proportion of patients alive and stroke free at one year; health related quality of life at 30 days; surgical adverse events, re-operation and re-admission rates; the relative cost of the two methods of anaesthesia; length of stay and intensive and high dependency bed occupancy

Generating Stable Bioluminescent Cell Lines by Retroviral Transduction

This project focuses on improving tumor in vivo imaging techniques. It is difficult to accurately in vivo image orthotopic tumors using CT scans alone. Muriglo provides the capability to visualize in vivo bioluminescent signals emanating from intact tissues. We can coregister CT scans captured with the SARRP with luciferase signals captured with the Muriglo and locate tumor isocenters for image-guided radiation. We may then direct radiation to neoplastic tissue while also sparing the maximum amount of healthy tissue. To do this we must genetically engineer cells to express luciferase. We propose a protocol to use a retrovirus to create stable cell lines that express firefly luciferase for bioluminescence. We will also determine the optimal dosage of the G418 antibiotic for selection

Adiabatic elimination in quantum stochastic models

We consider a physical system with a coupling to bosonic reservoirs via a quantum stochastic differential equation. We study the limit of this model as the coupling strength tends to infinity. We show that in this limit the solution to the quantum stochastic differential equation converges strongly to the solution of a limit quantum stochastic differential equation. In the limiting dynamics the excited states are removed and the ground states couple directly to the reservoirs.Comment: 17 pages, no figures, corrected mistake