Temporal-Difference Reinforcement Learning with Distributed Representations

Temporal-difference (TD) algorithms have been proposed as models of reinforcement learning (RL). We examine two issues of distributed representation in these TD algorithms: distributed representations of belief and distributed discounting factors. Distributed representation of belief allows the believed state of the world to distribute across sets of equivalent states. Distributed exponential discounting factors produce hyperbolic discounting in the behavior of the agent itself. We examine these issues in the context of a TD RL model in which state-belief is distributed over a set of exponentially-discounting “micro-Agents”, each of which has a separate discounting factor (γ). Each µAgent maintains an independent hypothesis about the state of the world, and a separate value-estimate of taking actions within that hypothesized state. The overall agent thus instantiates a flexible representation of an evolving world-state. As with other TD models, the value-error (δ) signal within the model matches dopamine signals recorded from animals in standard conditioning reward-paradigms. The distributed representation of belief provides an explanation for the decrease in dopamine at the conditioned stimulus seen in overtrained animals, for the differences between trace and delay conditioning, and for transient bursts of dopamine seen at movement initiation. Because each µAgent also includes its own exponential discounting factor, the overall agent shows hyperbolic discounting, consistent with behavioral experiments

Social security and conflict within the family

Social security, Savings, Bargaining, H55, D13, D91,

Historical Review and Update of Surgical Treatment for Corneal Endothelial Diseases

The cornea remains in a state of deturgescence, maintained by endothelial cell Na+/K+ ATPase and by tight junctions between endothelial cells that limit entrance of fluid into the stroma. Fuchs’ endothelial corneal dystrophy (FECD) was initially described by Fuchs in 1910 as a combination of epithelial and stromal edema in older patients. It manifests as bilateral, albeit asymmetric, central corneal guttae, corneal edema, and reduced vision. When edema is severe, the corneal epithelium can detach from its basement membrane, creating painful bullae on the anterior surface of the cornea. The course of this dystrophy can be further accelerated after intraocular surgery, specifically cataract extraction. Pseudophakic bullous keratopathy (PBK) is endothelial cell loss caused by surgery in the anterior chamber. If the corneal endothelium is damaged during surgery, the same spectrum of symptoms as found in FECD can develop. In the nineteenth century, penetrating keratoplasty was the only surgical procedure available for isolated endothelial disease. In the 1960s, Dr. José Barraquer described a method of endothelial keratoplasty using an anterior approach via laser-assisted in situ keratomileusis (LASIK) flap. In 1999, Melles and colleague described their technique of posterior lamellar keratoplasty. Later, Melles et al. started to change host dissection using simple “descemetorhexis” in a procedure known as Descemet’s stripping endothelial keratoplasty. Following the widespread adoption of Descemet’s stripping automated endothelial keratoplasty, the Melles group revisited selective Descemet’s membrane transplantation and reported the results of a new procedure, Descemet’s membrane endothelial keratoplasty (DMEK). Recently, some eye banks have experimented with the preparation of DMEK/Descemet’s membrane automated endothelial keratoplasty donor tissue that may help the surgeon avoid the risk of tissue loss during the stromal separation step. Recently, the authors described a new bimanual technique for insertion and positioning of endothelium–Descemet membrane grafts in DMEK. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1007/s40123-014-0022-y) contains supplementary material, which is available to authorized users

Dysfunctional tear syndrome: a Delphi approach to treatment recommendations.

PURPOSE: To develop current treatment recommendations for dry eye disease from consensus of expert advice. METHODS: Of 25 preselected international specialists on dry eye, 17 agreed to participate in a modified, 2-round Delphi panel approach. Based on available literature and standards of care, a survey was presented to each panelist. A two-thirds majority was used for consensus building from responses obtained. Treatment algorithms were created. Treatment recommendations for different types and severity levels of dry eye disease were the main outcome. RESULTS: A new term for dry eye disease was proposed: dysfunctional tear syndrome (DTS). Treatment recommendations were based primarily on patient symptoms and signs. Available diagnostic tests were considered of secondary importance in guiding therapy. Development of algorithms was based on the presence or absence of lid margin disease and disturbances of tear distribution and clearance. Disease severity was considered the most important factor for treatment decision-making and was categorized into 4 levels. Severity was assessed on the basis of tear substitute requirements, symptoms of ocular discomfort, and visual disturbance. Clinical signs present in lids, tear film, conjunctiva, and cornea were also used for categorization of severity. Consensus was reached on treatment algorithms for DTS with and without concurrent lid disease. CONCLUSION: Panelist opinion relied on symptoms and signs (not tests) for selection of treatment strategies. Therapy is chosen to match disease severity and presence versus absence of lid margin disease or tear distribution and clearance disturbances