Analyzing Transformer Dynamics as Movement through Embedding Space

Transformer language models exhibit intelligent behaviors such as understanding natural language, recognizing patterns, acquiring knowledge, reasoning, planning, reflecting and using tools. This paper explores how their underlying mechanics give rise to intelligent behaviors. We adopt a systems approach to analyze Transformers in detail and develop a mathematical framework that frames their dynamics as movement through embedding space. This novel perspective provides a principled way of thinking about the problem and reveals important insights related to the emergence of intelligence: 1. At its core the Transformer is a Embedding Space walker, mapping intelligent behavior to trajectories in this vector space. 2. At each step of the walk, it composes context into a single composite vector whose location in Embedding Space defines the next step. 3. No learning actually occurs during decoding; in-context learning and generalization are simply the result of different contexts composing into different vectors. 4. Ultimately the knowledge, intelligence and skills exhibited by the model are embodied in the organization of vectors in Embedding Space rather than in specific neurons or layers. These abilities are properties of this organization. 5. Attention's contribution boils down to the association-bias it lends to vector composition and which influences the aforementioned organization. However, more investigation is needed to ascertain its significance. 6. The entire model is composed from two principal operations: data independent filtering and data dependent aggregation. This generalization unifies Transformers with other sequence models and across modalities. Building upon this foundation we formalize and test a semantic space theory which posits that embedding vectors represent semantic concepts and find some evidence of its validity

The use of the patient-centered medical home for children with medical complexity

Children with medical complexity (CMC) comprise a small amount of total pediatric patients but contribute to a large portion of total pediatric spending and health care utilization. As a result, there is great interest in creating innovations in their health care delivery systems to improve their quality of care and contain costs. One unique aspect of the health care of CMC is its fragmented nature. Since CMC commonly have multiple co-morbidities, they often receive care from several different specialists at any given time. The patient-centered medical home (PCMH) was previously suggested as a possible tool to improve the fragmented care of CMC. Although definitions of the medical home vary, common themes are its function to better coordinate and integrate the care of patients. Current studies suggest that many CMC do not make regular primary care visits or receive care that fulfills a majority of the components of a medical home. In addition, according to the limited studies that exist which examine CMC and other children with special health care needs (CSHCN), primary care and medical home usage amongst these populations may reduce the occurrence of preventable medical events like hospital readmissions or emergency department visits. Therefore, further research and work should be conducted to examine the feasibility and actions that must be conducted in order to increase the prevalence of these medical home programs amongst CMC

Risk-sensitive Inverse Reinforcement Learning via Semi- and Non-Parametric Methods

The literature on Inverse Reinforcement Learning (IRL) typically assumes that humans take actions in order to minimize the expected value of a cost function, i.e., that humans are risk neutral. Yet, in practice, humans are often far from being risk neutral. To fill this gap, the objective of this paper is to devise a framework for risk-sensitive IRL in order to explicitly account for a human's risk sensitivity. To this end, we propose a flexible class of models based on coherent risk measures, which allow us to capture an entire spectrum of risk preferences from risk-neutral to worst-case. We propose efficient non-parametric algorithms based on linear programming and semi-parametric algorithms based on maximum likelihood for inferring a human's underlying risk measure and cost function for a rich class of static and dynamic decision-making settings. The resulting approach is demonstrated on a simulated driving game with ten human participants. Our method is able to infer and mimic a wide range of qualitatively different driving styles from highly risk-averse to risk-neutral in a data-efficient manner. Moreover, comparisons of the Risk-Sensitive (RS) IRL approach with a risk-neutral model show that the RS-IRL framework more accurately captures observed participant behavior both qualitatively and quantitatively, especially in scenarios where catastrophic outcomes such as collisions can occur.Comment: Submitted to International Journal of Robotics Research; Revision 1: (i) Clarified minor technical points; (ii) Revised proof for Theorem 3 to hold under weaker assumptions; (iii) Added additional figures and expanded discussions to improve readabilit

Micro-albuminuria in non-diabetic acute ischaemic stroke: prevalence and its co-relation with stroke severity

Background: Microalbuminuria is not only a predictor of subsequent kidney disease, but also an indicator of generalised endothelial injury and a manifestation of endothelial dysfunction. The present study is aimed to determine the prevalence of microalbuminuria among non–diabetic ischaemic stroke patients and find its correlation with ischaemic stroke which eventually will aid us in coming up with potent strategies to provide better prevention and cure.Methods: The present study was conducted in Department of Medicine in collaboration with Department of Biochemistry and Department of Radiology, Guru Nanak Dev Hospital, Amritsar, Punjab, India after taking approval from institutional thesis and ethical committee. The study included 60 patients (30 Cases + 30 Controls) in age group 20-80 years diagnosed as stroke and haemorrhage ruled out by NCCT Brain/MRI Brain at admission. Cases were patients with history of hypertension with acute ischaemic stroke. Controls were age and sex matched patients with no history of hypertension with acute ischaemic stroke. The microalbuminuria was assayed by immunoturbimetry. The stroke severity was assessed by NIH Stroke Severity scale. P value less than 0.05 was considered the level of significance.Results: The overall prevalence of microalbuminuria in acute ischaemic stroke patients was 41.67%. When comparing NIH SS (National Institutes of Health Stroke Scale) score with the levels of albumin in urine, there was a significant positive correlation with urinary albumin levels and stroke severity in the patients having urinary albumin levels in microalbuminuria range both in Case group and Control group with P value less than 0.05.Conclusions: Urine albumin excretion had a positive correlation with the NIH SS Score of the patient in acute ischemic stroke. Those with a higher NIH SS Score had a higher rate of urine albumin excretion and vice versa. Therefore, measurement of microalbuminuria may help to assess those who are at increased risk of severe stroke and may require a more aggressive management