The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute

The Languini Kitchen serves as both a research collective and codebase designed to empower researchers with limited computational resources to contribute meaningfully to the field of language modelling. We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. The number of tokens on which a model is trained is defined by the model's throughput and the chosen compute class. Notably, this approach avoids constraints on critical hyperparameters which affect total parameters or floating-point operations. For evaluation, we pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. On it, we compare methods based on their empirical scaling trends which are estimated through experiments at various levels of compute. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput. While the GPT baseline achieves better perplexity throughout all our levels of compute, our LSTM baseline exhibits a predictable and more favourable scaling law. This is due to the improved throughput and the need for fewer training tokens to achieve the same decrease in test perplexity. Extrapolating the scaling laws leads of both models results in an intersection at roughly 50,000 accelerator hours. We hope this work can serve as the foundation for meaningful and reproducible language modelling research

COMBINATORIAL EXPLORATION OF HALF-HEUSLER (Ta0.4 Nb0.4 Ti0.2)–Fe–Sb THIN FILMS VIA HIGH-THROUGHPUT POWER FACTOR MAPPING AND FREQUENCY-DOMAIN THERMOREFLECTANCE (FDTR)

Thermoelectrics (TEs) are a class of materials capabl¬¬e of converting heat into electricity in the solid state. Their widespread application is limited by the low efficiency (≈ 5 %) for commercial modules in applications such as waste heat recovery and refrigeration. Half-Heusler (hH) TE intermetallic alloys have good electrical properties that are easily tuned by doping but are limited in commercial deployment due to high thermal conductivity (TC). This limits the achievable thermal gradient across a TE module, reducing the efficiency. One method to improve hH alloy performance is to decrease the lattice contribution to the TC through solid-solution alloying. Combinatorial synthesis approaches have the advantage of rapid sample fabrication and characterization over a wide range of material compositions. This approach can provide insights into materials systems that could be missed using conventional synthesis approaches. Several publications reported p-type NbFeSb hH alloys can accommodate off-stoichiometry, which could positively impact the TE properties similar to TC decrease observed via Ta-alloying. Combinatorial thin film co-sputter synthesis of hH alloy (Ta0.40Nb0.40Ti0.20)-Fe-Sb composition spread libraries coupled with high throughput (HiTp) characterization is utilized to produce maps of the composition-structure-property relationships as a function of Fe- and Sb-content in this system for the first time. Continuous spread composition gradient and homogeneous discrete co-sputtered combinatorial thin film synthesis methodologies are leveraged to investigate the hH stability region and TE performance in (Ta0.40Nb0.40Ti0.20)-Fe-Sb. Combinatorial thin film characterization requires specialized custom or commercial instrumentation capable of scanning across samples. Established HiTp tools were utilized to characterize the crystal structure, electrical transport properties (Seebeck coefficient and electrical resistivity), and chemical composition of the films. A scanning thin film TC measurement instrument was not available prior to this dissertation. Without this ability, the dimensionless TE figure-of-merit zT cannot be calculated. To address this need, a custom, automated Frequency Domain Thermoreflectance (FDTR) instrument was designed and constructed. FDTR TC measurements are presented on single-phase F¯4 3m off stoichiometric discrete combinatorial hH (Ta0.40Nb0.40Ti0.20)-Fe-Sb for the first time. Maximum zT values at 296 K are calculated to be 0.076 for compositions (Nb0.412Ta0.327Ti0.261)28.5Fe40.3Sb31.2 and (Nb0.418Ta0.328Ti0.254)35.0Fe31.7Sb33.3 having TC values around 2.25 ± 0.27 W m-1 K-1

Recent Perceived Stress, Amygdala Reactivity to Acute Psychosocial Stress, and Alcohol and Cannabis Use in Adolescents and Young Adults With Bipolar Disorder

Background: Psychosocial stress negatively affects the clinical course of bipolar disorder. Studies primarily focused on adults with bipolar disorder suggest the impact of stress is progressive, i.e., stress response sensitizes with age. Neural mechanisms underlying stress sensitization are unknown. As stress-related mechanisms contribute to alcohol/substance use disorders, variation in stress response in youth with bipolar disorder may contribute to development of co-occurring alcohol/substance use disorders. This study investigated relations between psychosocial stress, amygdala reactivity, and alcohol and cannabis use in youth with bipolar disorder, compared to typically developing youth. Methods: Forty-two adolescents/young adults [19 with bipolar disorder, 23 typically developing, 71% female, agemean ± SD = 21 ± 2 years] completed the Perceived Stress Scale (PSS), Daily Drinking Questionnaire modified for heaviest drinking week, and a modified Montreal Imaging Stress functional MRI Task. Amygdala activation was measured for both the control and stress conditions. Main effects of group, condition, total PSS, and their interactions on amygdala activation were modeled. Relationships between amygdala response to acute stress with recent alcohol/cannabis use were investigated. Results: Greater perceived stress related to increased right amygdala activation in response to the stress, compared to control, condition in bipolar disorder, but not in typically developing youth (group × condition × PSS interaction, p = 0.02). Greater amygdala reactivity to acute stress correlated with greater quantity and frequency of alcohol use and frequency of cannabis use in bipolar disorder. Conclusion: Recent perceived stress is associated with changes in amygdala activation during acute stress with amygdala reactivity related to alcohol/cannabis use in youth with bipolar disorder

Alcohol Use and Prefrontal Cortex Volume Trajectories in Young Adults with Mood Disorders and Associated Clinical Outcomes

(1) Background: Alcohol use in the course of mood disorders is associated with worse clinical outcomes. The mechanisms by which alcohol use alters the course of illness are unclear but may relate to prefrontal cortical (PFC) sensitivity to alcohol. We investigated associations between alcohol use and PFC structural trajectories in young adults with a mood disorder compared to typically developing peers. (2) Methods: 41 young adults (24 with a mood disorder, agemean = 21 ± 2 years) completed clinical evaluations, assessment of alcohol use, and two structural MRI scans approximately one year apart. Freesurfer was used to segment PFC regions of interest (ROIs) (anterior cingulate, orbitofrontal cortex, and frontal pole). Effects of group, alcohol use, time, and interactions among these variables on PFC ROIs at baseline and follow-up were modeled. Associations were examined between alcohol use and longitudinal changes in PFC ROIs with prospective mood. (3) Results: Greater alcohol use was prospectively associated with decreased frontal pole volume in participants with a mood disorder, but not typically developing comparison participants (time-by-group-by-alcohol interaction; p = 0.007); however, this interaction became a statistical trend in a sensitivity analysis excluding one outlier in terms of alcohol use. Greater alcohol use and a decrease in frontal pole volume related to longer duration of major depression during follow-up (p’s < 0.05). (4) Conclusion: Preliminary findings support more research on alcohol use, PFC trajectories, and depression recurrence in young adults with a mood disorder including individuals with heavier drinking patterns

Neural Functional Connectivity Changes to Psychosocial Stress in Young Adults with Bipolar Disorder and Preliminary Associations with Clinical Trajectories

Background: Stress-related mechanisms are implicated in the pathophysiology of bipolar disorder and may contribute to heterogeneity in illness course. Yet, there is a lack of study investigating the neural mechanisms underlying the stress response in this condition. This study investigated changes in amygdala activation and functional connectivity in response to acute psychosocial stress in young adults with bipolar disorder and explored relations with clinical phenotype and prospective mood symptoms. Methods: 42 young adults [19 with bipolar disorder, agemean ± SD =21.4 ± 2.2 years] completed a modified version of the Montreal Imaging Stress Task. Amygdala activation and functional connectivity with prefrontal cortex (PFC) regions of interest was calculated for control and stress conditions. Main effects of group, condition, and group by condition interaction on amygdala activation and connectivity were modeled. A subset of bipolar participants completed 1-year follow-up assessments. Relations between neural responses to stress with concurrent substance use and prospective mood symptoms were explored. Results: There were no between-group differences in amygdala activation or functional connectivity during the control condition. Increased right amygdala-right rostral PFC (rPFC) functional connectivity to stress was observed in bipolar disorder, compared to typically developing controls. In bipolar disorder, greater increase in right amygdala-right rPFC functional connectivity to stress was associated with less frequent cannabis use, and prospectively with shorter duration and lower severity of depression symptoms over follow-up. Conclusion: Results from this preliminary study suggest differences in frontolimbic functional connectivity responses to stress in young adults with bipolar disorder and associations with cannabis use and prospective mood symptoms

Diversity-Carbon Flux Relationships in a Northwest Forest

While aboveground biomass and forest productivity can vary over abiotic gradients (e.g., temperature and moisture gradients), biotic factors such as biodiversity and tree species stand dominance can also strongly influence biomass accumulation. In this study we use a permanent plot network to assess variability in aboveground carbon (C) flux in forest tree annual aboveground biomass increment (ABI), tree aboveground net primary productivity (ANPPtree), and net soil CO2 efflux in relation to diversity of coniferous, deciduous, and a nitrogen (N)-fixing tree species (Alnus rubra). Four major findings arose: (1) overstory species richness and indices of diversity explained between one third and half of all variation in measured aboveground C flux, and diversity indices were the most robust models predicting measured aboveground C flux; (2) trends suggested decreases in annual tree biomass increment C with increasing stand dominance for four of the five most abundant tree species; (3) the presence of an N-fixing tree species (A. rubra) was not related to changes in aboveground C flux, was negatively related to soil CO2 efflux, and showed only a weak negative relationship with aboveground C pools; and (4) stands with higher overstory richness and diversity typically had higher soil CO2 efflux. Interestingly, presence of the N-fixing species was not correlated with soil inorganic N pools, and inorganic N pools were not correlated with any C flux or pool measure. We also did not detect any strong patterns between forest tree diversity and C pools, suggesting potential balancing of increased C flux both into and out of diverse forest stands. These data highlight variability in second-growth forests that may have implications for overstory community drivers of C dynamics

Childhood maltreatment, prefrontal-paralimbic gray matter volume, and substance use in young adults and interactions with risk for bipolar disorder

Childhood maltreatment is associated with adverse effects on the brain, and an increased risk for psychopathology, including mood and substance use disorders. Individuals vary on the degree to which they exhibit neurobiological and clinical differences following maltreatment. Individuals with bipolar disorder exhibit greater magnitude of maltreatment-related prefrontal-paralimbic gray matter volume (GMV) deficits compared to typically developing individuals. It is unclear if greater structural differences stem from greater neural vulnerability to maltreatment in bipolar disorder, or if they relate to presence of other clinical features associated with childhood maltreatment, e.g., elevated prevalence of comorbid substance use disorders. To investigate this, we compared young adults with a family history of bipolar disorder (n = 21), but who did not fulfill diagnostic criteria for bipolar disorder, with typically developing young adults without a family history of bipolar disorder (n = 26). Participants completed structural neuroimaging, clinical and family history interviews, and assessment of childhood maltreatment and recent alcohol and cannabis use patterns. We examined relations between childhood maltreatment and prefrontal-paralimbic GMV by modeling main effects of maltreatment and family history group by maltreatment interactions on prefrontal-paralimbic GMV. We also examined relations between maltreatment and associated GMV changes with recent alcohol and cannabis use. Childhood maltreatment correlated with lower ventral, rostral and dorsolateral prefrontal and insular cortical GMV across all participants regardless of the presence or absence of familial history of bipolar disorder. However, exploratory analyses did reveal greater maltreatment-related GMV differences in individuals with prodromal symptoms of depression. Lower insula GMV was associated with greater frequency of cannabis use across all participants and greater quantity of alcohol use only in those with familial risk for bipolar disorder. Results suggest familial risk for bipolar disorder, and presumably genetic risk, may relate to outcomes following childhood maltreatment and should be considered in prevention/early intervention strategies

Stimulation of the right entorhinal white matter enhances visual memory encoding in humans.

BackgroundWhile deep brain stimulation has been successful in treating movement disorders, such as in Parkinson's disease, its potential application in alleviating memory disorders is inconclusive.Objective/hypothesisWe investigated the role of the location of the stimulating electrode on memory improvement and hypothesized that entorhinal white versus gray matter stimulation would have differential effects on memory.MethodsIntracranial electrical stimulation was applied to the entorhinal area of twenty-two participants with already implanted electrodes as they completed visual memory tasks.ResultsWe found that stimulation of right entorhinal white matter during learning had a beneficial effect on subsequent memory, while stimulation of adjacent gray matter or left-sided stimulation was ineffective. This finding was consistent across three different visually guided memory tasks.ConclusionsOur results highlight the importance of precise stimulation site on modulation of human hippocampal-dependent memory and suggest that stimulation of afferent input into the right hippocampus may be an especially promising target for enhancement of visual memory