Small-scale proxies for large-scale Transformer training instabilities

Teams that have trained large Transformer-based models have reported training instabilities at large scale that did not appear when training with the same hyperparameters at smaller scales. Although the causes of such instabilities are of scientific interest, the amount of resources required to reproduce them has made investigation difficult. In this work, we seek ways to reproduce and study training stability and instability at smaller scales. First, we focus on two sources of training instability described in previous work: the growth of logits in attention layers (Dehghani et al., 2023) and divergence of the output logits from the log probabilities (Chowdhery et al., 2022). By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates, and that mitigations previously employed at large scales are equally effective in this regime. This prompts us to investigate the extent to which other known optimizer and model interventions influence the sensitivity of the final loss to changes in the learning rate. To this end, we study methods such as warm-up, weight decay, and the $\mu$Param (Yang et al., 2022), and combine techniques to train small models that achieve similar losses across orders of magnitude of learning rate variation. Finally, to conclude our exploration we study two cases where instabilities can be predicted before they emerge by examining the scaling behavior of model activation and gradient norms

Body size modifies the relationship between maternal serum 25-hydroxyvitamin D concentrations and gestational diabetes in high-risk women

Obesity increases the risk of low 25-hydroxyvitamin D (25(OH) D) concentrations and gestational diabetes (GDM). We explored whether the association between GDM and change in 25(OH) D concentrations measured in the first (7-18 wk) and second (20-27 wk) trimesters of pregnancy is dependent on maternal BMI. The study was a prospective study of 219 women with BMI of >= 30 kg/m2, a history of GDM, or both. The participants were stratified by first-trimester BMI: BMI of = 35 kg/m(2). In the BMI group >= 35 kg/m(2), those who did not develop GDM during the follow-up showed higher increase in serum 25(OH) D concentrations compared with women who developed GDM (43.2 vs. 11.5%; P <0.001). No associations between 25(OH) D concentrations and GDM were observed in other BMI groups. These findings give an important aspect of the role of maternal body size in the association between vitamin D and GDM in high-risk women.Peer reviewe

Common ADRB2 Haplotypes Derived from 26 Polymorphic Sites Direct β2-Adrenergic Receptor Expression and Regulation Phenotypes

The beta2-adrenergic receptor (beta2AR) is expressed on numerous cell-types including airway smooth muscle cells and cardiomyocytes. Drugs (agonists or antagonists) acting at these receptors for treatment of asthma, chronic obstructive pulmonary disease, and heart failure show substantial interindividual variability in response. The ADRB2 gene is polymorphic in noncoding and coding regions, but virtually all ADRB2 association studies have utilized the two common nonsynonymous coding SNPs, often reaching discrepant conclusions.We constructed the 8 common ADRB2 haplotypes derived from 26 polymorphisms in the promoter, 5'UTR, coding, and 3'UTR of the intronless ADRB2 gene. These were cloned into an expression construct lacking a vector-based promoter, so that beta2AR expression was driven by its promoter, and steady state expression could be modified by polymorphisms throughout ADRB2 within a haplotype. "Whole-gene" transfections were performed with COS-7 cells and revealed 4 haplotypes with increased cell surface beta2AR protein expression compared to the others. Agonist-promoted downregulation of beta2AR protein expression was also haplotype-dependent, and was found to be increased for 2 haplotypes. A phylogenetic tree of the haplotypes was derived and annotated by cellular phenotypes, revealing a pattern potentially driven by expression.Thus for obstructive lung disease, the initial bronchodilator response from intermittent administration of beta-agonist may be influenced by certain beta2AR haplotypes (expression phenotypes), while other haplotypes may influence tachyphylaxis during the response to chronic therapy (downregulation phenotypes). An ideal clinical outcome of high expression and less downregulation was found for two haplotypes. Haplotypes may also affect heart failure antagonist therapy, where beta2AR increase inotropy and are anti-apoptotic. The haplotype-specific expression and regulation phenotypes found in this transfection-based system suggest that the density of genetic information in the form of these haplotypes, or haplotype-clusters with similar phenotypes can potentially provide greater discrimination of phenotype in human disease and pharmacogenomic association studies

Comparing pregnancy, childbirth, and neonatal outcomes in women with different phenotypes of polycystic ovary syndrome and healthy women: a prospective cohort study

The aim of this study was to compare pregnancy, childbirth, and neonatal outcomes in women with different phenotypes of polycystic ovary syndrome (PCOS) with healthy women. A prospective cohort study from the beginning to the end of pregnancy for 41 pregnant women with PCOS (case) and 49 healthy pregnant women (control) was completed. Based on the presence or absence of menstrual dysfunction (M), hyperandrogenism (HA), and polycystic ovaries (PCO) on ultrasound, the PCOS (case) group were divided into three phenotypes (HA + PCO (  = 22), M + PCO (  = 9), HA + M+PCO (  = 10). Pre-eclampsia, gestational diabetes, and lower birth weight among newborns were significantly higher in the PCOS case group compared to the control group especially in the phenotype HA + M+PCO (  < .05). High BMI (  = 2.40; =.03) was the strongest predictor of pre-eclampsia in patients with PCOS. High androgen levels (free androgen index) (  = 13.71, 3.02;  < .05), was the strongest predictor of developing diabetes during pregnancy and reduced birth weight baby, respectively.These results suggest that PCOS, particularly in phenotype HA + M+PCO (  < .05), is a risk factor for adverse pregnancy and neonatal outcomes including gestational diabetes, pre-eclampsia, and reduced weight babies

Evaluation of coronary blood flow velocity during cardiac arrest with circulation maintained through mechanical chest compressions in a porcine model

<p>Abstract</p> <p>Background</p> <p>Mechanical chest compressions (CCs) have been shown capable of maintaining circulation in humans suffering cardiac arrest for extensive periods of time. Reports have documented a visually normalized coronary blood flow during angiography in such cases (TIMI III flow), but it has never been actually measured. Only indirect measurements of the coronary circulation during cardiac arrest with on-going mechanical CCs have been performed previously through measurement of the coronary perfusion pressure (CPP). In this study our aim was to correlate average peak coronary flow velocity (APV) to CPP during mechanical CCs.</p> <p>Methods</p> <p>In a closed chest porcine model, cardiac arrest was established through electrically induced ventricular fibrillation (VF) in eleven pigs. After one minute, mechanical chest compressions were initiated and then maintained for 10 minutes upon which the pigs were defibrillated. Measurements of coronary blood flow in the left anterior descending artery were made at baseline and during VF with a catheter based Doppler flow fire measuring APV. Furthermore measurements of central (thoracic) venous and arterial pressures were also made in order to calculate the theoretical CPP.</p> <p>Results</p> <p>Average peak coronary flow velocity was significantly higher compared to baseline during mechanical chests compressions and this was observed during the entire period of mechanical chest compressions (12 - 39% above baseline). The APV slowly declined during the 10 min period of mechanical chest compressions, but was still higher than baseline at the end of mechanical chest compressions. CPP was simultaneously maintained at > 20 mmHg during the 10 minute episode of cardiac arrest.</p> <p>Conclusion</p> <p>Our study showed good correlation between CPP and APV which was highly significant, during cardiac arrest with on-going mechanical CCs in a closed chest porcine model. In addition APV was even higher during mechanical CCs compared to baseline. Mechanical CCs can, at minimum, re-establish coronary blood flow in non-diseased coronary arteries during cardiac arrest.</p

Relationships of low serum vitamin D3 with anthropometry and markers of the metabolic syndrome and diabetes in overweight and obesity

Low serum 25 hydroxyvitamin D3 (vitamin D3) is known to perturb cellular function in many tissues, including the endocrine pancreas, which are involved in obesity and type II diabetes mellitus (TIIDM). Vitamin D3 insufficiency has been linked to obesity, whether obesity is assessed by body mass index (BMI) or waist circumference (waist). Central obesity, using waist as the surrogate, is associated with the metabolic syndrome (MetSyn), insulin resistance, TIIDM and atherosclerotic cardiovascular disease (CVD). We tested how vitamin D3 was related to measures of fat mass, MetSyn markers, haemoglobin A1c (HbA1c) and MetSyn in a cross-sectional sample of 250 overweight and obese adults of different ethnicities. There were modest inverse associations of vitamin D3 with body weight (weight) (r = -0.21, p = 0.0009), BMI (r = -0.18, p = 0.005), waist (r = -0.14, p = 0.03), [but not body fat % (r = -0.08, p = 0.24)], and HbA1c (r = -0.16, p = 0.01). Multivariable regression carried out separately for BMI and waist showed a decrease of 0.74 nmol/L (p = 0.002) in vitamin D3 per 1 kg/m2 increase in BMI and a decrease of 0.29 nmol/L (p = 0.01) per 1 cm increase in waist, with each explaining approximately 3% of the variation in vitamin D3 over and above gender, age, ethnicity and season

Solar ultraviolet-B radiation and vitamin D: a cross-sectional population-based study using data from the 2007 to 2009 Canadian Health Measures Survey

Background: Exposure to solar ultraviolet-B (UV-B) radiation is a major source of vitamin D3. Chemistry climate models project decreases in ground-level solar erythemal UV over the current century. It is unclear what impact this will have on vitamin D status at the population level. The purpose of this study was to measure the association between ground-level solar UV-B and serum concentrations of 25-hydroxyvitamin D (25(OH)D) using a secondary analysis of the 2007 to 2009 Canadian Health Measures Survey (CHMS). Methods: Blood samples collected from individuals aged 12 to 79 years sampled across Canada were analyzed for 25(OH)D (n=4,398). Solar UV-B irradiance was calculated for the 15 CHMS collection sites using the Tropospheric Ultraviolet and Visible Radiation Model. Multivariable linear regression was used to evaluate the association between 25(OH)D and solar UV-B adjusted for other predictors and to explore effect modification. Results: Cumulative solar UV-B irradiance averaged over 91 days (91-day UV-B) prior to blood draw correlated significantly with 25(OH)D. Independent of other predictors, a 1 kJ/m 2 increase in 91-day UV-B was associated with a significant 0.5 nmol/L (95% CI 0.3-0.8) increase in mean 25(OH)D (P =0.0001). The relationship was stronger among younger individuals and those spending more time outdoors. Based on current projections of decreases in ground-level solar UV-B, we predict less than a 1 nmol/L decrease in mean 25(OH)D for the population. Conclusions: In Canada, cumulative exposure to ambient solar UV-B has a small but significant association with 25(OH)D concentrations. Public health messages to improve vitamin D status should target safe sun exposure with sunscreen use, and also enhanced dietary and supplemental intake and maintenance of a healthy body weight

Determining Vitamin D Status: A Comparison between Commercially Available Assays

Background: Vitamin D is not only important for bone health but can also affect the development of several non-bone diseases. The definition of vitamin D insufficiency by serum levels of 25-hydroxyvitamin D depends on the clinical outcome but might also be a consequence of analytical methods used for the definition. Although numerous 25-hydroxyvitamin D assays are available, their comparability is uncertain. We therefore aim to investigate the precision, accuracy and clinical consequences of differences in performance between three common commercially available assays. Methodology/Principal Findings: Serum 25-hydroxyvitamin D levels from 204 twins from the Swedish Twin Registry were determined with high-pressure liquid chromatography-atmospheric pressure chemical ionization-mass spectrometry (HPLCAPCI-MS), a radioimmunoassay (RIA) and a chemiluminescent immunoassay (CLIA). High inter-assay disagreement was found. Mean 25-hydroxyvitamin D levels were highest for the HPLC-APCI-MS technique (85 nmol/L, 95% CI 81-89), intermediate for RIA (70 nmol/L, 95% CI 66-74) and lowest with CLIA (60 nmol/L, 95% CI 56-64). Using a 50-nmol/L cut-off, 8% of the subjects were insufficient using HPLC-APCI-MS, 22% with RIA and 43% by CLIA. Because of the heritable component of 25-hydroxyvitamin D status, the accuracy of each method could indirectly be assessed by comparison of within-twin pair correlations. The strongest correlation was found for HPLC-APCI-MS (r = 0.7), intermediate for RIA (r = 0.5) and lowest for CLIA (r = 0.4). Regression analyses between the methods revealed a non-uniform variance (p&lt;0.0001) depending on level of 25-hydroxyvitamin D. Conclusions/Significance: There are substantial inter-assay differences in performance. The most valid method was HPLCAPCI-MS. Calibration between 25-hydroxyvitamin D assays is intricate

Chronic Obstructive Pulmonary Disease Is Associated with Low Levels of Vitamin D

Introduction: COPD patients may be at increased risk for vitamin D (25(OH)D) deficiency, but risk factors for deficiency among COPD patients have not been extensively reported. Methods: Serum 25(OH)D levels were measured by liquid chromatography double mass spectrometry in subjects aged 40–76 years from Western Norway, including 433 COPD patients (GOLD stage II-IV) and 325 controls. Levels <20 ng/mL defined deficiency. Season, sex, age, body mass index (BMI), smoking, GOLD stage, exacerbation frequency, arterial oxygen tension (PaO2), respiratory symptoms, depression (CES-D score≥16), comorbidities (Charlson score), treatment for osteoporosis, use of inhaled steroids, and total white blood count were examined for associations with 25(OH)D in both linear and logistic regression models. Results: COPD patients had an increased risk for vitamin D deficiency compared to controls after adjustment for seasonality, age, smoking and BMI. Variables associated with lower 25(OH)D levels in COPD patients were obesity ( = −6.63), current smoking ( = −4.02), GOLD stage III- IV ( = −4.71, = −5.64), and depression ( = −3.29). Summertime decreased the risk of vitamin D deficiency (OR = 0.22). Conclusion: COPD was associated with an increased risk of vitamin D deficiency, and important disease characteristics were significantly related to 25(OH)D levels

High prevalence of vitamin D insufficiency and its association with obesity and metabolic syndrome among Malay adults in Kuala Lumpur, Malaysia

Background: Vitamin D status, as indicated by 25-hydroxyvitamin D is inversely associated with adiposity, glucose homeostasis, lipid profiles, and blood pressure along with its classic role in calcium homeostasis and bone metabolism. It is also shown to be inversely associated with metabolic syndrome and cardiovascular diseases in western populations. However, evidence from the Asian population is limited. Therefore, we aim to study the prevalence of vitamin D insufficiency (< 50 nmol/L) and the association of 25-hydroxyvitamin D with metabolic risk factors among an existing Malay cohort in Kuala Lumpur. Methods: This is an analytical cross sectional study. A total of 380 subjects were sampled and their vitamins D status (25-hydroxyvitamin D), fasting blood glucose, full lipid profile were assessed using venous blood. Systolic and diastolic blood pressure, weight, height and waist circumference were measured following standard protocols. Socio-demographic data such as sex, age, smoking status etc were also collected. Data was analysed using t-test, chi-square test, General Linear Model and multiple logistic regression. Results: Females made up 58 of the sample. The mean age of respondents was 48.5 (SD 5.2) years. Females had significantly lower mean Vitamin D levels (36.2; 95 CI: 34.5, 38.0 nmol/L) compared to males (56.2; 95 CI: 53.2, 59.2 nmol/L). Approximately 41 and 87 of males and females respectively had insufficient (< 50 nmol/L) levels of 25-hydroxyvitamin D (p < 0.001). The prevalence of Metabolic Syndrome for the whole sample was 38.4 (95 CI: 33.5, 43.3). In the multivariate model (adjusted for age, sex, abdominal obesity, HDL-cholesterol, diastolic blood pressure), insufficient Vitamin D status was significantly associated with 1-year age increments (OR: 0.93; 95 CI: 0.88, 0.98), being female (OR: 8.68; 95 CI: 5.08, 14.83) and abdominal obesity (OR: 2.57; 95 CI: 1.51, 4.39). Respondents with insufficient vitamin D were found to have higher odds of having Metabolic Syndrome (OR: 1.73; 95 CI: 1.02, 2.92) after adjusting for age and sex. Conclusions: Our results highlight the high prevalence of vitamin D insufficiency among Malay adults in Kuala Lumpur. Vitamin D insufficiency is independently associated with younger age, female sex and greater abdominal obesity. Vitamin D insufficiency is also associated with Metabolic Syndrome