A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

Pretraining is the preliminary and fundamental step in developing capable language models (LM). Despite this, pretraining data design is critically under-documented and often guided by empirically unsupported intuitions. To address this, we pretrain 28 1.5B parameter decoder-only models, training on data curated (1) at different times, (2) with varying toxicity and quality filters, and (3) with different domain compositions. First, we quantify the effect of pretraining data age. A temporal shift between evaluation data and pretraining data leads to performance degradation, which is not overcome by finetuning. Second, we explore the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data. We also find that the effects of different types of filtering are not predictable from text domain characteristics. Lastly, we empirically validate that the inclusion of heterogeneous data sources, like books and web, is broadly beneficial and warrants greater prioritization. These findings constitute the largest set of experiments to validate, quantify, and expose many undocumented intuitions about text pretraining, which we hope will help support more informed data-centric decisions in LM development

Making 'chemical cocktails' - evolution of urban geochemical processes across the periodic table of elements.

Urbanization contributes to the formation of novel elemental combinations and signatures in terrestrial and aquatic watersheds, also known as 'chemical cocktails.' The composition of chemical cocktails evolves across space and time due to: (1) elevated concentrations from anthropogenic sources, (2) accelerated weathering and corrosion of the built environment, (3) increased drainage density and intensification of urban water conveyance systems, and (4) enhanced rates of geochemical transformations due to changes in temperature, ionic strength, pH, and redox potentials. Characterizing chemical cocktails and underlying geochemical processes is necessary for: (1) tracking pollution sources using complex chemical mixtures instead of individual elements or compounds; (2) developing new strategies for co-managing groups of contaminants; (3) identifying proxies for predicting transport of chemical mixtures using continuous sensor data; and (4) determining whether interactive effects of chemical cocktails produce ecosystem-scale impacts greater than the sum of individual chemical stressors. First, we discuss some unique urban geochemical processes which form chemical cocktails, such as urban soil formation, human-accelerated weathering, urban acidification-alkalinization, and freshwater salinization syndrome. Second, we review and synthesize global patterns in concentrations of major ions, carbon and nutrients, and trace elements in urban streams across different world regions and make comparisons with reference conditions. In addition to our global analysis, we highlight examples from some watersheds in the Baltimore-Washington DC region, which show increased transport of major ions, trace metals, and nutrients across streams draining a well-defined land-use gradient. Urbanization increased the concentrations of multiple major and trace elements in streams draining human-dominated watersheds compared to reference conditions. Chemical cocktails of major and trace elements were formed over diurnal cycles coinciding with changes in streamflow, dissolved oxygen, pH, and other variables measured by high-frequency sensors. Some chemical cocktails of major and trace elements were also significantly related to specific conductance (p<0.05), which can be measured by sensors. Concentrations of major and trace elements increased, peaked, or decreased longitudinally along streams as watershed urbanization increased, which is consistent with distinct shifts in chemical mixtures upstream and downstream of other major cities in the world. Our global analysis of urban streams shows that concentrations of multiple elements along the Periodic Table significantly increase when compared with reference conditions. Furthermore, similar biogeochemical patterns and processes can be grouped among distinct mixtures of elements of major ions, dissolved organic matter, nutrients, and trace elements as chemical cocktails. Chemical cocktails form in urban waters over diurnal cycles, decades, and throughout drainage basins. We conclude our global review and synthesis by proposing strategies for monitoring and managing chemical cocktails using source control, ecosystem restoration, and green infrastructure. We discuss future research directions applying the watershed chemical cocktail approach to diagnose and manage environmental problems. Ultimately, a chemical cocktail approach targeting sources, transport, and transformations of different and distinct elemental combinations is necessary to more holistically monitor and manage the emerging impacts of chemical mixtures in the world's fresh waters

PaLM: Scaling Language Modeling with Pathways

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies

Synthesis report of the IPCC Sixth Assessment Report (AR6), Longer report. IPCC.

This Synthesis Report (SYR) of the IPCC Sixth Assessment Report (AR6) summarises the state of knowledge of climate change, its widespread impacts and risks, and climate change mitigation and adaptation, based on the peer-reviewed scientific, technical and socio-economic literature since the publication of the IPCC’s Fifth Assessment Report (AR5) in 2014. The assessment is undertaken within the context of the evolving international landscape, in particular, developments in the UN Framework Convention on Climate Change (UNFCCC) process, including the outcomes of the Kyoto Protocol and the adoption of the Paris Agreement. It reflects the increasing diversity of those involved in climate action. This report integrates the main findings of the AR6 Working Group reports1 and the three AR6 Special Reports. It recognizes the interdependence of climate, ecosystems and biodiversity, and human societies; the value of diverse forms of knowledge; and the close linkages between climate change adaptation, mitigation, ecosystem health, human well-being and sustainable development. Building on multiple analytical frameworks, including those from the physical and social sciences, this report identifies opportunities for transformative action which are effective, feasible, just and equitable using concepts of systems transitions and resilient development pathways. Different regional classification schemes are used for physical, social and economic aspects, reflecting the underlying literature. After this introduction, Section 2, ‘Current Status and Trends’, opens with the assessment of observational evidence for our changing climate, historical and current drivers of human-induced climate change, and its impacts. It assesses the current implementation of adaptation and mitigation response options. Section 3, ‘Long-Term Climate and Development Futures’, provides a long-term assessment of climate change to 2100 and beyond in a broad range of socio-economic futures. It considers long-term characteristics, impacts, risks and costs in adaptation and mitigation pathways in the context of sustainable development. Section 4, ‘Near-Term Responses in a Changing Climate’, assesses opportunities for scaling up effective action in the period up to 2040, in the context of climate pledges, and commitments, and the pursuit of sustainable development. Based on scientific understanding, key findings can be formulated as statements of fact or associated with an assessed level of confidence using the IPCC calibrated language5 . The scientific findings are drawn from the underlying reports and arise from their Summary for Policymakers (hereafter SPM), Technical Summary (hereafter TS), and underlying chapters and are indicated by {} brackets. Figure 1.1 shows the Synthesis Report Figures Key, a guide to visual icons that are used across multiple figures within this report

IPCC, 2023: Climate Change 2023: Synthesis Report, Summary for Policymakers. Contribution of Working Groups I, II and III to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change [Core Writing Team, H. Lee and J. Romero (eds.)]. IPCC, Geneva, Switzerland.

This Synthesis Report (SYR) of the IPCC Sixth Assessment Report (AR6) summarises the state of knowledge of climate change, its widespread impacts and risks, and climate change mitigation and adaptation. It integrates the main findings of the Sixth Assessment Report (AR6) based on contributions from the three Working Groups1 , and the three Special Reports. The summary for Policymakers (SPM) is structured in three parts: SPM.A Current Status and Trends, SPM.B Future Climate Change, Risks, and Long-Term Responses, and SPM.C Responses in the Near Term.This report recognizes the interdependence of climate, ecosystems and biodiversity, and human societies; the value of diverse forms of knowledge; and the close linkages between climate change adaptation, mitigation, ecosystem health, human well-being and sustainable development, and reflects the increasing diversity of actors involved in climate action. Based on scientific understanding, key findings can be formulated as statements of fact or associated with an assessed level of confidence using the IPCC calibrated language

Selective dynamical imaging of interferometric data

Recent developments in very long baseline interferometry (VLBI) have made it possible for the Event Horizon Telescope (EHT) to resolve the innermost accretion flows of the largest supermassive black holes on the sky. The sparse nature of the EHT’s (u, v)-coverage presents a challenge when attempting to resolve highly time-variable sources. We demonstrate that the changing (u, v)-coverage of the EHT can contain regions of time over the course of a single observation that facilitate dynamical imaging. These optimal time regions typically have projected baseline distributions that are approximately angularly isotropic and radially homogeneous. We derive a metric of coverage quality based on baseline isotropy and density that is capable of ranking array configurations by their ability to produce accurate dynamical reconstructions. We compare this metric to existing metrics in the literature and investigate their utility by performing dynamical reconstructions on synthetic data from simulated EHT observations of sources with simple orbital variability. We then use these results to make recommendations for imaging the 2017 EHT Sgr A* data sethttp://iopscience.iop.org/2041-8205Physic

Evaluating the acceptability of remote cognitive remediation from the perspective of psychosis service users

OBJECTIVES: Cognitive remediation (CR) can reduce the cognitive difficulties experienced by people with psychosis. Adapting CR to be delivered remotely provides new opportunities for extending its use. However, doing so requires further evaluation of its acceptability from service users' views. We evaluate the acceptability of therapist-supported remote CR from the perspectives of service users using participatory service user-centred methods.METHOD: After receiving 12 weeks of therapist-supported remote CR, service users were interviewed by a service user researcher following a semi-structured 18-question interview guide. Transcripts were analysed using reflexive thematic analysis with themes and codes further validated by a Lived Experience Advisory Panel and member checking.RESULTS: The study recruited 26 participants, almost all of whom reported high acceptability of remote CR, and some suggested improvements. Four themes emerged: (1) perceived treatment benefits, (2) remote versus in-person therapy, (3) the therapist's role, and (4) how it could be better.CONCLUSIONS: This study used comprehensive service user involvement methods. For some participants, technology use remained a challenge and addressing these difficulties detracted from the therapy experience. These outcomes align with existing research on remote therapy, suggesting that remote CR can expand choice and improve access to treatment for psychosis service users once barriers are addressed. Future use of remote CR should consider technology training and equipment provision to facilitate therapy for service users and therapists.</p