Knowledge is a Region in Weight Space for Fine-tuned Language Models

Research on neural networks has focused on understanding a single model trained on a single dataset. However, relatively little is known about the relationships between different models, particularly those trained or tested on different datasets. We address this by studying how the weight space and the underlying loss landscape of different models are interconnected. Specifically, we demonstrate that finetuned models that were optimized for high performance, reside in well-defined regions in weight space, and vice versa -- that any model that resides anywhere in those regions also exhibits high performance. Notably, we show that language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuned on different datasets from the same underlying task form a looser cluster. Moreover, traversing around the region between the models leads to new models that perform comparably or even better than models obtained via finetuning, even on tasks that the original models were not finetuned on. Our findings provide insight into the relationships between models, demonstrating that a model positioned between two similar models can acquire the knowledge of both. We leverage this and design a method for selecting a better model for efficient finetuning. Specifically, we show that starting from the center of the region is as effective, if not more, than using the pretrained model in 11 out of 12 datasets, resulting in an average accuracy improvement of 3.06

The dissipative quantum Duffing oscillator: a comparison of Floquet-based approaches

We study the dissipative quantum Duffing oscillator in the deep quantum regime with two different approaches: The first is based on the exact Floquet states of the linear oscillator and the nonlinearity is treated perturbatively. It well describes the nonlinear oscillator dynamics away from resonance. The second, in contrast, is applicable at and in the vicinity of a N-photon resonance and it exploits quasi-degenerate perturbation theory for the nonlinear oscillator in Floquet space. It is perturbative both in driving and nonlinearity. A combination of both approaches yields the possibility to cover the whole range of driving frequencies. As an example we discuss the dissipative dynamics of the Duffing oscillator near and at the one-photon resonance.Comment: 38 pages, 4 figure

Hypothalamus–Muscle Parallel Induction of Metabolic Pathways Following Physical Exercise

The modern lifestyle requires less physical activity and skills during our daily routine, leading to multiple pathologies related to physical disabilities and energy accessibility. Thus, exploring the mechanisms underlying the metabolic regulation of exercise is crucial. Here, we characterized the effect of forced and voluntary endurance exercises on three key metabolic signaling pathways, sirtuins, AMPK, and mTOR, across several metabolic tissues in mice: brain, muscles, and liver. Both voluntary and forced exercises induced AMPK with higher intensity in the first. The comparison between those metabolic tissues revealed that the hypothalamus and the hippocampus, two brain parts, showed different metabolic signaling activities. Strikingly, despite the major differences in the physiology of muscles and hypothalamic tissues, the hypothalamus replicates the metabolic response of the muscle in response to physical exercise. Specifically, muscles and hypothalamic tissues showed an increase and a decrease in AMPK and mTOR signaling, respectively. Overall, this study reveals new insight into the relation between the hypothalamus and muscles, which enhances the coordination within the muscle–brain axis and potentially improves the systemic response to physical activity performance and delaying health inactivity disorders

Theories of Reference: What Was the Question?

The new theory of reference has won popularity. However, a number of noted philosophers have also attempted to reply to the critical arguments of Kripke and others, and aimed to vindicate the description theory of reference. Such responses are often based on ingenious novel kinds of descriptions, such as rigidified descriptions, causal descriptions, and metalinguistic descriptions. This prolonged debate raises the doubt whether different parties really have any shared understanding of what the central question of the philosophical theory of reference is: what is the main question to which descriptivism and the causal-historical theory have presented competing answers. One aim of the paper is to clarify this issue. The most influential objections to the new theory of reference are critically reviewed. Special attention is also paid to certain important later advances in the new theory of reference, due to Devitt and others

Optically levitated nanoparticle as a model system for stochastic bistable dynamics

Nano-mechanical resonators have gained an increasing importance in nanotechnology owing to their contributions to both fundamental and applied science. Yet, their small dimensions and mass raises some challenges as their dynamics gets dominated by nonlinearities that degrade their performance, for instance in sensing applications. Here, we report on the precise control of the nonlinear and stochastic bistable dynamics of a levitated nanoparticle in high vacuum. We demonstrate how it can lead to efficient signal amplification schemes, including stochastic resonance. This work contributes to showing the use of levitated nanoparticles as a model system for stochastic bistable dynamics, with applications to a wide variety of fields.inancial support from the ERC- QnanoMECA (Grant No. 64790), the Spanish Ministry of Economy and Competitiveness, under grant FIS2016-80293-R and through the ‘Severo Ochoa’ Programme for Centres of Excellence in R&D (SEV-2015-0522), Fundació Privada CELLEX and from the CERCA Programme/Generalitat de Catalunya. J.G. has been supported by H2020-MSCA-IF-2014 under REA grant Agreement No. 655369. L.R. acknowledges support from an ETH Marie Curie Cofund Fellowship

Mechanical systems in the quantum regime

Mechanical systems are ideal candidates for studying quantumbehavior of macroscopic objects. To this end, a mechanical resonator has to be cooled to its ground state and its position has to be measured with great accuracy. Currently, various routes to reach these goals are being explored. In this review, we discuss different techniques for sensitive position detection and we give an overview of the cooling techniques that are being employed. The latter include sideband cooling and active feedback cooling. The basic concepts that are important when measuring on mechanical systems with high accuracy and/or at very low temperatures, such as thermal and quantum noise, linear response theory, and backaction, are explained. From this, the quantum limit on linear position detection is obtained and the sensitivities that have been achieved in recent opto and nanoelectromechanical experiments are compared to this limit. The mechanical resonators that are used in the experiments range from meter-sized gravitational wave detectors to nanomechanical systems that can only be read out using mesoscopic devices such as single-electron transistors or superconducting quantum interference devices. A special class of nanomechanical systems are bottom-up fabricated carbon-based devices, which have very high frequencies and yet a large zero-point motion, making them ideal for reaching the quantum regime. The mechanics of some of the different mechanical systems at the nanoscale is studied. We conclude this review with an outlook of how state-of-the-art mechanical resonators can be improved to study quantum {\it mechanics}.Comment: To appear in Phys. Re

Iron Behaving Badly: Inappropriate Iron Chelation as a Major Contributor to the Aetiology of Vascular and Other Progressive Inflammatory and Degenerative Diseases

The production of peroxide and superoxide is an inevitable consequence of aerobic metabolism, and while these particular "reactive oxygen species" (ROSs) can exhibit a number of biological effects, they are not of themselves excessively reactive and thus they are not especially damaging at physiological concentrations. However, their reactions with poorly liganded iron species can lead to the catalytic production of the very reactive and dangerous hydroxyl radical, which is exceptionally damaging, and a major cause of chronic inflammation. We review the considerable and wide-ranging evidence for the involvement of this combination of (su)peroxide and poorly liganded iron in a large number of physiological and indeed pathological processes and inflammatory disorders, especially those involving the progressive degradation of cellular and organismal performance. These diseases share a great many similarities and thus might be considered to have a common cause (i.e. iron-catalysed free radical and especially hydroxyl radical generation). The studies reviewed include those focused on a series of cardiovascular, metabolic and neurological diseases, where iron can be found at the sites of plaques and lesions, as well as studies showing the significance of iron to aging and longevity. The effective chelation of iron by natural or synthetic ligands is thus of major physiological (and potentially therapeutic) importance. As systems properties, we need to recognise that physiological observables have multiple molecular causes, and studying them in isolation leads to inconsistent patterns of apparent causality when it is the simultaneous combination of multiple factors that is responsible. This explains, for instance, the decidedly mixed effects of antioxidants that have been observed, etc...Comment: 159 pages, including 9 Figs and 2184 reference