Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply applying the unlikelihood training objective to target token sequences is effective at forgetting them with little to no degradation of general language modeling performances; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being orders of magnitude more computationally efficient. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning

Implicating genes, pleiotropy, and sexual dimorphism at blood lipid loci through multi-ancestry meta-analysis

Abstract Background Genetic variants within nearly 1000 loci are known to contribute to modulation of blood lipid levels. However, the biological pathways underlying these associations are frequently unknown, limiting understanding of these findings and hindering downstream translational efforts such as drug target discovery. Results To expand our understanding of the underlying biological pathways and mechanisms controlling blood lipid levels, we leverage a large multi-ancestry meta-analysis (N = 1,654,960) of blood lipids to prioritize putative causal genes for 2286 lipid associations using six gene prediction approaches. Using phenome-wide association (PheWAS) scans, we identify relationships of genetically predicted lipid levels to other diseases and conditions. We confirm known pleiotropic associations with cardiovascular phenotypes and determine novel associations, notably with cholelithiasis risk. We perform sex-stratified GWAS meta-analysis of lipid levels and show that 3–5% of autosomal lipid-associated loci demonstrate sex-biased effects. Finally, we report 21 novel lipid loci identified on the X chromosome. Many of the sex-biased autosomal and X chromosome lipid loci show pleiotropic associations with sex hormones, emphasizing the role of hormone regulation in lipid metabolism. Conclusions Taken together, our findings provide insights into the biological mechanisms through which associated variants lead to altered lipid levels and potentially cardiovascular disease risk

Implicating genes, pleiotropy, and sexual dimorphism at blood lipid loci through multi-ancestry meta-analysis

Funding Information: GMP, PN, and CW are supported by NHLBI R01HL127564. GMP and PN are supported by R01HL142711. AG acknowledge support from the Wellcome Trust (201543/B/16/Z), European Union Seventh Framework Programme FP7/2007–2013 under grant agreement no. HEALTH-F2-2013–601456 (CVGenes@Target) & the TriPartite Immunometabolism Consortium [TrIC]-Novo Nordisk Foundation’s Grant number NNF15CC0018486. JMM is supported by American Diabetes Association Innovative and Clinical Translational Award 1–19-ICTS-068. SR was supported by the Academy of Finland Center of Excellence in Complex Disease Genetics (Grant No 312062), the Finnish Foundation for Cardiovascular Research, the Sigrid Juselius Foundation, and University of Helsinki HiLIFE Fellow and Grand Challenge grants. EW was supported by the Finnish innovation fund Sitra (EW) and Finska Läkaresällskapet. CNS was supported by American Heart Association Postdoctoral Fellowships 15POST24470131 and 17POST33650016. Charles N Rotimi is supported by Z01HG200362. Zhe Wang, Michael H Preuss, and Ruth JF Loos are supported by R01HL142302. NJT is a Wellcome Trust Investigator (202802/Z/16/Z), is the PI of the Avon Longitudinal Study of Parents and Children (MRC & WT 217065/Z/19/Z), is supported by the University of Bristol NIHR Biomedical Research Centre (BRC-1215–2001) and the MRC Integrative Epidemiology Unit (MC_UU_00011), and works within the CRUK Integrative Cancer Epidemiology Programme (C18281/A19169). Ruth E Mitchell is a member of the MRC Integrative Epidemiology Unit at the University of Bristol funded by the MRC (MC_UU_00011/1). Simon Haworth is supported by the UK National Institute for Health Research Academic Clinical Fellowship. Paul S. de Vries was supported by American Heart Association grant number 18CDA34110116. Julia Ramierz acknowledges support by the People Programme of the European Union’s Seventh Framework Programme grant n° 608765 and Marie Sklodowska-Curie grant n° 786833. Maria Sabater-Lleal is supported by a Miguel Servet contract from the ISCIII Spanish Health Institute (CP17/00142) and co-financed by the European Social Fund. Jian Yang is funded by the Westlake Education Foundation. Olga Giannakopoulou has received funding from the British Heart Foundation (BHF) (FS/14/66/3129). CHARGE Consortium cohorts were supported by R01HL105756. Study-specific acknowledgements are available in the Additional file : Supplementary Note. The views expressed in this manuscript are those of the authors and do not necessarily represent the views of the National Heart, Lung, and Blood Institute; the National Institutes of Health; or the U.S. Department of Health and Human Services. Publisher Copyright: © 2022, The Author(s).Background: Genetic variants within nearly 1000 loci are known to contribute to modulation of blood lipid levels. However, the biological pathways underlying these associations are frequently unknown, limiting understanding of these findings and hindering downstream translational efforts such as drug target discovery. Results: To expand our understanding of the underlying biological pathways and mechanisms controlling blood lipid levels, we leverage a large multi-ancestry meta-analysis (N = 1,654,960) of blood lipids to prioritize putative causal genes for 2286 lipid associations using six gene prediction approaches. Using phenome-wide association (PheWAS) scans, we identify relationships of genetically predicted lipid levels to other diseases and conditions. We confirm known pleiotropic associations with cardiovascular phenotypes and determine novel associations, notably with cholelithiasis risk. We perform sex-stratified GWAS meta-analysis of lipid levels and show that 3–5% of autosomal lipid-associated loci demonstrate sex-biased effects. Finally, we report 21 novel lipid loci identified on the X chromosome. Many of the sex-biased autosomal and X chromosome lipid loci show pleiotropic associations with sex hormones, emphasizing the role of hormone regulation in lipid metabolism. Conclusions: Taken together, our findings provide insights into the biological mechanisms through which associated variants lead to altered lipid levels and potentially cardiovascular disease risk.Peer reviewe

A model for hydraulic fracturing and proppant placement in unconsolidated sands

Hydraulic fracturing in unconsolidated or poorly consolidated formations has been used as a technique for well stimulation and for sand control. Although a large number of hydraulic fracturing operations have been performed in soft formations, the exact mechanisms of failure and fracture propagation remain an unresolved issue. Conventional hydraulic fracturing models based on the theory of linear elastic fracture mechanics (LEFM) consistently predict lower net fracturing pressure, smaller fracture widths and longer fracture lengths in soft formations than observed in the field. Operators who want to design and analyze frac-pack treatments routinely use a hard rock model and need to calibrate and often manipulate input parameters beyond a physically reasonable range to match the net fracturing pressure and well performance data. In this dissertation, we have developed a fully-coupled, three-dimensional hydraulic fracture model for poro-elasto-plastic materials and fluid flow coupled with proppant transport. A computational framework for fluid-structure interaction (FSI) based on finite volume method was developed for modeling of hydraulic fracturing and proppant placement in soft formations. Two separate domains, a fracture and a reservoir domain, are discretized individually, separate equations are solved in the two domains, and their interactions are modeled. The model includes the fully coupled process of power-law fluid flow inside the fracture with proppant transport, fluid leak-off from the fracture into the porous reservoir, pore pressure diffusion into the reservoir, inelastic deformation of the poro-elasto-plastic reservoir, and fracture propagation using a cohesive zone model along with a dynamic meshing procedure. Fully-coupled processes between the two domains, and pressure, flow and displacement coupling within each domain are modeled by an iterative and segregated solution procedure, where each component of the field variable is solved separately, consecutively, and iteratively. We verified the essential components of the model by comparing our simulation results with several well-known analytical solutions (elastoplastic deformation and failure problem, KGD model in a 2-D elastic domain, and KGD model in storage-toughness dominated regime). We applied the model to design and analyze frac-pack operations conducted in a Gulf of Mexico oilfield. Our model is capable of capturing the high net fracturing pressure commonly observed during frac-packing operations without adjusting any input parameters. The model shows quantitatively that plasticity causes lower stress concentration around the fracture tip which shields the tip of the propagating fracture from the fracturing pressure, and retards fracture growth. Our model predicts shorter fracture lengths and wider widths compared to a hard rock model. Shear failure around the fracture and ahead of the tip are modeled. Low cohesion sands tend to fail in shear first then in tension if sufficient pore pressure builds up. We investigated the effect of fluid viscosity, injection rate, and proppant diameter on fracture growth and proppant placement using sensitivity studies. Higher apparent fluid viscosity and injection rate results in wider fractures with better proppant placement, when the fracture is expected to be contained within the payzone. Utilizing larger diameter of proppant leads to settling-dominant proppant placement resulting in the formation of a proppant bank at the bottom of the induced fracture. The new frac-pack model for the first time allows operators to design and analyze hydraulic fracturing stimulations in soft, elastoplastic formations when complex fracturing fluids are used. Our results also provide guidelines for the selection of fracturing fluid rheology, proppant size, and injection rates.Petroleum and Geosystems Engineerin

2次元平面流れと3次元流れをカップリングした数値モデルによる氾濫解析手法の開発とその都市域への適用

京都大学0048新制・課程博士博士(工学)甲第16056号工博第3379号新制||工||1510(附属図書館)28635京都大学大学院工学研究科社会基盤工学専攻(主査)教授 中川 一, 教授 戸田 圭一, 教授 藤田 正治学位規則第4条第1項該当Doctor of Philosophy (Engineering)Kyoto UniversityDFA