Heavy Hitters and the Structure of Local Privacy

We present a new locally differentially private algorithm for the heavy hitters problem which achieves optimal worst-case error as a function of all standardly considered parameters. Prior work obtained error rates which depend optimally on the number of users, the size of the domain, and the privacy parameter, but depend sub-optimally on the failure probability. We strengthen existing lower bounds on the error to incorporate the failure probability, and show that our new upper bound is tight with respect to this parameter as well. Our lower bound is based on a new understanding of the structure of locally private protocols. We further develop these ideas to obtain the following general results beyond heavy hitters. $\bullet$ Advanced Grouposition: In the local model, group privacy for $k$ users degrades proportionally to $\approx \sqrt{k}$, instead of linearly in $k$ as in the central model. Stronger group privacy yields improved max-information guarantees, as well as stronger lower bounds (via "packing arguments"), over the central model. $\bullet$ Building on a transformation of Bassily and Smith (STOC 2015), we give a generic transformation from any non-interactive approximate-private local protocol into a pure-private local protocol. Again in contrast with the central model, this shows that we cannot obtain more accurate algorithms by moving from pure to approximate local privacy

Private Multiplicative Weights Beyond Linear Queries

A wide variety of fundamental data analyses in machine learning, such as linear and logistic regression, require minimizing a convex function defined by the data. Since the data may contain sensitive information about individuals, and these analyses can leak that sensitive information, it is important to be able to solve convex minimization in a privacy-preserving way. A series of recent results show how to accurately solve a single convex minimization problem in a differentially private manner. However, the same data is often analyzed repeatedly, and little is known about solving multiple convex minimization problems with differential privacy. For simpler data analyses, such as linear queries, there are remarkable differentially private algorithms such as the private multiplicative weights mechanism (Hardt and Rothblum, FOCS 2010) that accurately answer exponentially many distinct queries. In this work, we extend these results to the case of convex minimization and show how to give accurate and differentially private solutions to *exponentially many* convex minimization problems on a sensitive dataset

Private Incremental Regression

Data is continuously generated by modern data sources, and a recent challenge in machine learning has been to develop techniques that perform well in an incremental (streaming) setting. In this paper, we investigate the problem of private machine learning, where as common in practice, the data is not given at once, but rather arrives incrementally over time. We introduce the problems of private incremental ERM and private incremental regression where the general goal is to always maintain a good empirical risk minimizer for the history observed under differential privacy. Our first contribution is a generic transformation of private batch ERM mechanisms into private incremental ERM mechanisms, based on a simple idea of invoking the private batch ERM procedure at some regular time intervals. We take this construction as a baseline for comparison. We then provide two mechanisms for the private incremental regression problem. Our first mechanism is based on privately constructing a noisy incremental gradient function, which is then used in a modified projected gradient procedure at every timestep. This mechanism has an excess empirical risk of $\approx\sqrt{d}$, where $d$ is the dimensionality of the data. While from the results of [Bassily et al. 2014] this bound is tight in the worst-case, we show that certain geometric properties of the input and constraint set can be used to derive significantly better results for certain interesting regression problems.Comment: To appear in PODS 201

Privacy Amplification for Matrix Mechanisms

Privacy amplification exploits randomness in data selection to provide tighter differential privacy (DP) guarantees. This analysis is key to DP-SGD's success in machine learning, but, is not readily applicable to the newer state-of-the-art algorithms. This is because these algorithms, known as DP-FTRL, use the matrix mechanism to add correlated noise instead of independent noise as in DP-SGD. In this paper, we propose "MMCC", the first algorithm to analyze privacy amplification via sampling for any generic matrix mechanism. MMCC is nearly tight in that it approaches a lower bound as $\epsilon\to0$. To analyze correlated outputs in MMCC, we prove that they can be analyzed as if they were independent, by conditioning them on prior outputs. Our "conditional composition theorem" has broad utility: we use it to show that the noise added to binary-tree-DP-FTRL can asymptotically match the noise added to DP-SGD with amplification. Our amplification algorithm also has practical empirical utility: we show it leads to significant improvement in the privacy-utility trade-offs for DP-FTRL algorithms on standard benchmarks

Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning

We introduce new differentially private (DP) mechanisms for gradient-based machine learning (ML) with multiple passes (epochs) over a dataset, substantially improving the achievable privacy-utility-computation tradeoffs. We formalize the problem of DP mechanisms for adaptive streams with multiple participations and introduce a non-trivial extension of online matrix factorization DP mechanisms to our setting. This includes establishing the necessary theory for sensitivity calculations and efficient computation of optimal matrices. For some applications like $>\!\! 10,000$ SGD steps, applying these optimal techniques becomes computationally expensive. We thus design an efficient Fourier-transform-based mechanism with only a minor utility loss. Extensive empirical evaluation on both example-level DP for image classification and user-level DP for language modeling demonstrate substantial improvements over all previous methods, including the widely-used DP-SGD . Though our primary application is to ML, our main DP results are applicable to arbitrary linear queries and hence may have much broader applicability.Comment: 9 pages main-text, 3 figures. 40 pages with 13 figures tota

Correlated Noise Provably Beats Independent Noise for Differentially Private Learning

Differentially private learning algorithms inject noise into the learning process. While the most common private learning algorithm, DP-SGD, adds independent Gaussian noise in each iteration, recent work on matrix factorization mechanisms has shown empirically that introducing correlations in the noise can greatly improve their utility. We characterize the asymptotic learning utility for any choice of the correlation function, giving precise analytical bounds for linear regression and as the solution to a convex program for general convex functions. We show, using these bounds, how correlated noise provably improves upon vanilla DP-SGD as a function of problem parameters such as the effective dimension and condition number. Moreover, our analytical expression for the near-optimal correlation function circumvents the cubic complexity of the semi-definite program used to optimize the noise correlation matrix in previous work. We validate our theory with experiments on private deep learning. Our work matches or outperforms prior work while being efficient both in terms of compute and memory.Comment: Christopher A. Choquette-Choo, Krishnamurthy Dvijotham, and Krishna Pillutla contributed equall

Luspatercept stimulates erythropoiesis, increases iron utilization, and redistributes body iron in transfusion-dependent thalassemia

Luspatercept, a ligand-trapping fusion protein, binds select TGF-β superfamily ligands implicated in thalassemic erythropoiesis, promoting late-stage erythroid maturation. Luspatercept reduced transfusion burden in the BELIEVE trial (NCT02604433) of 336 adults with transfusion-dependent thalassemia (TDT). Analysis of biomarkers in BELIEVE offers novel physiological and clinical insights into benefits offered by luspatercept. Transfusion iron loading rates decreased 20% by 1.4 g (~7 blood units; median iron loading rate difference: −0.05 ± 0.07 mg Fe/kg/day, p< .0001) and serum ferritin (s-ferritin) decreased 19.2% by 269.3 ± 963.7 μg/L (p < .0001), indicating reduced macrophage iron. However, liver iron content (LIC) did not decrease but showed statistically nonsignificant increases from 5.3 to 6.7 mg/g dw. Erythropoietin, growth differentiation factor 15, soluble transferrin receptor 1 (sTfR1), and reticulocytes rose by 93%, 59%, 66%, and 112%, respectively; accordingly, erythroferrone increased by 51% and hepcidin decreased by 53% (all p < .0001). Decreased transfusion with luspatercept in patients with TDT was associated with increased erythropoietic markers and decreasing hepcidin. Furthermore, s-ferritin reduction associated with increased erythroid iron incorporation (marked by sTfR1) allowed increased erythrocyte marrow output, consequently reducing transfusion needs and enhancing rerouting of hemolysis (heme) iron and non-transferrin-bound iron to the liver. LIC increased in patients with intact spleens, consistent with iron redistribution given the hepcidin reduction. Thus, erythropoietic and hepcidin changes with luspatercept in TDT lower transfusion dependency and may redistribute iron from macrophages to hepatocytes, necessitating the use of concomitant chelator cover for effective iron management

Ultrasonic Bioreactor as a Platform for Studying Cellular Response

The need for tissue-engineered constructs as replacement tissue continues to grow as the average age of the world’s population increases. However, additional research is required before the efficient production of laboratory-created tissue can be realized. The multitude of parameters that affect cell growth and proliferation is particularly daunting considering that optimized conditions are likely to change as a function of growth. Thus, a generalized research platform is needed in order for quantitative studies to be conducted. In this article, an ultrasonic bioreactor is described for use in studying the response of cells to ultrasonic stimulation. The work is focused on chondrocytes with a long-term view of generating tissue-engineered articular cartilage. Aspects of ultrasound (US) that would negatively affect cells, including temperature and cavitation, are shown to be insignificant for the US protocols used and which cover a wide range of frequencies and pressure amplitudes. The bioreactor is shown to have a positive influence on several factors, including cell proliferation, viability, and gene expression of select chondrocytic markers. Most importantly, we show that a total of 138 unique proteins are differentially expressed on exposure to ultrasonic stimulation, using mass-spectroscopy coupled proteomic analyses. We anticipate that this work will serve as the basis for additional research which will elucidate many of the mechanisms associated with cell response to ultrasonic stimulation

Making sense of the bizarre behaviour of horizons in the McVittie spacetime

The bizarre behaviour of the apparent (black hole and cosmological) horizons of the McVittie spacetime is discussed using, as an analogy, the Schwarzschild-de Sitter-Kottler spacetime (which is a special case of McVittie anyway). For a dust-dominated "background" universe, a black hole cannot exist at early times because its (apparent) horizon would be larger than the cosmological(apparent) horizon. A phantom-dominated "background" universe causes this situation, and the horizon behaviour, to be time-reversed.Comment: 8 pages, 3 figure

Ultrasonic Bioreactor as a Platform for Studying Cellular Response

The need for tissue-engineered constructs as replacement tissue continues to grow as the average age of the world’s population increases. However, additional research is required before the efficient production of laboratory-created tissue can be realized. The multitude of parameters that affect cell growth and proliferation is particularly daunting considering that optimized conditions are likely to change as a function of growth. Thus, a generalized research platform is needed in order for quantitative studies to be conducted. In this article, an ultrasonic bioreactor is described for use in studying the response of cells to ultrasonic stimulation. The work is focused on chondrocytes with a long-term view of generating tissue-engineered articular cartilage. Aspects of ultrasound (US) that would negatively affect cells, including temperature and cavitation, are shown to be insignificant for the US protocols used and which cover a wide range of frequencies and pressure amplitudes. The bioreactor is shown to have a positive influence on several factors, including cell proliferation, viability, and gene expression of select chondrocytic markers. Most importantly, we show that a total of 138 unique proteins are differentially expressed on exposure to ultrasonic stimulation, using mass-spectroscopy coupled proteomic analyses. We anticipate that this work will serve as the basis for additional research which will elucidate many of the mechanisms associated with cell response to ultrasonic stimulation