Privately Releasing Conjunctions and the Statistical Query Barrier

Suppose we would like to know all answers to a set of statistical queries C on a data set up to small error, but we can only access the data itself using statistical queries. A trivial solution is to exhaustively ask all queries in C. Can we do any better? + We show that the number of statistical queries necessary and sufficient for this task is---up to polynomial factors---equal to the agnostic learning complexity of C in Kearns' statistical query (SQ) model. This gives a complete answer to the question when running time is not a concern. + We then show that the problem can be solved efficiently (allowing arbitrary error on a small fraction of queries) whenever the answers to C can be described by a submodular function. This includes many natural concept classes, such as graph cuts and Boolean disjunctions and conjunctions. While interesting from a learning theoretic point of view, our main applications are in privacy-preserving data analysis: Here, our second result leads to the first algorithm that efficiently releases differentially private answers to of all Boolean conjunctions with 1% average error. This presents significant progress on a key open problem in privacy-preserving data analysis. Our first result on the other hand gives unconditional lower bounds on any differentially private algorithm that admits a (potentially non-privacy-preserving) implementation using only statistical queries. Not only our algorithms, but also most known private algorithms can be implemented using only statistical queries, and hence are constrained by these lower bounds. Our result therefore isolates the complexity of agnostic learning in the SQ-model as a new barrier in the design of differentially private algorithms

Addressing Membership Inference Attack in Federated Learning with Model Compression

Federated Learning (FL) has been proposed as a privacy-preserving solution for machine learning. However, recent works have shown that Federated Learning can leak private client data through membership attacks. In this paper, we show that the effectiveness of these attacks on the clients negatively correlates with the size of the client datasets and model complexity. Based on this finding, we propose model-agnostic Federated Learning as a privacy-enhancing solution because it enables the use of models of varying complexity in the clients. To this end, we present $\texttt{MaPP-FL}$, a novel privacy-aware FL approach that leverages model compression on the clients while keeping a full model on the server. We compare the performance of $\texttt{MaPP-FL}$ against state-of-the-art model-agnostic FL methods on the CIFAR-10, CIFAR-100, and FEMNIST vision datasets. Our experiments show the effectiveness of $\texttt{MaPP-FL}$ in preserving the clients' and the server's privacy while achieving competitive classification accuracies

Expectations, Learning and Macroeconomic Persistence

This paper presents an estimated model with learning and provides evidence that learning can improve the fit of popular monetary DSGE models and endogenously generate realistic levels of persistence. The paper starts with an agnostic view, developing a model that nests learning and some of the structural sources of persistence, such as habit formation in consumption and inflation indexation, that are typically needed in monetary models with rational expectations to match the persistence of macroeconomic variables. I estimate the model by likelihood-based Bayesian methods, which allow the estimation of the learning gain coefficient jointly with the "deep" parameters of the economy. The empirical results show that when learning replaces rational expectations, the estimated degrees of habits and indexation drop near zero. This ?nding suggests that persistence arises in the model economy mainly from expectations and learning. The posterior model probabilities show that the specification with learning fits significantly better than does the specification with rational expectations. Finally, if learning rather than mechanical sources of persistence provides a more appropriate representation of the economy, the implied optimal policy will be different. The policymaker will also incur substantial costs from misspecifying private expectations formation.Persistence, Constant-gain learning, Expectations, Habit formation in consumption, Inflation inertia; Phillips curve; Bayesian econometrics; New-Keynesian model.

Expectations, Learning and Macroeconomic Persistence

This paper presents an estimated model with learning and provides evidence that learning can improve the fit of popular monetary DSGE models and endogenously generate realistic levels of persistence. The paper starts with an agnostic view, developing a model that nests learning and some of the structural sources of persistence, such as habit formation in consumption and inflation indexation, that are typically needed in monetary models with rational expectations to match the persistence of macroeconomic variables. I estimate the model by likelihood-based Bayesian methods, which allow the estimation of the learning gain coefficient jointly with the `deep' parameters of the economy. The empirical results show that when learning replaces rational expectations, the estimated degrees of habits and indexation drop near zero. This finding suggests that persistence arises in the model economy mainly from expectations and learning. The posterior model probabilities show that the specification with learning fits significantly better than does the specification with rational expectations. Finally, if learning rather than mechanical sources of persistence provides a more appropriate representation of the economy, the implied optimal policy will be different. The policymaker will also incur substantial costs from misspecifying private expectations formation.persistence, constant-gain learning, expectations, habit formation in consumption, inflation inertia, Phillips curve, Bayesian econometrics, New-Keynesian model.

Sample Complexity Bounds on Differentially Private Learning via Communication Complexity

In this work we analyze the sample complexity of classification by differentially private algorithms. Differential privacy is a strong and well-studied notion of privacy introduced by Dwork et al. (2006) that ensures that the output of an algorithm leaks little information about the data point provided by any of the participating individuals. Sample complexity of private PAC and agnostic learning was studied in a number of prior works starting with (Kasiviswanathan et al., 2008) but a number of basic questions still remain open, most notably whether learning with privacy requires more samples than learning without privacy. We show that the sample complexity of learning with (pure) differential privacy can be arbitrarily higher than the sample complexity of learning without the privacy constraint or the sample complexity of learning with approximate differential privacy. Our second contribution and the main tool is an equivalence between the sample complexity of (pure) differentially private learning of a concept class $C$ (or $SCDP(C)$) and the randomized one-way communication complexity of the evaluation problem for concepts from $C$. Using this equivalence we prove the following bounds: 1. $SCDP(C) = \Omega(LDim(C))$, where $LDim(C)$ is the Littlestone's (1987) dimension characterizing the number of mistakes in the online-mistake-bound learning model. Known bounds on $LDim(C)$ then imply that $SCDP(C)$ can be much higher than the VC-dimension of $C$. 2. For any $t$, there exists a class $C$ such that $LDim(C)=2$ but $SCDP(C) \geq t$. 3. For any $t$, there exists a class $C$ such that the sample complexity of (pure) $\alpha$-differentially private PAC learning is $\Omega(t/\alpha)$ but the sample complexity of the relaxed $(\alpha,\beta)$-differentially private PAC learning is $O(\log(1/\beta)/\alpha)$. This resolves an open problem of Beimel et al. (2013b).Comment: Extended abstract appears in Conference on Learning Theory (COLT) 201