MACE: A Flexible Framework for Membership Privacy Estimation in Generative Models

Abstract

In this work, we formally study the membership privacy risk of generative models and propose a membership privacy estimation framework. We formulate the membership privacy risk as a statistical divergence between training samples and hold-out samples, and propose sample-based methods to estimate this divergence. Unlike previous works, our proposed metric and estimators make realistic and flexible assumptions. First, we offer a generalizable metric as an alternative to accuracy for imbalanced datasets. Second, our estimators are capable of estimating the membership privacy risk given any scalar or vector valued attributes from the learned model, while prior work require access to specific attributes. This allows our framework to provide data-driven certificates for trained generative models in terms of membership privacy risk. Finally, we show a connection to differential privacy, which allows our proposed estimators to be used to understand the privacy budget 'epsilon' needed for differentially private generative models. We demonstrate the utility of our framework through experimental demonstrations on different generative models using various model attributes yielding some new insights about membership leakage and vulnerabilities of models