50 research outputs found
MACE: A Flexible Framework for Membership Privacy Estimation in Generative Models
In this work, we formally study the membership privacy risk of generative
models and propose a membership privacy estimation framework. We formulate the
membership privacy risk as a statistical divergence between training samples
and hold-out samples, and propose sample-based methods to estimate this
divergence. Unlike previous works, our proposed metric and estimators make
realistic and flexible assumptions. First, we offer a generalizable metric as
an alternative to accuracy for imbalanced datasets. Second, our estimators are
capable of estimating the membership privacy risk given any scalar or vector
valued attributes from the learned model, while prior work require access to
specific attributes. This allows our framework to provide data-driven
certificates for trained generative models in terms of membership privacy risk.
Finally, we show a connection to differential privacy, which allows our
proposed estimators to be used to understand the privacy budget 'epsilon'
needed for differentially private generative models. We demonstrate the utility
of our framework through experimental demonstrations on different generative
models using various model attributes yielding some new insights about
membership leakage and vulnerabilities of models
Revisiting pre-trained remote sensing model benchmarks: resizing and normalization matters
Research in self-supervised learning (SSL) with natural images has progressed
rapidly in recent years and is now increasingly being applied to and
benchmarked with datasets containing remotely sensed imagery. A common
benchmark case is to evaluate SSL pre-trained model embeddings on datasets of
remotely sensed imagery with small patch sizes, e.g., 32x32 pixels, whereas
standard SSL pre-training takes place with larger patch sizes, e.g., 224x224.
Furthermore, pre-training methods tend to use different image normalization
preprocessing steps depending on the dataset. In this paper, we show, across
seven satellite and aerial imagery datasets of varying resolution, that by
simply following the preprocessing steps used in pre-training (precisely, image
sizing and normalization methods), one can achieve significant performance
improvements when evaluating the extracted features on downstream tasks -- an
important detail overlooked in previous work in this space. We show that by
following these steps, ImageNet pre-training remains a competitive baseline for
satellite imagery based transfer learning tasks -- for example we find that
these steps give +32.28 to overall accuracy on the So2Sat random split dataset
and +11.16 on the EuroSAT dataset. Finally, we report comprehensive benchmark
results with a variety of simple baseline methods for each of the seven
datasets, forming an initial benchmark suite for remote sensing imagery
Assessment of Differentially Private Synthetic Data for Utility and Fairness in End-to-End Machine Learning Pipelines for Tabular Data
Differentially private (DP) synthetic data sets are a solution for sharing
data while preserving the privacy of individual data providers. Understanding
the effects of utilizing DP synthetic data in end-to-end machine learning
pipelines impacts areas such as health care and humanitarian action, where data
is scarce and regulated by restrictive privacy laws. In this work, we
investigate the extent to which synthetic data can replace real, tabular data
in machine learning pipelines and identify the most effective synthetic data
generation techniques for training and evaluating machine learning models. We
investigate the impacts of differentially private synthetic data on downstream
classification tasks from the point of view of utility as well as fairness. Our
analysis is comprehensive and includes representatives of the two main types of
synthetic data generation algorithms: marginal-based and GAN-based. To the best
of our knowledge, our work is the first that: (i) proposes a training and
evaluation framework that does not assume that real data is available for
testing the utility and fairness of machine learning models trained on
synthetic data; (ii) presents the most extensive analysis of synthetic data set
generation algorithms in terms of utility and fairness when used for training
machine learning models; and (iii) encompasses several different definitions of
fairness. Our findings demonstrate that marginal-based synthetic data
generators surpass GAN-based ones regarding model training utility for tabular
data. Indeed, we show that models trained using data generated by
marginal-based algorithms can exhibit similar utility to models trained using
real data. Our analysis also reveals that the marginal-based synthetic data
generator MWEM PGM can train models that simultaneously achieve utility and
fairness characteristics close to those obtained by models trained with real
data.Comment: arXiv admin note: text overlap with arXiv:2106.1024
Dwelling Type Classification for Disaster Risk Assessment Using Satellite Imagery
Vulnerability and risk assessment of neighborhoods is essential for effective
disaster preparedness. Existing traditional systems, due to dependency on
time-consuming and cost-intensive field surveying, do not provide a scalable
way to decipher warnings and assess the precise extent of the risk at a
hyper-local level. In this work, machine learning was used to automate the
process of identifying dwellings and their type to build a potentially more
effective disaster vulnerability assessment system. First, satellite imageries
of low-income settlements and vulnerable areas in India were used to identify 7
different dwelling types. Specifically, we formulated the dwelling type
classification as a semantic segmentation task and trained a U-net based neural
network model, namely TernausNet, with the data we collected. Then a risk score
assessment model was employed, using the determined dwelling type along with an
inundation model of the regions. The entire pipeline was deployed to multiple
locations prior to natural hazards in India in 2020. Post hoc ground-truth data
from those regions was collected to validate the efficacy of this model which
showed promising performance. This work can aid disaster response organizations
and communities at risk by providing household-level risk information that can
inform preemptive actions.Comment: Accepted for presentation in AI+HADR workshop, Neurips 202