252 research outputs found
Almost Tight L0-norm Certified Robustness of Top-k Predictions against Adversarial Perturbations
Top- predictions are used in many real-world applications such as machine
learning as a service, recommender systems, and web searches. -norm
adversarial perturbation characterizes an attack that arbitrarily modifies some
features of an input such that a classifier makes an incorrect prediction for
the perturbed input. -norm adversarial perturbation is easy to
interpret and can be implemented in the physical world. Therefore, certifying
robustness of top- predictions against -norm adversarial
perturbation is important. However, existing studies either focused on
certifying -norm robustness of top- predictions or -norm
robustness of top- predictions. In this work, we aim to bridge the gap. Our
approach is based on randomized smoothing, which builds a provably robust
classifier from an arbitrary classifier via randomizing an input. Our major
theoretical contribution is an almost tight -norm certified robustness
guarantee for top- predictions. We empirically evaluate our method on
CIFAR10 and ImageNet. For instance, our method can build a classifier that
achieves a certified top-3 accuracy of 69.2\% on ImageNet when an attacker can
arbitrarily perturb 5 pixels of a testing image
Scalable Certified Segmentation via Randomized Smoothing
We present a new certification method for image and point cloud segmentation
based on randomized smoothing. The method leverages a novel scalable algorithm
for prediction and certification that correctly accounts for multiple testing,
necessary for ensuring statistical guarantees. The key to our approach is
reliance on established multiple-testing correction mechanisms as well as the
ability to abstain from classifying single pixels or points while still
robustly segmenting the overall input. Our experimental evaluation on synthetic
data and challenging datasets, such as Pascal Context, Cityscapes, and
ShapeNet, shows that our algorithm can achieve, for the first time, competitive
accuracy and certification guarantees on real-world segmentation tasks. We
provide an implementation at https://github.com/eth-sri/segmentation-smoothing.Comment: ICML'2
Robustness of Machine Learning Models Beyond Adversarial Attacks
Correctly quantifying the robustness of machine learning models is a central
aspect in judging their suitability for specific tasks, and thus, ultimately,
for generating trust in the models. We show that the widely used concept of
adversarial robustness and closely related metrics based on counterfactuals are
not necessarily valid metrics for determining the robustness of ML models
against perturbations that occur "naturally", outside specific adversarial
attack scenarios. Additionally, we argue that generic robustness metrics in
principle are insufficient for determining real-world-robustness. Instead we
propose a flexible approach that models possible perturbations in input data
individually for each application. This is then combined with a probabilistic
approach that computes the likelihood that a real-world perturbation will
change a prediction, thus giving quantitative information of the robustness of
the trained machine learning model. The method does not require access to the
internals of the classifier and thus in principle works for any black-box
model. It is, however, based on Monte-Carlo sampling and thus only suited for
input spaces with small dimensions. We illustrate our approach on two dataset,
as well as on analytically solvable cases. Finally, we discuss ideas on how
real-world robustness could be computed or estimated in high-dimensional input
spaces.Comment: 25 pages, 7 figure
How Technology Impacts and Compares to Humans in Socially Consequential Arenas
One of the main promises of technology development is for it to be adopted by
people, organizations, societies, and governments -- incorporated into their
life, work stream, or processes. Often, this is socially beneficial as it
automates mundane tasks, frees up more time for other more important things, or
otherwise improves the lives of those who use the technology. However, these
beneficial results do not apply in every scenario and may not impact everyone
in a system the same way. Sometimes a technology is developed which produces
both benefits and inflicts some harm. These harms may come at a higher cost to
some people than others, raising the question: {\it how are benefits and harms
weighed when deciding if and how a socially consequential technology gets
developed?} The most natural way to answer this question, and in fact how
people first approach it, is to compare the new technology to what used to
exist. As such, in this work, I make comparative analyses between humans and
machines in three scenarios and seek to understand how sentiment about a
technology, performance of that technology, and the impacts of that technology
combine to influence how one decides to answer my main research question.Comment: Doctoral thesis proposal. arXiv admin note: substantial text overlap
with arXiv:2110.08396, arXiv:2108.12508, arXiv:2006.1262
Towards Specifying And Evaluating The Trustworthiness Of An AI-Enabled System
Applied AI has shown promise in the data processing of key industries and government agencies to extract actionable information used to make important strategical decisions. One of the core features of AI-enabled systems is the trustworthiness of these systems which has an important implication for the robustness and full acceptance of these systems. In this paper, we explain what trustworthiness in AI-enabled systems means, and the key technical challenges of specifying, and verifying trustworthiness. Toward solving these technical challenges, we propose a method to specify and evaluate the trustworthiness of AI-based systems using quality-attribute scenarios and design tactics. Using our trustworthiness scenarios and design tactics, we can analyze the architectural design of AI-enabled systems to ensure that trustworthiness has been properly expressed and achieved.The contributions of the thesis include (i) the identification of the trustworthiness sub-attributes that affect the trustworthiness of AI systems (ii) the proposal of trustworthiness scenarios to specify trustworthiness in an AI system (iii) a design checklist to support the analysis of the trustworthiness of AI systems and (iv) the identification of design tactics that can be used to achieve trustworthiness in an AI system
Recommended from our members
Security, Privacy, and Transparency Guarantees for Machine Learning Systems
Machine learning (ML) is transforming a wide range of applications, promising to bring immense economic and social benefits. However, it also raises substantial security, privacy, and transparency challenges. ML workloads indeed push companies toward aggressive data collection and loose data access policies, placing troves of sensitive user information at risk if the company is hacked. ML also introduces new attack vectors, such as adversarial example attacks, which can completely nullify models’ accuracy under attack. Finally, ML models make complex data-driven decisions, which are opaque to the end-users, and difficult to inspect for programmers. In this dissertation we describe three systems we developed. Each system addresses a dimension of the previous challenges, by combining new practical systems techniques with rigorous theory to achieve a guaranteed level of protection, and make systems easier to understand. First we present Sage, a differentially private ML platform that enforces a meaningful protection semantic for the troves of personal information amassed by today’s companies. Second we describe PixelDP, a defense against adversarial examples that leverages differential privacy theory to provide a guaranteed level of accuracy under attack. Third we introduce Sunlight, a tool to enhance the transparency of opaque targeting services, using rigorous causal inference theory to explain targeting decisions to end-users
- …