1,191 research outputs found
Artificial intelligence in the cyber domain: Offense and defense
Artificial intelligence techniques have grown rapidly in recent years, and their applications in practice can be seen in many fields, ranging from facial recognition to image analysis. In the cybersecurity domain, AI-based techniques can provide better cyber defense tools and help adversaries improve methods of attack. However, malicious actors are aware of the new prospects too and will probably attempt to use them for nefarious purposes. This survey paper aims at providing an overview of how artificial intelligence can be used in the context of cybersecurity in both offense and defense.Web of Science123art. no. 41
The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions
We introduce dataset multiplicity, a way to study how inaccuracies,
uncertainty, and social bias in training datasets impact test-time predictions.
The dataset multiplicity framework asks a counterfactual question of what the
set of resultant models (and associated test-time predictions) would be if we
could somehow access all hypothetical, unbiased versions of the dataset. We
discuss how to use this framework to encapsulate various sources of uncertainty
in datasets' factualness, including systemic social bias, data collection
practices, and noisy labels or features. We show how to exactly analyze the
impacts of dataset multiplicity for a specific model architecture and type of
uncertainty: linear models with label errors. Our empirical analysis shows that
real-world datasets, under reasonable assumptions, contain many test samples
whose predictions are affected by dataset multiplicity. Furthermore, the choice
of domain-specific dataset multiplicity definition determines what samples are
affected, and whether different demographic groups are disparately impacted.
Finally, we discuss implications of dataset multiplicity for machine learning
practice and research, including considerations for when model outcomes should
not be trusted.Comment: 25 pages, 8 figures. Accepted at FAccT '2
Certifying the Fairness of KNN in the Presence of Dataset Bias
We propose a method for certifying the fairness of the classification result
of a widely used supervised learning algorithm, the k-nearest neighbors (KNN),
under the assumption that the training data may have historical bias caused by
systematic mislabeling of samples from a protected minority group. To the best
of our knowledge, this is the first certification method for KNN based on three
variants of the fairness definition: individual fairness, -fairness,
and label-flipping fairness. We first define the fairness certification problem
for KNN and then propose sound approximations of the complex arithmetic
computations used in the state-of-the-art KNN algorithm. This is meant to lift
the computation results from the concrete domain to an abstract domain, to
reduce the computational cost. We show effectiveness of this abstract
interpretation based technique through experimental evaluation on six datasets
widely used in the fairness research literature. We also show that the method
is accurate enough to obtain fairness certifications for a large number of test
inputs, despite the presence of historical bias in the datasets
PECAN: A Deterministic Certified Defense Against Backdoor Attacks
Neural networks are vulnerable to backdoor poisoning attacks, where the
attackers maliciously poison the training set and insert triggers into the test
input to change the prediction of the victim model. Existing defenses for
backdoor attacks either provide no formal guarantees or come with
expensive-to-compute and ineffective probabilistic guarantees. We present
PECAN, an efficient and certified approach for defending against backdoor
attacks. The key insight powering PECAN is to apply off-the-shelf test-time
evasion certification techniques on a set of neural networks trained on
disjoint partitions of the data. We evaluate PECAN on image classification and
malware detection datasets. Our results demonstrate that PECAN can (1)
significantly outperform the state-of-the-art certified backdoor defense, both
in defense strength and efficiency, and (2) on real back-door attacks, PECAN
can reduce attack success rate by order of magnitude when compared to a range
of baselines from the literature
Certifiable Robustness for Nearest Neighbor Classifiers
ML models are typically trained using large datasets of high quality. However, training datasets often contain inconsistent or incomplete data. To tackle this issue, one solution is to develop algorithms that can check whether a prediction of a model is certifiably robust. Given a learning algorithm that produces a classifier and given an example at test time, a classification outcome is certifiably robust if it is predicted by every model trained across all possible worlds (repairs) of the uncertain (inconsistent) dataset. This notion of robustness falls naturally under the framework of certain answers. In this paper, we study the complexity of certifying robustness for a simple but widely deployed classification algorithm, k-Nearest Neighbors (k-NN). Our main focus is on inconsistent datasets when the integrity constraints are functional dependencies (FDs). For this setting, we establish a dichotomy in the complexity of certifying robustness w.r.t. the set of FDs: the problem either admits a polynomial time algorithm, or it is coNP-hard. Additionally, we exhibit a similar dichotomy for the counting version of the problem, where the goal is to count the number of possible worlds that predict a certain label. As a byproduct of our study, we also establish the complexity of a problem related to finding an optimal subset repair that may be of independent interest
- …