8,307 research outputs found
Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence
An important challenge for safety in machine learning and artificial
intelligence systems is a~set of related failures involving specification
gaming, reward hacking, fragility to distributional shifts, and Goodhart's or
Campbell's law. This paper presents additional failure modes for interactions
within multi-agent systems that are closely related. These multi-agent failure
modes are more complex, more problematic, and less well understood than the
single-agent case, and are also already occurring, largely unnoticed. After
motivating the discussion with examples from poker-playing artificial
intelligence (AI), the paper explains why these failure modes are in some
senses unavoidable. Following this, the paper categorizes failure modes,
provides definitions, and cites examples for each of the modes: accidental
steering, coordination failures, adversarial misalignment, input spoofing and
filtering, and goal co-option or direct hacking. The paper then discusses how
extant literature on multi-agent AI fails to address these failure modes, and
identifies work which may be useful for the mitigation of these failure modes.Comment: 12 Pages, This version re-submitted to Big Data and Cognitive
Computing, Special Issue "Artificial Superintelligence: Coordination &
Strategy
Privacy Risks of Securing Machine Learning Models against Adversarial Examples
The arms race between attacks and defenses for machine learning models has
come to a forefront in recent years, in both the security community and the
privacy community. However, one big limitation of previous research is that the
security domain and the privacy domain have typically been considered
separately. It is thus unclear whether the defense methods in one domain will
have any unexpected impact on the other domain.
In this paper, we take a step towards resolving this limitation by combining
the two domains. In particular, we measure the success of membership inference
attacks against six state-of-the-art defense methods that mitigate the risk of
adversarial examples (i.e., evasion attacks). Membership inference attacks
determine whether or not an individual data record has been part of a model's
training set. The accuracy of such attacks reflects the information leakage of
training algorithms about individual members of the training set. Adversarial
defense methods against adversarial examples influence the model's decision
boundaries such that model predictions remain unchanged for a small area around
each input. However, this objective is optimized on training data. Thus,
individual data records in the training set have a significant influence on
robust models. This makes the models more vulnerable to inference attacks.
To perform the membership inference attacks, we leverage the existing
inference methods that exploit model predictions. We also propose two new
inference methods that exploit structural properties of robust models on
adversarially perturbed data. Our experimental evaluation demonstrates that
compared with the natural training (undefended) approach, adversarial defense
methods can indeed increase the target model's risk against membership
inference attacks.Comment: ACM CCS 2019, code is available at
https://github.com/inspire-group/privacy-vs-robustnes
Securing Recommender System via Cooperative Training
Recommender systems are often susceptible to well-crafted fake profiles,
leading to biased recommendations. Among existing defense methods,
data-processing-based methods inevitably exclude normal samples, while
model-based methods struggle to enjoy both generalization and robustness. To
this end, we suggest integrating data processing and the robust model to
propose a general framework, Triple Cooperative Defense (TCD), which employs
three cooperative models that mutually enhance data and thereby improve
recommendation robustness. Furthermore, Considering that existing attacks
struggle to balance bi-level optimization and efficiency, we revisit poisoning
attacks in recommender systems and introduce an efficient attack strategy,
Co-training Attack (Co-Attack), which cooperatively optimizes the attack
optimization and model training, considering the bi-level setting while
maintaining attack efficiency. Moreover, we reveal a potential reason for the
insufficient threat of existing attacks is their default assumption of
optimizing attacks in undefended scenarios. This overly optimistic setting
limits the potential of attacks. Consequently, we put forth a Game-based
Co-training Attack (GCoAttack), which frames the proposed CoAttack and TCD as a
game-theoretic process, thoroughly exploring CoAttack's attack potential in the
cooperative training of attack and defense. Extensive experiments on three real
datasets demonstrate TCD's superiority in enhancing model robustness.
Additionally, we verify that the two proposed attack strategies significantly
outperform existing attacks, with game-based GCoAttack posing a greater
poisoning threat than CoAttack.Comment: arXiv admin note: text overlap with arXiv:2210.1376
Game Theory Meets Network Security: A Tutorial at ACM CCS
The increasingly pervasive connectivity of today's information systems brings
up new challenges to security. Traditional security has accomplished a long way
toward protecting well-defined goals such as confidentiality, integrity,
availability, and authenticity. However, with the growing sophistication of the
attacks and the complexity of the system, the protection using traditional
methods could be cost-prohibitive. A new perspective and a new theoretical
foundation are needed to understand security from a strategic and
decision-making perspective. Game theory provides a natural framework to
capture the adversarial and defensive interactions between an attacker and a
defender. It provides a quantitative assessment of security, prediction of
security outcomes, and a mechanism design tool that can enable
security-by-design and reverse the attacker's advantage. This tutorial provides
an overview of diverse methodologies from game theory that includes games of
incomplete information, dynamic games, mechanism design theory to offer a
modern theoretic underpinning of a science of cybersecurity. The tutorial will
also discuss open problems and research challenges that the CCS community can
address and contribute with an objective to build a multidisciplinary bridge
between cybersecurity, economics, game and decision theory
- …