6 research outputs found
NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems
This paper presents the Neural Network Verification (NNV) software tool, a
set-based verification framework for deep neural networks (DNNs) and
learning-enabled cyber-physical systems (CPS). The crux of NNV is a collection
of reachability algorithms that make use of a variety of set representations,
such as polyhedra, star sets, zonotopes, and abstract-domain representations.
NNV supports both exact (sound and complete) and over-approximate (sound)
reachability algorithms for verifying safety and robustness properties of
feed-forward neural networks (FFNNs) with various activation functions. For
learning-enabled CPS, such as closed-loop control systems incorporating neural
networks, NNV provides exact and over-approximate reachability analysis schemes
for linear plant models and FFNN controllers with piecewise-linear activation
functions, such as ReLUs. For similar neural network control systems (NNCS)
that instead have nonlinear plant models, NNV supports over-approximate
analysis by combining the star set analysis used for FFNN controllers with
zonotope-based analysis for nonlinear plant dynamics building on CORA. We
evaluate NNV using two real-world case studies: the first is safety
verification of ACAS Xu networks and the second deals with the safety
verification of a deep learning-based adaptive cruise control system
Verifying Safety of Neural Networks from Topological Perspectives
Neural networks (NNs) are increasingly applied in safety-critical systems
such as autonomous vehicles. However, they are fragile and are often
ill-behaved. Consequently, their behaviors should undergo rigorous guarantees
before deployment in practice. In this paper, we propose a set-boundary
reachability method to investigate the safety verification problem of NNs from
a topological perspective. Given an NN with an input set and a safe set, the
safety verification problem is to determine whether all outputs of the NN
resulting from the input set fall within the safe set. In our method, the
homeomorphism property and the open map property of NNs are mainly exploited,
which establish rigorous guarantees between the boundaries of the input set and
the boundaries of the output set. The exploitation of these two properties
facilitates reachability computations via extracting subsets of the input set
rather than the entire input set, thus controlling the wrapping effect in
reachability analysis and facilitating the reduction of computation burdens for
safety verification. The homeomorphism property exists in some widely used NNs
such as invertible residual networks (i-ResNets) and Neural ordinary
differential equations (Neural ODEs), and the open map is a less strict
property and easier to satisfy compared with the homeomorphism property. For
NNs establishing either of these properties, our set-boundary reachability
method only needs to perform reachability analysis on the boundary of the input
set. Moreover, for NNs that do not feature these properties with respect to the
input set, we explore subsets of the input set for establishing the local
homeomorphism property and then abandon these subsets for reachability
computations. Finally, some examples demonstrate the performance of the
proposed method.Comment: 25 pages, 11 figures. arXiv admin note: substantial text overlap with
arXiv:2210.0417
A Review of Formal Methods applied to Machine Learning
We review state-of-the-art formal methods applied to the emerging field of
the verification of machine learning systems. Formal methods can provide
rigorous correctness guarantees on hardware and software systems. Thanks to the
availability of mature tools, their use is well established in the industry,
and in particular to check safety-critical applications as they undergo a
stringent certification process. As machine learning is becoming more popular,
machine-learned components are now considered for inclusion in critical
systems. This raises the question of their safety and their verification. Yet,
established formal methods are limited to classic, i.e. non machine-learned
software. Applying formal methods to verify systems that include machine
learning has only been considered recently and poses novel challenges in
soundness, precision, and scalability.
We first recall established formal methods and their current use in an
exemplar safety-critical field, avionic software, with a focus on abstract
interpretation based techniques as they provide a high level of scalability.
This provides a golden standard and sets high expectations for machine learning
verification. We then provide a comprehensive and detailed review of the formal
methods developed so far for machine learning, highlighting their strengths and
limitations. The large majority of them verify trained neural networks and
employ either SMT, optimization, or abstract interpretation techniques. We also
discuss methods for support vector machines and decision tree ensembles, as
well as methods targeting training and data preparation, which are critical but
often neglected aspects of machine learning. Finally, we offer perspectives for
future research directions towards the formal verification of machine learning
systems
How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
Context: Machine Learning (ML) has been at the heart of many innovations over
the past years. However, including it in so-called 'safety-critical' systems
such as automotive or aeronautic has proven to be very challenging, since the
shift in paradigm that ML brings completely changes traditional certification
approaches.
Objective: This paper aims to elucidate challenges related to the
certification of ML-based safety-critical systems, as well as the solutions
that are proposed in the literature to tackle them, answering the question 'How
to Certify Machine Learning Based Safety-critical Systems?'.
Method: We conduct a Systematic Literature Review (SLR) of research papers
published between 2015 to 2020, covering topics related to the certification of
ML systems. In total, we identified 217 papers covering topics considered to be
the main pillars of ML certification: Robustness, Uncertainty, Explainability,
Verification, Safe Reinforcement Learning, and Direct Certification. We
analyzed the main trends and problems of each sub-field and provided summaries
of the papers extracted.
Results: The SLR results highlighted the enthusiasm of the community for this
subject, as well as the lack of diversity in terms of datasets and type of
models. It also emphasized the need to further develop connections between
academia and industries to deepen the domain study. Finally, it also
illustrated the necessity to build connections between the above mention main
pillars that are for now mainly studied separately.
Conclusion: We highlighted current efforts deployed to enable the
certification of ML based software systems, and discuss some future research
directions.Comment: 60 pages (92 pages with references and complements), submitted to a
journal (Automated Software Engineering). Changes: Emphasizing difference
traditional software engineering / ML approach. Adding Related Works, Threats
to Validity and Complementary Materials. Adding a table listing papers
reference for each section/subsection