How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

Context: Machine Learning (ML) has been at the heart of many innovations over the past years. However, including it in so-called 'safety-critical' systems such as automotive or aeronautic has proven to be very challenging, since the shift in paradigm that ML brings completely changes traditional certification approaches. Objective: This paper aims to elucidate challenges related to the certification of ML-based safety-critical systems, as well as the solutions that are proposed in the literature to tackle them, answering the question 'How to Certify Machine Learning Based Safety-critical Systems?'. Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems. In total, we identified 217 papers covering topics considered to be the main pillars of ML certification: Robustness, Uncertainty, Explainability, Verification, Safe Reinforcement Learning, and Direct Certification. We analyzed the main trends and problems of each sub-field and provided summaries of the papers extracted. Results: The SLR results highlighted the enthusiasm of the community for this subject, as well as the lack of diversity in terms of datasets and type of models. It also emphasized the need to further develop connections between academia and industries to deepen the domain study. Finally, it also illustrated the necessity to build connections between the above mention main pillars that are for now mainly studied separately. Conclusion: We highlighted current efforts deployed to enable the certification of ML based software systems, and discuss some future research directions.Comment: 60 pages (92 pages with references and complements), submitted to a journal (Automated Software Engineering). Changes: Emphasizing difference traditional software engineering / ML approach. Adding Related Works, Threats to Validity and Complementary Materials. Adding a table listing papers reference for each section/subsection

A General Framework for Uncertainty Estimation in Deep Learning

Neural networks predictions are unreliable when the input sample is out of the training distribution or corrupted by noise. Being able to detect such failures automatically is fundamental to integrate deep learning algorithms into robotics. Current approaches for uncertainty estimation of neural networks require changes to the network and optimization process, typically ignore prior knowledge about the data, and tend to make over-simplifying assumptions which underestimate uncertainty. To address these limitations, we propose a novel framework for uncertainty estimation. Based on Bayesian belief networks and Monte-Carlo sampling, our framework not only fully models the different sources of prediction uncertainty, but also incorporates prior data information, e.g. sensor noise. We show theoretically that this gives us the ability to capture uncertainty better than existing methods. In addition, our framework has several desirable properties: (i) it is agnostic to the network architecture and task; (ii) it does not require changes in the optimization process; (iii) it can be applied to already trained architectures. We thoroughly validate the proposed framework through extensive experiments on both computer vision and control tasks, where we outperform previous methods by up to 23% in accuracy.Comment: Accepted for publication in the Robotics and Automation Letters 2020, and for presentation at the International Conference on Robotics and Automation (ICRA) 202

Neural representation in active inference: using generative models to interact with -- and understand -- the lived world

This paper considers neural representation through the lens of active inference, a normative framework for understanding brain function. It delves into how living organisms employ generative models to minimize the discrepancy between predictions and observations (as scored with variational free energy). The ensuing analysis suggests that the brain learns generative models to navigate the world adaptively, not (or not solely) to understand it. Different living organisms may possess an array of generative models, spanning from those that support action-perception cycles to those that underwrite planning and imagination; namely, from "explicit" models that entail variables for predicting concurrent sensations, like objects, faces, or people - to "action-oriented models" that predict action outcomes. It then elucidates how generative models and belief dynamics might link to neural representation and the implications of different types of generative models for understanding an agent's cognitive capabilities in relation to its ecological niche. The paper concludes with open questions regarding the evolution of generative models and the development of advanced cognitive abilities - and the gradual transition from "pragmatic" to "detached" neural representations. The analysis on offer foregrounds the diverse roles that generative models play in cognitive processes and the evolution of neural representation

From Bayesian principles to Bayesian processes

This thesis considers the free energy principle (FEP) and its corollary, active inference, which form an explanatory framework that prescribes a Bayesian interpretation of self-organizing systems. The FEP originated in the domain of neuroscience, where it underwrote a unified theory that described perception, action and learning as emerging from minimizing a single objective function - variational free energy. However, since its conception, the FEP has transcended into physics and pure mathematics. Here, it presents itself as a set of mathematical arguments culminating in an inferential interpretation of a specific class of systems. The result has fundamentally changed the epistemological status of the FEP, moving it from the world of empirical hypotheses to the unfalsifiable territory of mathematical equivalences and tautological constructions. While the FEP may present a historical development that further unravels the symmetries that govern the laws of (our own) physics, its growth has left a range of epistemological confusion. In the current thesis, we evaluate how to maneuver from the principles of the FEP to the processes it purportedly explains. We identify four key areas in which the FEP can inform empirical science: 1) The FEP can aid us in designing intelligent agents by providing novel functionals that respect inherent uncertainty in the environment. We demonstrate equivalences between active inference and reinforcement learning, offer a novel implementation of active inference that utilizes amortized inference, and show that the proposed algorithm enables efficient exploration while offering improved sample efficiency compared to modern reinforcement learning algorithms. 2) We describe how the FEP can help us understand the nature of representation in living systems. Specifically, we show how the normative aspects of the FEP promote learning representations oriented towards action rather than veridical reconstructions of the environment. 3) We show how the FEP provides a framework for modeling perception, action, and learning in systems that can be empirically measured. An eye-tracking study demonstrates that an active inference model best explains human information-seeking, offering insights into the underlying mechanisms of perception and action. 4) In the final section, we ask whether active inference can inform the development of novel process theories in computational neuroscience. A biologically-plausible learning algorithm is developed and verified on various computer vision and reinforcement learning tasks. The resulting model explains a range of empirical phenomena and offers a new perspective on the role of bottom-up information in perception. This thesis affirms the role of the FEP and active inference as a generative framework for developing testable scientific theories