ReLEx: Regularisation for Linear Extrapolation in Neural Networks with Rectified Linear Units

Despite the great success of neural networks in recent years, they are not providing useful extrapolation. In regression tasks, the popular Rectified Linear Units do enable unbounded linear extrapolation by neural networks, but their extrapolation behaviour varies widely and is largely independent of the training data. Our goal is instead to continue the local linear trend at the margin of the training data. Here we introduce ReLEx, a regularising method composed of a set of loss terms design to achieve this goal and reduce the variance of the extrapolation. We present a ReLEx implementation for single input, single output, and single hidden layer feed-forward networks. Our results demonstrate that ReLEx has little cost in terms of standard learning, i.e. interpolation, but enables controlled univariate linear extrapolation with ReLU neural networks

Calibration method to improve transfer from simulation to quadruped robots

Using passive compliance in robotic locomotion has been seen as a cheap and straightforward way of increasing the performance in energy consumption and robustness. However, the control for such systems remains quite challenging when using traditional robotic techniques. The progress in machine learning opens a horizon of new possibilities in this direction but the training methods are generally too long and laborious to be conducted on a real robot platform. On the other hand, learning a control policy in simulation also raises a lot of complication in the transfer. In this paper, we designed a cheap quadruped robot and detail a calibration method to optimize a simulation model in order to facilitate the transfer of parametric motor primitives. We present results validating the transfer of Central Pattern Generators (CPG) learned in simulation to the robot which already give positive insights on the validity of this method

Pharmacometric covariate modeling using symbolic regression networks

A central challenge within pharmacometrics is to establish a relation between pharmacological model parameters, such as compartment volumes and diffusion rate constants, and known population covariates, such as age and body mass. There is rich literature dedicated to the learning of functional mappings from the covariates to the model parameters, once a search class of functions has been determined. However, the state-of-the-art selection of the search class itself is ad hoc. We demonstrate how neural network-based symbolic regression can be used to simultaneously find the function form and its parameters. The method is put in relation to the literature on symbolic regression and equation learning. A conceptual demonstration is provided through examples, as is a road map to full-scale employment to pharmacological data sets, relevant to closed-loop anesthesia

Interpretable Scientific Discovery with Symbolic Regression: A Review

Symbolic regression is emerging as a promising machine learning method for learning succinct underlying interpretable mathematical expressions directly from data. Whereas it has been traditionally tackled with genetic programming, it has recently gained a growing interest in deep learning as a data-driven model discovery method, achieving significant advances in various application domains ranging from fundamental to applied sciences. This survey presents a structured and comprehensive overview of symbolic regression methods and discusses their strengths and limitations

Learning stable and predictive structures in kinetic systems: Benefits of a causal approach

Learning kinetic systems from data is one of the core challenges in many fields. Identifying stable models is essential for the generalization capabilities of data-driven inference. We introduce a computationally efficient framework, called CausalKinetiX, that identifies structure from discrete time, noisy observations, generated from heterogeneous experiments. The algorithm assumes the existence of an underlying, invariant kinetic model, a key criterion for reproducible research. Results on both simulated and real-world examples suggest that learning the structure of kinetic systems benefits from a causal perspective. The identified variables and models allow for a concise description of the dynamics across multiple experimental settings and can be used for prediction in unseen experiments. We observe significant improvements compared to well established approaches focusing solely on predictive performance, especially for out-of-sample generalization

Information Fusion via Symbolic Regression: A Tutorial in the Context of Human Health

This tutorial paper provides a general overview of symbolic regression (SR) with specific focus on standards of interpretability. We posit that interpretable modeling, although its definition is still disputed in the literature, is a practical way to support the evaluation of successful information fusion. In order to convey the benefits of SR as a modeling technique, we demonstrate an application within the field of health and nutrition using publicly available National Health and Nutrition Examination Survey (NHANES) data from the Centers for Disease Control and Prevention (CDC), fusing together anthropometric markers into a simple mathematical expression to estimate body fat percentage. We discuss the advantages and challenges associated with SR modeling and provide qualitative and quantitative analyses of the learned models