Fast derivatives of likelihood functionals for ODE based models using adjoint-state method

We consider time series data modeled by ordinary differential equations (ODEs), widespread models in physics, chemistry, biology and science in general. The sensitivity analysis of such dynamical systems usually requires calculation of various derivatives with respect to the model parameters. We employ the adjoint state method (ASM) for efficient computation of the first and the second derivatives of likelihood functionals constrained by ODEs with respect to the parameters of the underlying ODE model. Essentially, the gradient can be computed with a cost (measured by model evaluations) that is independent of the number of the ODE model parameters and the Hessian with a linear cost in the number of the parameters instead of the quadratic one. The sensitivity analysis becomes feasible even if the parametric space is high-dimensional. The main contributions are derivation and rigorous analysis of the ASM in the statistical context, when the discrete data are coupled with the continuous ODE model. Further, we present a highly optimized implementation of the results and its benchmarks on a number of problems. The results are directly applicable in (e.g.) maximum-likelihood estimation or Bayesian sampling of ODE based statistical models, allowing for faster, more stable estimation of parameters of the underlying ODE model.Comment: 5 figure

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently based on second order derivatives of a layer-wise error function with respect to the corresponding parameters. We prove that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer. Therefore, there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. We conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of our pruning method compared with several state-of-the-art baseline methods

Delineating Parameter Unidentifiabilities in Complex Models

Scientists use mathematical modelling to understand and predict the properties of complex physical systems. In highly parameterised models there often exist relationships between parameters over which model predictions are identical, or nearly so. These are known as structural or practical unidentifiabilities, respectively. They are hard to diagnose and make reliable parameter estimation from data impossible. They furthermore imply the existence of an underlying model simplification. We describe a scalable method for detecting unidentifiabilities, and the functional relations defining them, for generic models. This allows for model simplification, and appreciation of which parameters (or functions thereof) cannot be estimated from data. Our algorithm can identify features such as redundant mechanisms and fast timescale subsystems, as well as the regimes in which such approximations are valid. We base our algorithm on a novel quantification of regional parametric sensitivity: multiscale sloppiness. Traditionally, the link between parametric sensitivity and the conditioning of the parameter estimation problem is made locally, through the Fisher Information Matrix. This is valid in the regime of infinitesimal measurement uncertainty. We demonstrate the duality between multiscale sloppiness and the geometry of confidence regions surrounding parameter estimates made where measurement uncertainty is non-negligible. Further theoretical relationships are provided linking multiscale sloppiness to the Likelihood-ratio test. From this, we show that a local sensitivity analysis (as typically done) is insufficient for determining the reliability of parameter estimation, even with simple (non)linear systems. Our algorithm provides a tractable alternative. We finally apply our methods to a large-scale, benchmark Systems Biology model of NF-$\kappa$B, uncovering previously unknown unidentifiabilities

Shape Hessian for generalized Oseen flow by differentiability of a minimax: A Lagrangian approach

summary:The goal of this paper is to compute the shape Hessian for a generalized Oseen problem with nonhomogeneous Dirichlet boundary condition by the velocity method. The incompressibility will be treated by penalty approach. The structure of the shape gradient and shape Hessian with respect to the shape of the variable domain for a given cost functional are established by an application of the Lagrangian method with function space embedding technique

Parametric Level Set Methods for Inverse Problems

In this paper, a parametric level set method for reconstruction of obstacles in general inverse problems is considered. General evolution equations for the reconstruction of unknown obstacles are derived in terms of the underlying level set parameters. We show that using the appropriate form of parameterizing the level set function results a significantly lower dimensional problem, which bypasses many difficulties with traditional level set methods, such as regularization, re-initialization and use of signed distance function. Moreover, we show that from a computational point of view, low order representation of the problem paves the path for easier use of Newton and quasi-Newton methods. Specifically for the purposes of this paper, we parameterize the level set function in terms of adaptive compactly supported radial basis functions, which used in the proposed manner provides flexibility in presenting a larger class of shapes with fewer terms. Also they provide a "narrow-banding" advantage which can further reduce the number of active unknowns at each step of the evolution. The performance of the proposed approach is examined in three examples of inverse problems, i.e., electrical resistance tomography, X-ray computed tomography and diffuse optical tomography

Errors in quantum optimal control and strategy for the search of easily implementable control pulses

We introduce a new approach to assess the error of control problems we aim to optimize. The method offers a strategy to define new control pulses that are not necessarily optimal but still able to yield an error not larger than some fixed a priori threshold, and therefore provide control pulses that might be more amenable for an experimental implementation. The formalism is applied to an exactly solvable model and to the Landau-Zener model, whose optimal control problem is solvable only numerically. The presented method is of importance for applications where a high degree of controllability of the dynamics of quantum systems is required.Comment: 13 pages, 3 figure

On the Second-Order Shape Derivative of the Kohn-Vogelius Objective Functional Using the Velocity Method

The exterior Bernoulli free boundary problem was studied via shape optimization technique. The problem was reformulated into the minimization of the so-called Kohn-Vogelius objective functional, where two state variables involved satisfy two boundary value problems, separately. The paper focused on solving the second-order shape derivative of the objective functional using the velocity method with nonautonomous velocity fields. This work confirms the classical results of Delfour and Zolésio in relating shape derivatives of functionals using velocity method and perturbation of identity technique