Automatic differentiation in machine learning: a survey

Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.Comment: 43 pages, 5 figure

Distributed-memory large deformation diffeomorphic 3D image registration

We present a parallel distributed-memory algorithm for large deformation diffeomorphic registration of volumetric images that produces large isochoric deformations (locally volume preserving). Image registration is a key technology in medical image analysis. Our algorithm uses a partial differential equation constrained optimal control formulation. Finding the optimal deformation map requires the solution of a highly nonlinear problem that involves pseudo-differential operators, biharmonic operators, and pure advection operators both forward and back- ward in time. A key issue is the time to solution, which poses the demand for efficient optimization methods as well as an effective utilization of high performance computing resources. To address this problem we use a preconditioned, inexact, Gauss-Newton- Krylov solver. Our algorithm integrates several components: a spectral discretization in space, a semi-Lagrangian formulation in time, analytic adjoints, different regularization functionals (including volume-preserving ones), a spectral preconditioner, a highly optimized distributed Fast Fourier Transform, and a cubic interpolation scheme for the semi-Lagrangian time-stepping. We demonstrate the scalability of our algorithm on images with resolution of up to $1024^3$ on the "Maverick" and "Stampede" systems at the Texas Advanced Computing Center (TACC). The critical problem in the medical imaging application domain is strong scaling, that is, solving registration problems of a moderate size of $256^3$---a typical resolution for medical images. We are able to solve the registration problem for images of this size in less than five seconds on 64 x86 nodes of TACC's "Maverick" system.Comment: accepted for publication at SC16 in Salt Lake City, Utah, USA; November 201

Reduction of the Random Access Memory Size in Adjoint Algorithmic Differentiation by Overloading

Adjoint algorithmic differentiation by operator and function overloading is based on the interpretation of directed acyclic graphs resulting from evaluations of numerical simulation programs. The size of the computer system memory required to store the graph grows proportional to the number of floating-point operations executed by the underlying program. It quickly exceeds the available memory resources. Naive adjoint algorithmic differentiation often becomes infeasible except for relatively simple numerical simulations. Access to the data associated with the graph can be classified as sequential and random. The latter refers to memory access patterns defined by the adjacency relationship between vertices within the graph. Sequentially accessed data can be decomposed into blocks. The blocks can be streamed across the system memory hierarchy thus extending the amount of available memory, for example, to hard discs. Asynchronous i/o can help to mitigate the increased cost due to accesses to slower memory. Much larger problem instances can thus be solved without resorting to technically challenging user intervention such as checkpointing. Randomly accessed data should not have to be decomposed. Its block-wise streaming is likely to yield a substantial overhead in computational cost due to data accesses across blocks. Consequently, the size of the randomly accessed memory required by an adjoint should be kept minimal in order to eliminate the need for decomposition. We propose a combination of dedicated memory for adjoint $L$-values with the exploitation of remainder bandwidth as a possible solution. Test results indicate significant savings in random access memory size while preserving overall computational efficiency

Development and Applications of Adjoint-Based Aerodynamic and Aeroacoustic Multidisciplinary Optimization for Rotorcraft

Urban Air Mobility (UAM) is one of the most popular proposed solutions for alleviating traffic problems in populated areas. In this context, the proposed types of vehicles mainly consist of rotors and propellers powered by electric motors. However, those rotary-wing components can contribute excessively to noise generation. Therefore, a significant noise concern emerges due to urban air vehicles in or around residential areas. Reducing noise emitted by air vehicles is critically important to improve public acceptance of such vehicles for operations in densely populated areas. Two main objectives of the present dissertation are: (1) to expand the multidisciplinary optimization to utilize adjoint-based aeroacoustic and aerodynamic sensitivities; (2) to optimize the shape of proprotor blades to improve the overall performance of selected rotorcraft from both aerodynamic and aeroacoustic perspectives. This dissertation reports on the development and application of an unsteady discrete adjoint solver for aerodynamic and aeroacoustic coupling to obtain an improved design for quieter rotorcraft. The optimization framework developed through this dissertation can be utilized for multiple flight conditions, multiple receivers, and multiple optimization objectives within the same design process. SU2-based code development involves the implementation of aeroacoustic analysis, adjoint computations, and integrations into a multidisciplinary rotorcraft optimization suite. A computational aeroacoustics tool is embedded into the SU2-suite to predict the propagation of the emitted noise from the moving sources with high fidelity. Capabilities of the developed computational aeroacoustics tool are demonstrated for a range of rotor, propeller, and proprotor applications, and they are verified by comparing with wind tunnel data whenever it is available. The aeroacoustic tool also computes sensitivities with respect to the conserved variables and grid coordinates by employing the algorithmic differentiation method. Integration of an acoustic solver into the discrete adjoint solver and related modifications enable the code to compute aeroacoustic sensitivities with respect to the design variables. Applying the developed optimization framework for a proprotor aims to reduce the noise radiation without sacrificing the required aerodynamic performance value. As an outcome of the optimization during forward-flight and hover, the reshaped blade design emits and propagates lower noise levels as perceived by multiple observers. The major contributions are: (1) a multidisciplinary optimization framework that presents an optimized rotorcraft design for better aeroacoustics and aerodynamics; (2) a novel adjoint-based formulation for aeroacoustic sensitivities with respect to design variables; (3) single acoustic objective function including multiple flight conditions and multiple microphone positions; (4) implementation of Farassat 1A formulation into opensource software, SU2, to compute noise propagation emitted from moving sources. In summary, this dissertation provides the results with high fidelity, a well-integrated and rapidly converging optimization tool to improve the rotorcraft\u27s aeroacoustic performance while retaining or improving the aerodynamic performance. Among the conclusions are the following: (1) Computational fluid dynamics analyses (SU2-CFD) can produce accurate results for various rotorcraft applications. (2) The developed aeroacoustic code predicts noise propagation emitted from propellers, rotors, and proprotors with high-fidelity. (3) The acoustic interaction between propeller and wing components can be assessed by employing the aeroacoustic solver. (4) The multidisciplinary optimization framework successively reduces noise level emitted by a proprotor in multiple flight configurations. (5) The optimized design improves emitted noise radiation while satisfying the given aerodynamic constraint(s)