A Hitchhiker's Guide to Automatic Differentiation

This article provides an overview of some of the mathematical prin- ciples of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator, and explain why they all reduce to the same actual chain of computations. We further give a short mathematical description of some methods of higher-order Forward AD and, at the end of this paper, brie y describe the Reverse Mode of Automatic Differentiation

A Hitchhiker's Guide to Automatic Differentiation

This article provides an overview of some of the mathematical prin- ciples of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator, and explain why they all reduce to the same actual chain of computations. We further give a short mathematical description of some methods of higher-order Forward AD and, at the end of this paper, brie y describe the Reverse Mode of Automatic Differentiation

Fluctuation-Response Relations for Multi-Time Correlations

We show that time-correlation functions of arbitrary order for any random variable in a statistical dynamical system can be calculated as higher-order response functions of the mean history of the variable. The response is to a ``control term'' added as a modification to the master equation for statistical distributions. The proof of the relations is based upon a variational characterization of the generating functional of the time-correlations. The same fluctuation-response relations are preserved within moment-closures for the statistical dynamical system, when these are constructed via the variational Rayleigh-Ritz procedure. For the 2-time correlations of the moment-variables themselves, the fluctuation-response relation is equivalent to an ``Onsager regression hypothesis'' for the small fluctuations. For correlations of higher-order, there is a new effect in addition to such linear propagation of fluctuations present instantaneously: the dynamical generation of correlations by nonlinear interaction of fluctuations. In general, we discuss some physical and mathematical aspects of the {\it Ans\"{a}tze} required for an accurate calculation of the time correlations. We also comment briefly upon the computational use of these relations, which is well-suited for automatic differentiation tools. An example will be given of a simple closure for turbulent energy decay, which illustrates the numerical application of the relations.Comment: 28 pages, 1 figure, submitted to Phys. Rev.

Study of automatic differentiation in topology optimization

This bachelor final thesis presents a study on the integration of automatic differentiation functions into basic topology optimisation algorithms to improve not only computation speed, but also efficiency and accuracy. The main goal is to develop a fully functional automatic differentiation script capable of deriving topological expression, linear or non linear ones, aiming to find the optimal distribution. Moreover, the objectives of this research are to explore the application of automatic differentiation in fields related to topology optimisation, analyse the benefits of applying this methods and its disadvantages and review its computational efficiency. The thesis begins by introducing the fundamentals of automatic differentiation. Beforehand, during the initial stages of the project extensive practice of Matlab programming and object oriented programming was taught but it is not included in this report. A literature review is conducted to examine existing studies and approaches that utilize AD techniques. Furthermore, we dig into different AD methods, including forward mode and reverse mode, highlighting its approach with Matlab language and their advantages and limitations. Additionally, specific topologic optimisation tools and software packages commonly used are reviewed but are not included in this report. The report continues by presenting the different discretisation cases in finite element method and developing an AD based case to solve the discretisation of different 2 dimension problems. Deriving its shape functions and testing AD for a future, more sophisticated, topological problem. To achieve the main objective of performing, at least, one topology case using automatic differentiation, an AD based algorithm using different iterative methods is developed and implemented. The algorithm is tested with basic shapes and problems to be improved and more efficient, including all the possible casuistry of a mathematical expression. Ending with the research, we test different topological cases using different iterative methods. One of these methods, newton iteration method, will need an improvement to higher order gradients of the automatic differentiation algorithm. With this improvement we will test and compare both methods for several cases to conclude about the efficiency, accuracy and computing time of both iterative methods and automatic differentiation algorithm applied to topological problems. The results of the study demonstrate that automatic differentiation significantly enhances the efficiency and accuracy of topological optimisation for a certain type of problems. For these cases, AD exhibits faster convergence, improved accuracy in gradient computation, and reduced computational time. Moreover, the AD-based approach proves to be robust and applicable to not only deriving structural function, but also different problem domains, highlighting its versatility and practicality. Overall, The research highlights the potential of AD in many fields

Automatic differentiation in machine learning: a survey

Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.Comment: 43 pages, 5 figure