Investigation of Adjoint Based Shape Optimization Techniques in NASCART-GT using Automatic Reverse Differentiation

Automated shape optimization involves making suitable modifications to a geometry that can lead to significant improvements in aerodynamic performance. Currently available mid-fdelity Aerodynamic Optimizers cannot be utilized in the late stages of the design process for performing minor, but consequential, tweaks in geometry. Automated shape optimization involves making suitable modifications to a geometry that can lead to significant improvements in aerodynamic performance. Currently available mid-fidelity Aerodynamic Optimizers cannot be utilized in the late stages of the design process for performing minor, but consequential, tweaks in geometry. High-fidelity shape optimization techniques are explored which, even though computationally demanding, are invaluable since they can account for realistic effects like turbulence and viscocity. The high computational costs associated with the optimization have been avoided by using an indirect optimization approach, which was used to dcouple the effect of the flow field variables on the gradients involved. The main challenge while performing the optimization was to maintain low sensitivity to the number of input design variables. This necessitated the use of Reverse Automatic differentiation tools to generate the gradient. All efforts have been made to keep computational costs to a minimum, thereby enabling hi-fidelity optimization to be used even in the initial design stages. A preliminary roadmap has been laid out for an initial implementation of optimization algorithms using the adjoint approach, into the high fidelity CFD code NASCART-GT.High-fidelity shape optimization techniques are explored which, even though computationally demanding, are invaluable since they can account for realistic effects like turbulence and viscocity. The high computational costs associated with the optimization have been avoided by using an indirect optimization approach, which was used to dcouple the effect of the flow field variables on the gradients involved. The main challenge while performing the optimization was to maintain low sensitivity to the number of input design variables. This necessitated the use of Reverse Automatic differentiation tools to generate the gradient. All efforts have been made to keep computational costs to a minimum, thereby enabling hi-fidelity optimization to be used even in the initial design stages. A preliminary roadmap has been laid out for an initial implementation of optimization algorithms using the adjoint approach, into the high fidelity CFD code NASCART-GT.Ruffin, Stephen - Faculty Mentor ; Feron, Eric - Committee Member/Second Reader ; Sankar, Lakshmi - Committee Member/Second Reade

Automatic Differentiation of Algorithms for Machine Learning

Automatic differentiation---the mechanical transformation of numeric computer programs to calculate derivatives efficiently and accurately---dates to the origin of the computer age. Reverse mode automatic differentiation both antedates and generalizes the method of backwards propagation of errors used in machine learning. Despite this, practitioners in a variety of fields, including machine learning, have been little influenced by automatic differentiation, and make scant use of available tools. Here we review the technique of automatic differentiation, describe its two main modes, and explain how it can benefit machine learning practitioners. To reach the widest possible audience our treatment assumes only elementary differential calculus, and does not assume any knowledge of linear algebra.Comment: 7 pages, 1 figur

Small steps and giant leaps: Minimal Newton solvers for Deep Learning

We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement. Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, a procedure that is both costly and sensitive to noise. Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration. This estimate has the same size and is similar to the momentum variable that is commonly used in SGD. No estimate of the Hessian is maintained. We first validate our method, called CurveBall, on small problems with known closed-form solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers seem to struggle. We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning. Code is available