Compiler Support for Operator Overloading and Algorithmic Differentiation in C++

Multiphysics software needs derivatives for, e.g., solving a system of non-linear equations, conducting model verification, or sensitivity studies. In C++, algorithmic differentiation (AD), based on operator overloading (overloading), can be used to calculate derivatives up to machine precision. To that end, the built-in floating-point type is replaced by the user-defined AD type. It overloads all required operators, and calculates the original value and the corresponding derivative based on the chain rule of calculus. While changing the underlying type seems straightforward, several complications arise concerning software and performance engineering. This includes (1) fundamental language restrictions of C++ w.r.t. user-defined types, (2) type correctness of distributed computations with the Message Passing Interface (MPI) library, and (3) identification and mitigation of AD induced overheads. To handle these issues, AD experts may spend a significant amount of time to enhance a code with AD, verify the derivatives and ensure optimal application performance. Hence, in this thesis, we propose a modern compiler-based tooling approach to support and accelerate the AD-enhancement process of C++ target codes. In particular, we make contributions to three aspects of AD. The initial type change - While the change to the AD type in a target code is conceptually straightforward, the type change often leads to a multitude of compiler error messages. This is due to the different treatment of built-in floating-point types and user-defined types by the C++ language standard. Previously legal code constructs in the target code subsequently violate the language standard when the built-in floating-point type is replaced with a user-defined AD type. We identify and classify these problematic code constructs and their root cause is shown. Solutions by localized source transformation are proposed. To automate this rather mechanical process, we develop a static code analyser and source transformation tool, called OO-Lint, based on the Clang compiler framework. It flags instances of these problematic code constructs and applies source transformations to make the code compliant with the requirements of the language standard. To show the overall relevance of complications with user-defined types, OO-Lint is applied to several well-known scientific codes, some of which have already been AD enhanced by others. In all of these applications, except the ones manually treated for AD overloading, problematic code constructs are detected. Type correctness of MPI communication - MPI is the de-facto standard for programming high performance, distributed applications. At the same time, MPI has a complex interface whose usage can be error-prone. For instance, MPI derived data types require manual construction by specifying memory locations of the underlying data. Specifying wrong offsets can lead to subtle bugs that are hard to detect. In the context of AD, special libraries exist that handle the required derivative book-keeping by replacing the MPI communication calls with overloaded variants. However, on top of the AD type change, the MPI communication routines have to be changed manually. In addition, the AD type fundamentally changes memory layout assumptions as it has a different extent than the built-in types. Previously legal layout assumptions have, thus, to be reverified. As a remedy, to detect any type-related errors, we developed a memory sanitizer tool, called TypeART, based on the LLVM compiler framework and the MPI correctness checker MUST. It tracks all memory allocations relevant to MPI communication to allow for checking the underlying type and extent of the typeless memory buffer address passed to any MPI routine. The overhead induced by TypeART w.r.t. several target applications is manageable. AD domain-specific profiling - Applying AD in a black-box manner, without consideration of the target code structure, can have a significant impact on both runtime and memory consumption. An AD expert is usually required to apply further AD-related optimizations for the reduction of these induced overheads. Traditional profiling techniques are, however, insufficient as they do not reveal any AD domain-specific metrics. Of interest for AD code optimization are, e.g., specific code patterns, especially on a function level, that can be treated efficiently with AD. To that end, we developed a static profiling tool, called ProAD, based on the LLVM compiler framework. For each function, it generates the computational graph based on the static data flow of the floating-point variables. The framework supports pattern analysis on the computational graph to identify the optimal application of the chain rule. We show the potential of the optimal application of AD with two case studies. In both cases, significant runtime improvements can be achieved when the knowledge of the code structure, provided by our tool, is exploited. For instance, with a stencil code, a speedup factor of about 13 is achieved compared to a naive application of AD and a factor of 1.2 compared to hand-written derivative code

Automatic Source Transformation and Performance Evaluation for Optimized Utilization of the Matlab Runtime System

The ADiMat software is a tool that offers Automatic Differentiation of any Matlab function using a hybrid approach. Depending on the configuration, the resulting derivative function is executed using a derivative container class. Earlier performance tests showed a weakness that was attributed to the object oriented class system of Matlab. In a first step to validate this assumption, the Matlab runtime environment is tested regarding function and method call overheads as well as property access overhead with Matlabs objects. Furthermore, using a simple test function, a specific derivative class is compared to a set of manually created Matlab functions offering the same functionality as the class. It was shown that the resulting speedup is substantial. An automatic method, transforming the XML based abstract syntax tree created with the existing ADiMat toolchain, was developed. The process completely removes the derivative object usage from any derivative function created with ADiMat. It is implemented as a set of XSLT stylesheets. The removal of object orientation works by resolving the class operators and methods statically. The necessary type inference process is based on the identification of the prefix of all derivative variables introduced by the differentiation process. Additionally, the respective classes are semi automatically transformed to a set of functions based on heuristics. A final performance assessment utilizing a PDE function is done comparing the benefits of the implemented transformation. It was shown that the improvement depend on the way the classes store the directional derivatives

OO-Lint for Operator Overloading in C++

OO-Lint is a Clang tool which examines C++ source programs, detecting implicit conversions of variables and other code patterns likely to cause errors in the use of operator overloading

Automatic Differentiation for Matlab With ADiMat

The ADiMat software is a tool that offers automatic differentiation of any Matlab function using a hybrid approach combining source transformation and operator overloading. We give a brief overview of ADiMat and its functionality and recent performance improvements

Source Transformation for the Optimized Utilization of the Matlab Runtime System for Automatic Differentiation

The ADiMat software is a tool that offers automatic differentiation of Matlab functions using a hybrid approach that combines source transformation and operator overloading. Performance tests showed a weakness with a derivative class of ADiMat. In a first step, the Matlab runtime environment is tested regarding function and method call overheads as well as property access overhead with Matlabs objects. An automatic method, transforming the XML based abstract syntax tree created with ADiMats toolchain through a set of stylesheets, was developed. The process completely removes the derivative object usage from any derivative function created with ADiMat. As a result, performance is improved considerable depending on the data container storing the derivative directions

Source Transformation of C++ Codes for Compatibility with Operator Overloading

In C++, new features and semantics can be added to an existing software package without sweeping code changes by introducing a user-defined type using operator overloading. This approach is used, for example, to add capabilities such as algorithmic differentiation. However, the introduction of operator overloading can cause a multitude of compilation errors. In a previous paper, we identified code constructs that cause a violation of the C++ language standard after a type change, and a tool called OO-Lint based on the Clang compiler that identifies these code constructs with lint-like messages. In this paper, we present an extension of this work that automatically transforms such problematic code constructs in order to make an existing code base compatible with a semantic augmentation through operator overloading. We applied our tool to the CFD software OpenFOAM and detected and transformed 23 instances of problematic code constructs in 160,000 lines of code. A significant amount of these root causes are included up to 425 times in other files causing a tremendous compiler error amplification. In addition, we show the significance of our work with a case study of the evolution of the ice flow modeling software ISSM, comparing a recent version which was manually type changed with a legacy version. The recent version shows no signs of problematic code constructs. In contrast, our tool detected and transformed a remarkable amount of issues in the legacy version that previously had to be manually located and fixed

Checking C++ Codes for Compatibility with Operator Overloading

Operator overloading allows the semantic extension of existing code without the need for sweeping code changes. For example, automatic differentiation tools in C++ commonly use this feature to enhance the code with additional derivative computation. To this end, a floating point data type is changed to a complex user-defined type. While conceptually straightforward, this type change often leads to compilation errors that can be tedious to decipher and resolve. This is due to the fact that the built-in floating point types in C++ are treated differently than user-defined types, and code constructs that are legal for floating point types can be a violation of the C++ standard for complex user-defined types. We identify and classify such problematic code constructs and suggest how the code can be changed to avoid these errors, while still allowing the use of operator overloading. To automatically flag such occurrences, we developed a Clang-based tool for the static analysis of C++ code based on our assessment of constructs problematic in operator overloading for numeric types. It automatically finds instances of problematic code locations and prints Lint-like warning messages. To showcase the relevance of this topic and the usefulness of our tool, we consider the basic routines of the OpenFOAM CFD software package, consisting of 1,476 C++ source and header files, for a total of over 150,000 lines of code. Altogether, we found 74 distinct occurrences of problematic code constructs in 21 files. As some of these files are included in over 400 different locations in the OpenFOAM base, errors in these files create a torrent of error messages that often are difficult to comprehend. In summary, the classification of problematic instances aids developers in writing numerical code that is fit for operator overloading and the tool helps programmers that augment legacy code in spotting problematic code constructs

Source Transformation for the Optimized Utilization of the Matlab Runtime System for Automatic Differentiation

Operator overloading in Matlab allows for user-defined types to semantically augment existing Matlab codes without changes. However, given sufficient knowledge about types and operand semantics, operator overloading can be replaced by equivalent function calls. The ADiMat software tool provides automatic differentiation of Matlab functions using a hybrid approach that combines source transformation and operator overloading. It can also be used as a general framework for user-defined transformations of Matlab codes. Tests showed the potential for performance improvement in a derivative class providing essential linear algebra functionality for ADiMat. The Matlab runtime environment was benchmarked regarding function and method call overheads as well as property access overhead with Matlab's objects. These tests identify the cell class property access as the main performance culprit. Hence, an automatic method, transforming the XML based abstract syntax tree created with ADiMat's toolchain through a set of stylesheets, was developed. This process completely removes the derivative object usage and hence the opreator overloading and the property access overhead from any derivative function created with ADiMat. Experimental results show that performance is improved considerably depending on the data container storing the derivative directions