73 research outputs found
Functorial String Diagrams for Reverse-Mode Automatic Differentiation
We formulate a reverse-mode automatic differentiation (RAD) algorithm for (applied) simply typed lambda calculus in the style of Pearlmutter and Siskind [Barak A. Pearlmutter and Jeffrey Mark Siskind, 2008], using the graphical formalism of string diagrams. Thanks to string diagram rewriting, we are able to formally prove for the first time the soundness of such an algorithm. Our approach requires developing a calculus of string diagrams with hierarchical features in the spirit of functorial boxes, in order to model closed monoidal (and cartesian closed) structure. To give an efficient yet principled implementation of the RAD algorithm, we use foliations of our hierarchical string diagrams
Smart Choices and the Selection Monad
Describing systems in terms of choices and their resulting costs and rewards
offers the promise of freeing algorithm designers and programmers from
specifying how those choices should be made; in implementations, the choices
can be realized by optimization techniques and, increasingly, by machine
learning methods. We study this approach from a programming-language
perspective. We define two small languages that support decision-making
abstractions: one with choices and rewards, and the other additionally with
probabilities. We give both operational and denotational semantics.
In the case of the second language we consider three denotational semantics,
with varying degrees of correlation between possible program values and
expected rewards. The operational semantics combine the usual semantics of
standard constructs with optimization over spaces of possible execution
strategies.
The denotational semantics, which are compositional and can also be viewed as
an implementation by translation to a simpler language, rely on the selection
monad, to handle choice, combined with an auxiliary monad, to handle other
effects such as rewards or probability.
We establish adequacy theorems that the two semantics coincide in all cases.
We also prove full abstraction at ground types, with varying notions of
observation in the probabilistic case corresponding to the various degrees of
correlation. We present axioms for choice combined with rewards and
probability, establishing completeness at ground types for the case of rewards
without probability
Differential logical relations, part II increments and derivatives
International audienceWe study the deep relations existing between differential logical relations and incremental computing, by showing how self-differences in the former precisely correspond to derivatives in the latter. We also show how differential logical relations can be seen as a powerful metatheoretical tool in the analysis of incremental computations, enabling an easy proof of soundness of differentiation
CHAD: Combinatory Homomorphic Automatic Differentiation
We introduce Combinatory Homomorphic Automatic Differentiation (CHAD), a
principled, pure, provably correct define-then-run method for performing
forward and reverse mode automatic differentiation (AD) on programming
languages with expressive features. It implements AD as a compositional,
type-respecting source-code transformation that generates purely functional
code. This code transformation is principled in the sense that it is the unique
homomorphic (structure preserving) extension to expressive languages of
Elliott's well-known and unambiguous definitions of AD for a first-order
functional language. Correctness of the method follows by a (compositional)
logical relations argument that shows that the semantics of the syntactic
derivative is the usual calculus derivative of the semantics of the original
program.
In their most elegant formulation, the transformations generate code with
linear types. However, the code transformations can be implemented in a
standard functional language lacking linear types: while the correctness proof
requires tracking of linearity, the actual transformations do not. In fact,
even in a standard functional language, we can get all of the type-safety that
linear types give us: we can implement all linear types used to type the
transformations as abstract types, by using a basic module system.
In this paper, we detail the method when applied to a simple higher-order
language for manipulating statically sized arrays. However, we explain how the
methodology applies, more generally, to functional languages with other
expressive features. Finally, we discuss how the scope of CHAD extends beyond
applications in AD to other dynamic program analyses that accumulate data in a
commutative monoid.Comment: arXiv admin note: substantial text overlap with arXiv:2007.0528
Higher Order Automatic Differentiation of Higher Order Functions
We present semantic correctness proofs of automatic differentiation (AD). We
consider a forward-mode AD method on a higher order language with algebraic
data types, and we characterise it as the unique structure preserving macro
given a choice of derivatives for basic operations. We describe a rich
semantics for differentiable programming, based on diffeological spaces. We
show that it interprets our language, and we phrase what it means for the AD
method to be correct with respect to this semantics. We show that our
characterisation of AD gives rise to an elegant semantic proof of its
correctness based on a gluing construction on diffeological spaces. We explain
how this is, in essence, a logical relations argument. Throughout, we show how
the analysis extends to AD methods for computing higher order derivatives using
a Taylor approximation.Comment: 34 pages, 5 figures, submitted at LMCS 2020. arXiv admin note:
substantial text overlap with arXiv:2001.0220
- …