22 research outputs found
Backprop as Functor: A compositional perspective on supervised learning
A supervised learning algorithm searches over a set of functions
parametrised by a space to find the best approximation to some ideal
function . It does this by taking examples , and updating the parameter according to some rule. We define a
category where these update rules may be composed, and show that gradient
descent---with respect to a fixed step size and an error function satisfying a
certain property---defines a monoidal functor from a category of parametrised
functions to this category of update rules. This provides a structural
perspective on backpropagation, as well as a broad generalisation of neural
networks.Comment: 13 pages + 4 page appendi
Categorical Foundations of Gradient-Based Learning
We propose a categorical semantics of gradient-based machine learning
algorithms in terms of lenses, parametrised maps, and reverse derivative
categories. This foundation provides a powerful explanatory and unifying
framework: it encompasses a variety of gradient descent algorithms such as
ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions
such as as MSE and Softmax cross-entropy, shedding new light on their
similarities and differences. Our approach to gradient-based learning has
examples generalising beyond the familiar continuous domains (modelled in
categories of smooth maps) and can be realized in the discrete setting of
boolean circuits. Finally, we demonstrate the practical significance of our
framework with an implementation in Python.Comment: 14 page
Space-time tradeoffs of lenses and optics via higher category theory
Optics and lenses are abstract categorical gadgets that model systems with
bidirectional data flow. In this paper we observe that the denotational
definition of optics - identifying two optics as equivalent by observing their
behaviour from the outside - is not suitable for operational, software oriented
approaches where optics are not merely observed, but built with their internal
setups in mind. We identify operational differences between denotationally
isomorphic categories of cartesian optics and lenses: their different
composition rule and corresponding space-time tradeoffs, positioning them at
two opposite ends of a spectrum. With these motivations we lift the existing
categorical constructions and their relationships to the 2-categorical level,
showing that the relevant operational concerns become visible. We define the
2-category whose 2-cells explicitly track
optics' internal configuration. We show that the 1-category
arises by locally quotienting out the connected
components of this 2-category. We show that the embedding of lenses into
cartesian optics gets weakened from a functor to an oplax functor whose
oplaxator now detects the different composition rule. We determine the
difficulties in showing this functor forms a part of an adjunction in any of
the standard 2-categories. We establish a conjecture that the well-known
isomorphism between cartesian lenses and optics arises out of the lax
2-adjunction between their double-categorical counterparts. In addition to
presenting new research, this paper is also meant to be an accessible
introduction to the topic.Comment: 28 page
Profunctor optics, a categorical update
Optics are bidirectional data accessors that capture data transformation patterns such as accessing subfields or iterating over containers. Profunctor optics are a particular choice of representation supporting modularity, meaning that we can construct accessors for complex structures by combining simpler ones. Profunctor optics have previously been studied only in an unenriched and non-mixed setting, in which both directions of access are modelled in the same category. However, functional programming languages are arguably better described by enriched categories; and we have found that some structures in the literature are actually mixed optics, with access directions modelled in different categories. Our work generalizes a classic result by Pastro and Street on Tambara theory and uses it to describe mixed V-enriched profunctor optics and to endow them with V-category structure. We provide some original families of optics and derivations, including an elementary one for traversals. Finally, we discuss a Haskell implementation