Deep learning as optimal control problems: Models and numerical methods

We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretisation. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. The differential equation setting lends itself to learning additional parameters such as the time discretisation. We explore this extension alongside natural constraints (e.g. time steps lie in a simplex). We compare these deep learning algorithms numerically in terms of induced flow and generalisation ability

Deep learning as optimal control problems

We briefly review recent work where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We report here new preliminary experiments with implicit symplectic Runge-Kutta methods. In this paper, we discuss ongoing and future research in this area

Structure-preserving deep learning

Over the past few years, deep learning has risen to the foreground as a topic of massive interest, mainly as a result of successes obtained in solving large-scale image processing tasks. There are multiple challenging mathematical problems involved in applying deep learning: most deep learning methods require the solution of hard optimisation problems, and a good understanding of the tradeoff between computational effort, amount of data and model complexity is required to successfully design a deep learning approach for a given problem. A large amount of progress made in deep learning has been based on heuristic explorations, but there is a growing effort to mathematically understand the structure in existing deep learning methods and to systematically design new deep learning methods to preserve certain types of structure in deep learning. In this article, we review a number of these directions: some deep neural networks can be understood as discretisations of dynamical systems, neural networks can be designed to have desirable properties such as invertibility or group equivariance, and new algorithmic frameworks based on conformal Hamiltonian systems and Riemannian manifolds to solve the optimisation problems have been proposed. We conclude our review of each of these topics by discussing some open problems that we consider to be interesting directions for future research

On post-Lie algebras, Lie--Butcher series and moving frames

Pre-Lie (or Vinberg) algebras arise from flat and torsion-free connections on differential manifolds. They have been studied extensively in recent years, both from algebraic operadic points of view and through numerous applications in numerical analysis, control theory, stochastic differential equations and renormalization. Butcher series are formal power series founded on pre-Lie algebras, used in numerical analysis to study geometric properties of flows on euclidean spaces. Motivated by the analysis of flows on manifolds and homogeneous spaces, we investigate algebras arising from flat connections with constant torsion, leading to the definition of post-Lie algebras, a generalization of pre-Lie algebras. Whereas pre-Lie algebras are intimately associated with euclidean geometry, post-Lie algebras occur naturally in the differential geometry of homogeneous spaces, and are also closely related to Cartan's method of moving frames. Lie--Butcher series combine Butcher series with Lie series and are used to analyze flows on manifolds. In this paper we show that Lie--Butcher series are founded on post-Lie algebras. The functorial relations between post-Lie algebras and their enveloping algebras, called D-algebras, are explored. Furthermore, we develop new formulas for computations in free post-Lie algebras and D-algebras, based on recursions in a magma, and we show that Lie--Butcher series are related to invariants of curves described by moving frames.Comment: added discussion of post-Lie algebroid

Roaring high and low: composition and possible functions of the Iberian stag's vocal repertoire

We provide a detailed description of the rutting vocalisations of free-ranging male Iberian deer (Cervus elaphus hispanicus, Hilzheimer 1909), a geographically isolated and morphologically differentiated subspecies of red deer Cervus elaphus. We combine spectrographic examinations, spectral analyses and automated classifications to identify different call types, and compare the composition of the vocal repertoire with that of other red deer subspecies. Iberian stags give bouts of roars (and more rarely, short series of barks) that are typically composed of two different types of calls. Long Common Roars are mostly given at the beginning or at the end of the bout, and are characterised by a high fundamental frequency (F0) resulting in poorly defined formant frequencies but a relatively high amplitude. In contrast, Short Common Roars are typically given in the middle or at the end of the bout, and are characterised by a lower F0 resulting in relatively well defined vocal tract resonances, but low amplitude. While we did not identify entirely Harsh Roars (as described in the Scottish red deer subspecies (Cervus elaphus scoticus), a small percentage of Long Common Roars contained segments of deterministic chaos. We suggest that the evolution of two clearly distinct types of Common Roars may reflect divergent selection pressures favouring either vocal efficiency in high pitched roars or the communication of body size in low-pitched, high spectral density roars highlighting vocal tract resonances. The clear divergence of the Iberian red deer vocal repertoire from those of other documented European red deer populations reinforces the status of this geographical variant as a distinct subspecies

Do red deer stags (Cervus elaphus) use roar fundamental frequency (F0) to assess rivals?

It is well established that in humans, male voices are disproportionately lower pitched than female voices, and recent studies suggest that this dimorphism in fundamental frequency (F0) results from both intrasexual (male competition) and intersexual (female mate choice) selection for lower pitched voices in men. However, comparative investigations indicate that sexual dimorphism in F0 is not universal in terrestrial mammals. In the highly polygynous and sexually dimorphic Scottish red deer Cervus elaphus scoticus, more successful males give sexually-selected calls (roars) with higher minimum F0s, suggesting that high, rather than low F0s advertise quality in this subspecies. While playback experiments demonstrated that oestrous females prefer higher pitched roars, the potential role of roar F0 in male competition remains untested. Here we examined the response of rutting red deer stags to playbacks of re-synthesized male roars with different median F0s. Our results show that stags’ responses (latencies and durations of attention, vocal and approach responses) were not affected by the F0 of the roar. This suggests that intrasexual selection is unlikely to strongly influence the evolution of roar F0 in Scottish red deer stags, and illustrates how the F0 of terrestrial mammal vocal sexual signals may be subject to different selection pressures across species. Further investigations on species characterized by different F0 profiles are needed to provide a comparative background for evolutionary interpretations of sex differences in mammalian vocalizations

Statistical Computing on Non-Linear Spaces for Computational Anatomy

International audienceComputational anatomy is an emerging discipline that aims at analyzing and modeling the individual anatomy of organs and their biological variability across a population. However, understanding and modeling the shape of organs is made difficult by the absence of physical models for comparing different subjects, the complexity of shapes, and the high number of degrees of freedom implied. Moreover, the geometric nature of the anatomical features usually extracted raises the need for statistics on objects like curves, surfaces and deformations that do not belong to standard Euclidean spaces. We explain in this chapter how the Riemannian structure can provide a powerful framework to build generic statistical computing tools. We show that few computational tools derive for each Riemannian metric can be used in practice as the basic atoms to build more complex generic algorithms such as interpolation, filtering and anisotropic diffusion on fields of geometric features. This computational framework is illustrated with the analysis of the shape of the scoliotic spine and the modeling of the brain variability from sulcal lines where the results suggest new anatomical findings