Backpropagation for long sequences: beyond memory constraints with constant overheads

Naive backpropagation through time has a memory footprint that grows linearly in the sequence length, due to the need to store each state of the forward propagation. This is a problem for large networks. Strategies have been developed to trade memory for added computations, which results in a sublinear growth of memory footprint or computation overhead. In this work, we present a library that uses asynchronous storing and prefetching to move data to and from slow and cheap stor- age. The library only stores and prefetches states as frequently as possible without delaying the computation, and uses the optimal Revolve backpropagation strategy for the computations in between. The memory footprint of the backpropagation can thus be reduced to any size (e.g. to fit into DRAM), while the computational overhead is constant in the sequence length, and only depends on the ratio between compute and transfer times on a given hardware. We show in experiments that by exploiting asyncronous data transfer, our strategy is always at least as fast, and usually faster than the previously studied "optimal" strategies

Cartan-Preserving Automorphisms and the Weyl Group of Kac-Moody Algebras

The group Ñ defined as the normalizer of the Cartan subalgebra in the group of all (inner and outer) automorphisms of affine Kac-Moody (KM) algebras is shown to play a fundamental role in the structure of these algebras. It is a (discrete) Galilean group which incorporates the affine and Weyl group structure of the KM algebra and the space-time structure of the bosonic string. It links the Virasoro and KM algebras in a non-trivial way and it plays a key role in the ‘vertex’ construction

On caustics associated with the linearized vorticity equation

summary:The linearized vorticity equation serves to model a number of wave phenomena in geophysical fluid dynamics. One technique that has been applied to this equation is the geometrical optics, or multi-dimensional WKB technique. Near caustics, this technique does not apply. A related technique that does apply near caustics is the Lagrange Manifold Formalism. Here we apply the Lagrange Manifold Formalism to determine an asymptotic solution of the linearized vorticity equation and to study associated wave phenomena on the caustic curve

Optimal Moments for the Analysis of Peculiar Velocity Surveys

We present a new method for the analysis of peculiar velocity surveys which removes contributions to velocities from small scale, nonlinear velocity modes while retaining information about large scale motions. Our method utilizes Karhunen--Lo\`eve methods of data compression to construct a set of moments out of the velocities which are minimally sensitive to small scale power. The set of moments are then used in a likelihood analysis. We develop criteria for the selection of moments, as well as a statistic to quantify the overall sensitivity of a set of moments to small scale power. Although we discuss our method in the context of peculiar velocity surveys, it may also prove useful in other situations where data filtering is required.Comment: 25 Pages, 3 figures. Submitted to Ap

Implications of surface noise for the motional coherence of trapped ions

Electric noise from metallic surfaces is a major obstacle towards quantum applications with trapped ions due to motional heating of the ions. Here, we discuss how the same noise source can also lead to pure dephasing of motional quantum states. The mechanism is particularly relevant at small ion-surface distances, thus imposing a new constraint on trap miniaturization. By means of a free induction decay experiment, we measure the dephasing time of the motion of a single ion trapped 50~$\mu$m above a Cu-Al surface. From the dephasing times we extract the integrated noise below the secular frequency of the ion. We find that none of the most commonly discussed surface noise models for ion traps describes both, the observed heating as well as the measured dephasing, satisfactorily. Thus, our measurements provide a benchmark for future models for the electric noise emitted by metallic surfaces.Comment: (5 pages, 4 figures

Calculation of a line list for MnH for brown dwarf and exoplanetary applications

otherPeer reviewe

Lightweight pressurization system for the Apollo service propulsion system Final report

Lightweight cryogenic He cascade pressurization concept for Apollo service propulsion syste

Observations on the vibration of axially-tensioned elastomeric pipes conveying fluids

A study of the effect of axial tension on the vibration of a single-span elastomeric pipe clamped at both ends conveying fluid has been carried out both experimentally and theoretically. A new mathematical model using a penalty function technique and the method of kinematic correction and fictitious loads has been developed. The influence of flowing fluid and axial tension on natural frequencies and mode shapes of the system has been described using this model and compared with experimental observations. Linear and non-linear dynamic response of the harmonically excited pipe has also been investigated for varying flow velocities and initial axial tensions