6 research outputs found
Backpropagation for long sequences: beyond memory constraints with constant overheads
Naive backpropagation through time has a memory footprint that grows linearly in the sequence length, due to the need to store each state of the forward propagation. This is a problem for large networks. Strategies have been developed to trade memory for added computations, which results in a sublinear growth of memory footprint or computation overhead. In this work, we present a library that uses asynchronous storing and prefetching to move data to and from slow and cheap stor- age. The library only stores and prefetches states as frequently as possible without delaying the computation, and uses the optimal Revolve backpropagation strategy for the computations in between. The memory footprint of the backpropagation can thus be reduced to any size (e.g. to fit into DRAM), while the computational overhead is constant in the sequence length, and only depends on the ratio between compute and transfer times on a given hardware. We show in experiments that by exploiting asyncronous data transfer, our strategy is always at least as fast, and usually faster than the previously studied "optimal" strategies
Far-wake meandering induced by atmospheric eddies in flow past a wind turbine
A novel algorithm is developed to calculate the nonlinear optimal boundary perturbations in three-dimensional incompressible flow. An optimal step length in the optimisation loop is calculated without any additional calls to the Navier-Stokes equations. The algorithm is applied to compute the optimal in flow eddies for the flow around a wind turbine to clarify the mechanisms behind wake meandering, a phenomenon usually observed in wind farms. The turbine is modelled as an actuator disc using an immersed boundary method with the loading prescribed as a body force. At Reynolds number (based on free-stream velocity and turbine radius) Re = 1000, the most energetic inflow perturbation has a frequency ω = 0:8 ~ 2, and is in the form of an azimuthal wave with wavenumber m = 1 and the same radius as the actuator disc. The inflow perturbation is amplified by the strong shear downstream of the edge of the disc and then tilts the rolling-up vortex rings to induce wake meandering. This mechanism is verified by studying randomly perturbed flow at Re ≤ 8000. At five turbine diameters downstream of the disc, the axial velocity oscillates at a magnitude of more than 60% of the free-stream velocity when the magnitude of the inflow perturbation is 6% of the free-stream wind speed. The dominant Strouhal number of the wake oscillation is 0.16 at Re = 3000 and keeps approximately constant at higher Re. This Strouhal number agrees well with previous experimental findings. Overall the observations indicate that the well observed stochastic wake meandering phenomenon appearing far downstream of wind turbines is induced by large-scale (the same order as the turbine rotor) and low-frequency free-stream eddies
H-Revolve: A Framework for Adjoint Computation on Synchronous Hierarchical Platforms
International audienceWe study the problem of checkpointing strategies for adjoint computation on synchronous hierarchicalplatforms, specifically computational platforms with several levels of storage with different writing andreading costs. When reversing a large adjoint chain, choosing which data to checkpoint and where is a criticaldecision for the overall performance of the computation. We introduce H-Revolve, an optimal algorithm forthis problem. We make it available in a public Python library along with the implementation of several state-of-the-art algorithms for the variant of the problem with two levels of storage. We provide a detailed descriptionof how one can use this library in an adjoint computation software in the field of automatic differentiationor backpropagation. Finally, we evaluate the performance of H-Revolve and other checkpointing heuristicsthough an extensive campaign of simulation