23,598 research outputs found
Dynamic Control Flow in Large-Scale Machine Learning
Many recent machine learning models rely on fine-grained dynamic control flow
for training and inference. In particular, models based on recurrent neural
networks and on reinforcement learning depend on recurrence relations,
data-dependent conditional execution, and other features that call for dynamic
control flow. These applications benefit from the ability to make rapid
control-flow decisions across a set of computing devices in a distributed
system. For performance, scalability, and expressiveness, a machine learning
system must support dynamic control flow in distributed and heterogeneous
environments.
This paper presents a programming model for distributed machine learning that
supports dynamic control flow. We describe the design of the programming model,
and its implementation in TensorFlow, a distributed machine learning system.
Our approach extends the use of dataflow graphs to represent machine learning
models, offering several distinctive features. First, the branches of
conditionals and bodies of loops can be partitioned across many machines to run
on a set of heterogeneous devices, including CPUs, GPUs, and custom ASICs.
Second, programs written in our model support automatic differentiation and
distributed gradient computations, which are necessary for training machine
learning models that use control flow. Third, our choice of non-strict
semantics enables multiple loop iterations to execute in parallel across
machines, and to overlap compute and I/O operations.
We have done our work in the context of TensorFlow, and it has been used
extensively in research and production. We evaluate it using several real-world
applications, and demonstrate its performance and scalability.Comment: Appeared in EuroSys 2018. 14 pages, 16 figure
A hierarchy of models for simulating experimental results from a 3D heterogeneous porous medium
In this work we examine the dispersion of conservative tracers (bromide and
fluorescein) in an experimentally-constructed three-dimensional dual-porosity
porous medium. The medium is highly heterogeneous (), and
consists of spherical, low-hydraulic-conductivity inclusions embedded in a
high-hydraulic-conductivity matrix. The bi-modal medium was saturated with
tracers, and then flushed with tracer-free fluid while the effluent
breakthrough curves were measured. The focus for this work is to examine a
hierarchy of four models (in the absence of adjustable parameters) with
decreasing complexity to assess their ability to accurately represent the
measured breakthrough curves. The most information-rich model was (1) a direct
numerical simulation of the system in which the geometry, boundary and initial
conditions, and medium properties were fully independently characterized
experimentally with high fidelity. The reduced models included; (2) a
simplified numerical model identical to the fully-resolved direct numerical
simulation (DNS) model, but using a domain that was one-tenth the size; (3) an
upscaled mobile-immobile model that allowed for a time-dependent mass-transfer
coefficient; and, (4) an upscaled mobile-immobile model that assumed a
space-time constant mass-transfer coefficient. The results illustrated that all
four models provided accurate representations of the experimental breakthrough
curves as measured by global RMS error. The primary component of error induced
in the upscaled models appeared to arise from the neglect of convection within
the inclusions. Interestingly, these results suggested that the conventional
convection-dispersion equation, when applied in a way that resolves the
heterogeneities, yields models with high fidelity without requiring the
imposition of a more complex non-Fickian model.Comment: 27 pages, 9 Figure
- …