Spot instances are virtual machines offered at 60-90% lower cost that can be
reclaimed at any time, with only a short warning period. Spot instances have
already been used to significantly reduce the cost of processing workloads in
the cloud. However, leveraging spot instances to reduce the cost of stateful
cloud applications is much more challenging, as the sudden preemptions lead to
data loss. In this work, we propose leveraging spot instances to decrease the
cost of ephemeral data management in distributed data analytics applications.
We specifically target ephemeral data as this large class of data in modern
analytics workloads has low durability requirements; if lost, the data can be
regenerated by re-executing compute tasks. We design an elastic, distributed
ephemeral datastore that handles node preemptions transparently to user
applications and minimizes data loss by redistributing data during node
preemption warning periods. We implement our elastic datastore on top of the
Apache Crail datastore and evaluate the system with various workloads and VM
types. By leveraging spot instances, we show that we can run TPC-DS queries
with 60\% lower cost compared to using on-demand VMs for the datastore, while
only increasing end-to-end execution time by 2.1%