Distributed data analytic engines like Spark are common choices to process
massive data in industry. However, the performance of Spark SQL highly depends
on the choice of configurations, where the optimal ones vary with the executed
workloads. Among various alternatives for Spark SQL tuning, Bayesian
optimization (BO) is a popular framework that finds near-optimal configurations
given sufficient budget, but it suffers from the re-optimization issue and is
not practical in real production. When applying transfer learning to accelerate
the tuning process, we notice two domain-specific challenges: 1) most previous
work focus on transferring tuning history, while expert knowledge from Spark
engineers is of great potential to improve the tuning performance but is not
well studied so far; 2) history tasks should be carefully utilized, where using
dissimilar ones lead to a deteriorated performance in production. In this
paper, we present Rover, a deployed online Spark SQL tuning service for
efficient and safe search on industrial workloads. To address the challenges,
we propose generalized transfer learning to boost the tuning performance based
on external knowledge, including expert-assisted Bayesian optimization and
controlled history transfer. Experiments on public benchmarks and real-world
tasks show the superiority of Rover over competitive baselines. Notably, Rover
saves an average of 50.1% of the memory cost on 12k real-world Spark SQL tasks
in 20 iterations, among which 76.2% of the tasks achieve a significant memory
reduction of over 60%.Comment: Accepted by KDD 202