We study optimal variance reduction solutions for count and ratio metrics in
online controlled experiments. Our methods leverage flexible machine learning
tools to incorporate covariates that are independent from the treatment but
have predictive power for the outcomes, and employ the cross-fitting technique
to remove the bias in complex machine learning models. We establish CLT-type
asymptotic inference based on our estimators under mild convergence conditions.
Our procedures are optimal (efficient) for the corresponding targets as long as
the machine learning estimators are consistent, without any requirement for
their convergence rates. In complement to the general optimal procedure, we
also derive a linear adjustment method for ratio metrics as a special case that
is computationally efficient and can flexibly incorporate any pre-treatment
covariates. We evaluate the proposed variance reduction procedures with
comprehensive simulation studies and provide practical suggestions regarding
commonly adopted assumptions in computing ratio metrics. When tested on real
online experiment data from LinkedIn, the proposed optimal procedure for ratio
metrics can reduce up to 80\% of variance compared to the standard
difference-in-mean estimator and also further reduce up to 30\% of variance
compared to the CUPED approach by going beyond linearity and incorporating a
large number of extra covariates