In linear regression we wish to estimate the optimum linear least squares
predictor for a distribution over d-dimensional input points and real-valued
responses, based on a small sample. Under standard random design analysis,
where the sample is drawn i.i.d. from the input distribution, the least squares
solution for that sample can be viewed as the natural estimator of the optimum.
Unfortunately, this estimator almost always incurs an undesirable bias coming
from the randomness of the input points. In this paper we show that it is
possible to draw a non-i.i.d. sample of input points such that, regardless of
the response model, the least squares solution is an unbiased estimator of the
optimum. Moreover, this sample can be produced efficiently by augmenting a
previously drawn i.i.d. sample with an additional set of d points drawn jointly
from the input distribution rescaled by the squared volume spanned by the
points. Motivated by this, we develop a theoretical framework for studying
volume-rescaled sampling, and in the process prove a number of new matrix
expectation identities. We use them to show that for any input distribution and
ϵ>0 there is a random design consisting of O(dlogd+d/ϵ)
points from which an unbiased estimator can be constructed whose square loss
over the entire distribution is with high probability bounded by 1+ϵ
times the loss of the optimum. We provide efficient algorithms for generating
such unbiased estimators in a number of practical settings and support our
claims experimentally