Missing data frequently occurs in datasets across various domains, such as
medicine, sports, and finance. In many cases, to enable proper and reliable
analyses of such data, the missing values are often imputed, and it is
necessary that the method used has a low root mean square error (RMSE) between
the imputed and the true values. In addition, for some critical applications,
it is also often a requirement that the imputation method is scalable and the
logic behind the imputation is explainable, which is especially difficult for
complex methods that are, for example, based on deep learning. Based on these
considerations, we propose a new algorithm named "conditional
Distribution-based Imputation of Missing Values with Regularization" (DIMV).
DIMV operates by determining the conditional distribution of a feature that has
missing entries, using the information from the fully observed features as a
basis. As will be illustrated via experiments in the paper, DIMV (i) gives a
low RMSE for the imputed values compared to state-of-the-art methods; (ii) fast
and scalable; (iii) is explainable as coefficients in a regression model,
allowing reliable and trustable analysis, makes it a suitable choice for
critical domains where understanding is important such as in medical fields,
finance, etc; (iv) can provide an approximated confidence region for the
missing values in a given sample; (v) suitable for both small and large scale
data; (vi) in many scenarios, does not require a huge number of parameters as
deep learning approaches; (vii) handle multicollinearity in imputation
effectively; and (viii) is robust to the normally distributed assumption that
its theoretical grounds rely on