Nearly optimal Bayesian Shrinkage for High Dimensional Regression

During the past decade, shrinkage priors have received much attention in Bayesian analysis of high-dimensional data. In this paper, we study the problem for high-dimensional linear regression models. We show that if the shrinkage prior has a heavy and flat tail, and allocates a sufficiently large probability mass in a very small neighborhood of zero, then its posterior properties are as good as those of the spike-and-slab prior. While enjoying its efficiency in Bayesian computation, the shrinkage prior can lead to a nearly optimal contraction rate and selection consistency as the spike-and-slab prior. Our numerical results show that under posterior consistency, Bayesian methods can yield much better results in variable selection than the regularization methods, such as Lasso and SCAD. We also establish a Bernstein von-Mises type results comparable to Castillo et al (2015), this result leads to a convenient way to quantify uncertainties of the regression coefficient estimates, which has been beyond the ability of regularization methods

Optimal False Discovery Control of Minimax Estimator

In the analysis of high dimensional regression models, there are two important objectives: statistical estimation and variable selection. In literature, most works focus on either optimal estimation, e.g., minimax $L_2$ error, or optimal selection behavior, e.g., minimax Hamming loss. However in this study, we investigate the subtle interplay between the estimation accuracy and selection behavior. Our result shows that an estimator's $L_2$ error rate critically depends on its performance of type I error control. Essentially, the minimax convergence rate of false discovery rate over all rate-minimax estimators is a polynomial of the true sparsity ratio. This result helps us to characterize the false positive control of rate-optimal estimators under different sparsity regimes. More specifically, under near-linear sparsity, the number of yielded false positives always explodes to infinity under worst scenario, but the false discovery rate still converges to 0; under linear sparsity, even the false discovery rate doesn't asymptotically converge to 0. On the other side, in order to asymptotically eliminate all false discoveries, the estimator must be sub-optimal in terms of its convergence rate. This work attempts to offer rigorous analysis on the incompatibility phenomenon between selection consistency and rate-minimaxity observed in the high dimensional regression literature

Fair Supervised Learning with A Simple Random Sampler of Sensitive Attributes

As the data-driven decision process becomes dominating for industrial applications, fairness-aware machine learning arouses great attention in various areas. This work proposes fairness penalties learned by neural networks with a simple random sampler of sensitive attributes for non-discriminatory supervised learning. In contrast to many existing works that critically rely on the discreteness of sensitive attributes and response variables, the proposed penalty is able to handle versatile formats of the sensitive attributes, so it is more extensively applicable in practice than many existing algorithms. This penalty enables us to build a computationally efficient group-level in-processing fairness-aware training framework. Empirical evidence shows that our framework enjoys better utility and fairness measures on popular benchmark data sets than competing methods. We also theoretically characterize estimation errors and loss of utility of the proposed neural-penalized risk minimization problem

Personalized Federated X -armed Bandit

In this work, we study the personalized federated $\mathcal{X}$-armed bandit problem, where the heterogeneous local objectives of the clients are optimized simultaneously in the federated learning paradigm. We propose the \texttt{PF-PNE} algorithm with a unique double elimination strategy, which safely eliminates the non-optimal regions while encouraging federated collaboration through biased but effective evaluations of the local objectives. The proposed \texttt{PF-PNE} algorithm is able to optimize local objectives with arbitrary levels of heterogeneity, and its limited communications protects the confidentiality of the client-wise reward data. Our theoretical analysis shows the benefit of the proposed algorithm over single-client algorithms. Experimentally, \texttt{PF-PNE} outperforms multiple baselines on both synthetic and real life datasets