Reweighted nuclear norm regularization: A SPARSEVA approach

The aim of this paper is to develop a method to estimate high order FIR and ARX models using least squares with re-weighted nuclear norm regularization. Typically, the choice of the tuning parameter in the reweighting scheme is computationally expensive, hence we propose the use of the SPARSEVA (SPARSe Estimation based on a VAlidation criterion) framework to overcome this problem. Furthermore, we suggest the use of the prediction error criterion (PEC) to select the tuning parameter in the SPARSEVA algorithm. Numerical examples demonstrate the veracity of this method which has close ties with the traditional technique of cross validation, but using much less computations.Comment: This paper is accepted and will be published in The Proceedings of the 17th IFAC Symposium on System Identification (SYSID 2015), Beijing, China, 201

Stability Approach to Regularization Selection (StARS) for High Dimensional Graphical Models

A challenging problem in estimating high-dimensional graphical models is to choose the regularization parameter in a data-dependent way. The standard techniques include $K$-fold cross-validation ($K$-CV), Akaike information criterion (AIC), and Bayesian information criterion (BIC). Though these methods work well for low-dimensional problems, they are not suitable in high dimensional settings. In this paper, we present StARS: a new stability-based method for choosing the regularization parameter in high dimensional inference for undirected graphs. The method has a clear interpretation: we use the least amount of regularization that simultaneously makes a graph sparse and replicable under random sampling. This interpretation requires essentially no conditions. Under mild conditions, we show that StARS is partially sparsistent in terms of graph estimation: i.e. with high probability, all the true edges will be included in the selected model even when the graph size diverges with the sample size. Empirically, the performance of StARS is compared with the state-of-the-art model selection procedures, including $K$-CV, AIC, and BIC, on both synthetic data and a real microarray dataset. StARS outperforms all these competing procedures

MMP-DCD-CV based Sparse Channel Estimation Algorithm for Underwater Acoustic Transform Domain Communication System

In this paper, we propose a computationally efficient multipath matching pursuit (MMP) channel estimation algorithm for underwater acoustic (UWA) transform domain communication systems (TDCSs). The algorithm, referred to as the MMP-DCD-CV algorithm, is based on the dichotomous coordinate descent (DCD) iterations and cross validation (CV). The MMP-DCD-CV sparse channel estimator in each iteration searches for multiple promising path candidates most relevant to a residual vector and chooses the best candidate. The DCD iterations are used to solve the corresponding least squares problem with low complexity and numerical stability. The CV provides a stopping criterion of the algorithm without a priori information on the channel sparsity and noise level and examines whether the algorithm overfits its data, thus improving the estimation accuracy. The performance of the proposed algorithm is evaluated under simulated sparse UWA channels. The numerical results show that the algorithm achieves better performance than the original MMP algorithm, has lower complexity, and does not require prior knowledge on the channel sparsity and noise level. We also propose an UWA TDCS with sparse channel estimation based on the proposed MMP-DCD-CV algorithm. The proposed UWA communication system is tested by the Waymark simulator, providing the virtual signal transmission in the UWA channel, with a measured Sound Speed Profile and bathymetry. Numerical results demonstrate that the UWA TDCS with the proposed sparse channel estimator offers considerable improvement in system performance compared to other TDCS schemes

Sparse kernel density construction using orthogonal forward regression with leave-one-out test score and local regularization

The paper presents an efficient construction algorithm for obtaining sparse kernel density estimates based on a regression approach that directly optimizes model generalization capability. Computational efficiency of the density construction is ensured using an orthogonal forward regression, and the algorithm incrementally minimizes the leave-one-out test score. A local regularization method is incorporated naturally into the density construction process to further enforce sparsity. An additional advantage of the proposed algorithm is that it is fully automatic and the user is not required to specify any criterion to terminate the density construction procedure. This is in contrast to an existing state-of-art kernel density estimation method using the support vector machine (SVM), where the user is required to specify some critical algorithm parameter. Several examples are included to demonstrate the ability of the proposed algorithm to effectively construct a very sparse kernel density estimate with comparable accuracy to that of the full sample optimized Parzen window density estimate. Our experimental results also demonstrate that the proposed algorithm compares favourably with the SVM method, in terms of both test accuracy and sparsity, for constructing kernel density estimates

Predicting with sparse data

It is well known that effective prediction of project cost related factors is an important aspect of software engineering. Unfortunately, despite extensive research over more than 30 years, this remains a significant problem for many practitioners. A major obstacle is the absence of reliable and systematic historic data, yet this is a sine qua non for almost all proposed methods: statistical, machine learning or calibration of existing models. In this paper we describe our sparse data method (SDM) based upon a pairwise comparison technique and Saaty's Analytic Hierarchy Process (AHP). Our minimum data requirement is a single known point. The technique is supported by a software tool known as DataSalvage. We show, for data from two companies, how our approach — based upon expert judgement — adds value to expert judgement by producing significantly more accurate and less biased results. A sensitivity analysis shows that our approach is robust to pairwise comparison errors. We then describe the results of a small usability trial with a practising project manager. From this empirical work we conclude that the technique is promising and may help overcome some of the present barriers to effective project prediction