Let's Make Block Coordinate Descent Go Fast: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergence

Block coordinate descent (BCD) methods are widely-used for large-scale numerical optimization because of their cheap iteration costs, low memory requirements, amenability to parallelization, and ability to exploit problem structure. Three main algorithmic choices influence the performance of BCD methods: the block partitioning strategy, the block selection rule, and the block update rule. In this paper we explore all three of these building blocks and propose variations for each that can lead to significantly faster BCD methods. We (i) propose new greedy block-selection strategies that guarantee more progress per iteration than the Gauss-Southwell rule; (ii) explore practical issues like how to implement the new rules when using "variable" blocks; (iii) explore the use of message-passing to compute matrix or Newton updates efficiently on huge blocks for problems with a sparse dependency between variables; and (iv) consider optimal active manifold identification, which leads to bounds on the "active set complexity" of BCD methods and leads to superlinear convergence for certain problems with sparse solutions (and in some cases finite termination at an optimal solution). We support all of our findings with numerical results for the classic machine learning problems of least squares, logistic regression, multi-class logistic regression, label propagation, and L1-regularization

A 1998 Social Accounting Matrix for Malawi

The last few years have seen a proliferation of attempts by various institutions to create a framework that would enable analysts to have a broad overview of all transactions in the Malawian economy. It was decided that 1998, the most recent year for which a comprehensive data set is available, would be the base year for the SAM. The National Statistical Office conducted a major household survey which provided information on budget shares, incomes and many other social-economic characteristics of households.Social accounting Malawi. ,Economic surveys Malaw. ,Household surveys Malawi. ,TMD ,

Comodularity and detection of co-communities

This paper introduces the notion of comodularity, to cocluster observations of bipartite networks into co-communities. The task of coclustering is to group together nodes of one type with nodes of another type, according to the interactions that are the most similar. The measure of comodularity is introduced to assess the strength of co-communities, as well as to arrange the representation of nodes and clusters for visualization, and to define an objective function for optimization. We demonstrate the usefulness of our proposed methodology on simulated data, and with examples from genomics and consumer-product reviews

Making Sense of Direction: Proximity and Order in Asymmetric Paired Comparison Data

In a square asymmetric matrix, the relationships among objects in the lower triangular half-matrix, differ from the relationships among the same objects in the upper triangular half. Square, asymmetric matrices can arise in similarity and preference data, when the direction of comparison is important. An asymmetric matrix can be rendered symmetric by averaging corresponding entries above and below the main diagonal. The difference between the original and the symmetric matrix is purely asymmetric, or skew-symmetric. The symmetric and skew-symmetric pans are orthogonal. An eigenvector-eigenvalue decomposition analyses the asymmetries into rank 2 skew-symmetric matrices, having an optimum least squares fit to the asymmetries (Gower, 1977). In this dissertation I derive an alternating least squares, nonmetric analogue of the canonical decomposition of asymmetry, suitable for ordinal-level data. In simulation studies, the nonmetric version gives better metric and nonmetric recovery, than does the canonical decomposition, when the asymmetries have been distorted by a range-compressing monotonic transform. The nonmetric technique appears to out-perform the canonical decomposition in detecting simplexes, and possibly in recovering multiplicative bias coefficients. However, canonical decomposition gives superior recovery after range-expanding monotonic transforms, and in the presence of error. An eigenvalue ratio test is proposed for determining the number of eigenvectors to extract in the canonical decomposition. The test quantifies changes in the slope of the log eigenvalue plot. In simulation studies the test appears to maintain its anticipated Type I error rate. The test is under-powered , which may help it to extract only well-identified eigenvectors. Finally, directional similarity judgments were collected for all possible pairs of exemplars of two semantic categories. The exemplars differed in typicality. After Tversky (1977) this should produce asymmetries related to the typicality. No asymmetries were found, however. Power analysis indicated that a correlation ratio for the asymmetries of .05 could have been detected 90% of the time. An extreme groups analysis also did not indicate asymmetry. The first eigenvector underlying the symmetric data, however, was highly correlated with typicality. Hence, Tversky\u27s model was not supported

Classes of Split-Plot Response Surface Designs for Equivalent Estimation

When planning an experimental investigation, we are frequently faced with factors that are difficult or time consuming to manipulate, thereby making complete randomization impractical. A split-plot structure differentiates between the experimental units associated with these hard-to-change factors and others that are relatively easy-to-change and provides an efficient strategy that integrates the restrictions imposed by the experimental apparatus. Several industrial and scientific examples are presented to illustrate design considerations encountered in the restricted randomization context. In this paper, we propose classes of split-plot response designs that provide an intuitive and natural extension from the completely randomized context. For these designs, the ordinary least squares estimates of the model are equivalent to the generalized least squares estimates. This property provides best linear unbiased estimators and simplifies model estimation. The design conditions that allow for equivalent estimation are presented enabling design construction strategies to transform completely randomized Box-Behnken, equiradial, and small composite designs into a split-plot structure

Twenty years of P-splines

P-splines first appeared in the limelight twenty years ago. Since then they have become popular in applications and in theoretical work. The combination of a rich B-spline basis and a simple difference penalty lends itself well to a variety of generalizations, because it is based on regression. In effect, P-splines allow the building of a “backbone” for the “mixing and matching” of a variety of additive smooth structure components, while inviting all sorts of extensions: varying-coefficient effects, signal (functional) regressors, two-dimensional surfaces, non-normal responses, quantile (expectile) modelling, among others. Strong connections with mixed models and Bayesian analysis have been established. We give an overview of many of the central developments during the first two decades of P-splines.Peer Reviewe

Twenty years of P-splines

P-splines first appeared in the limelight twenty years ago. Since then they have become popular in applications and in theoretical work. The combination of a rich B-spline basis and a simple difference penalty lends itself well to a variety of generalizations, because it is based on regression. In effect, P-splines allow the building of a “backbone” for the “mixing and matching” of a variety of additive smooth structure components, while inviting all sorts of extensions: varying-coefficient effects, signal (functional) regressors, two-dimensional surfaces, non-normal responses, quantile (expectile) modelling, among others. Strong connections with mixed models and Bayesian analysis have been established. We give an overview of many of the central developments during the first two decades of P-splines