Frugal Optimization for Cost-related Hyperparameters

The increasing demand for democratizing machine learning algorithms calls for hyperparameter optimization (HPO) solutions at low cost. Many machine learning algorithms have hyperparameters which can cause a large variation in the training cost. But this effect is largely ignored in existing HPO methods, which are incapable to properly control cost during the optimization process. To address this problem, we develop a new cost-frugal HPO solution. The core of our solution is a simple but new randomized direct-search method, for which we prove a convergence rate of $O(\frac{\sqrt{d}}{\sqrt{K}})$ and an $O(d\epsilon^{-2})$-approximation guarantee on the total cost. We provide strong empirical results in comparison with state-of-the-art HPO methods on large AutoML benchmarks.Comment: 29 pages (including supplementary appendix

Randomized Composable Core-sets for Distributed Submodular Maximization

An effective technique for solving optimization problems over massive data sets is to partition the data into smaller pieces, solve the problem on each piece and compute a representative solution from it, and finally obtain a solution inside the union of the representative solutions for all pieces. This technique can be captured via the concept of {\em composable core-sets}, and has been recently applied to solve diversity maximization problems as well as several clustering problems. However, for coverage and submodular maximization problems, impossibility bounds are known for this technique \cite{IMMM14}. In this paper, we focus on efficient construction of a randomized variant of composable core-sets where the above idea is applied on a {\em random clustering} of the data. We employ this technique for the coverage, monotone and non-monotone submodular maximization problems. Our results significantly improve upon the hardness results for non-randomized core-sets, and imply improved results for submodular maximization in a distributed and streaming settings. In summary, we show that a simple greedy algorithm results in a $1/3$-approximate randomized composable core-set for submodular maximization under a cardinality constraint. This is in contrast to a known $O({\log k\over \sqrt{k}})$ impossibility result for (non-randomized) composable core-set. Our result also extends to non-monotone submodular functions, and leads to the first 2-round MapReduce-based constant-factor approximation algorithm with $O(n)$ total communication complexity for either monotone or non-monotone functions. Finally, using an improved analysis technique and a new algorithm $\mathsf{PseudoGreedy}$, we present an improved $0.545$-approximation algorithm for monotone submodular maximization, which is in turn the first MapReduce-based algorithm beating factor $1/2$ in a constant number of rounds

Possible solution of the Coriolis attenuation problem

The most consistently useful simple model for the study of odd deformed nuclei, the particle-rotor model (strong coupling limit of the core-particle coupling model) has nevertheless been beset by a long-standing problem: It is necessary in many cases to introduce an ad hoc parameter that reduces the size of the Coriolis interaction coupling the collective and single-particle motions. Of the numerous suggestions put forward for the origin of this supplementary interaction, none of those actually tested by calculations has been accepted as the solution of the problem. In this paper we seek a solution of the difficulty within the framework of a general formalism that starts from the spherical shell model and is capable of treating an arbitrary linear combination of multipole and pairing forces. With the restriction of the interaction to the familiar sum of a quadrupole multipole force and a monopole pairing force, we have previously studied a semi-microscopic version of the formalism whose framework is nevertheless more comprehensive than any previously applied to the problem. We obtained solutions for low-lying bands of several strongly deformed odd rare earth nuclei and found good agreement with experiment, except for an exaggerated staggering of levels for K=1/2 bands, which can be understood as a manifestation of the Coriolis attenuation problem. We argue that within the formalism utilized, the only way to improve the physics is to add interactions to the model Hamiltonian. We verify that by adding a magnetic dipole interaction of essentially fixed strength, we can fit the K=1/2 bands without destroying the agreement with other bands. In addition we show that our solution also fits 163Er, a classic test case of Coriolis attenuation that we had not previously studied.Comment: revtex, including 7 figures(postscript), submitted to Phys.Rev.

Efficient Progressive Minimum k-Core Search

As one of the most representative cohesive subgraph models, k-core model has recently received significant attention in the literature. In this paper, we investigate the problem of the minimum k-core search: given a graph G, an integer k and a set of query vertices Q = q, we aim to find the smallest k-core subgraph containing every query vertex q ∈ Q. It has been shown that this problem is NP-hard with a huge search space, and it is very challenging to find the optimal solution. There are several heuristic algorithms for this problem, but they rely on simple scoring functions and there is no guarantee as to the size of the resulting subgraph, compared with the optimal solution. Our empirical study also indicates that the size of their resulting subgraphs may be large in practice. In this paper, we develop an effective and efficient progressive algorithm, namely PSA, to provide a good trade-off between the quality of the result and the search time. Novel lower and upper bound techniques for the minimum k-core search are designed. Our extensive experiments on 12 real-life graphs demonstrate the effectiveness and efficiency of the new techniques

Distributed Consensus Algorithm for Decision-Making in Multi-agent Multi-armed Bandit

We study a structured multi-agent multi-armed bandit (MAMAB) problem in a dynamic environment. A graph reflects the information-sharing structure among agents, and the arms' reward distributions are piecewise-stationary with several unknown change points. The agents face the identical piecewise-stationary MAB problem. The goal is to develop a decision-making policy for the agents that minimizes the regret, which is the expected total loss of not playing the optimal arm at each time step. Our proposed solution, Restarted Bayesian Online Change Point Detection in Cooperative Upper Confidence Bound Algorithm (RBO-Coop-UCB), involves an efficient multi-agent UCB algorithm as its core enhanced with a Bayesian change point detector. We also develop a simple restart decision cooperation that improves decision-making. Theoretically, we establish that the expected group regret of RBO-Coop-UCB is upper bounded by $\mathcal{O}(KNM\log T + K\sqrt{MT\log T})$, where K is the number of agents, M is the number of arms, and T is the number of time steps. Numerical experiments on synthetic and real-world datasets demonstrate that our proposed method outperforms the state-of-the-art algorithms

Characterizing Structurally Cohesive Clusters in Networks: Theory and Algorithms

This dissertation aims at developing generalized network models and solution approaches for studying cluster detection problems that typically arise in networks. More specifically, we consider graph theoretic relaxations of clique as models for characterizing structurally cohesive and robust subgroups, developing strong upper bounds for the maximum clique problem, and present a new relaxation that is useful in clustering applications. We consider the clique relaxation models of k-block, and k-robust 2-club for describing cohesive clusters that are reliable and robust to disruptions, and introduce a new relaxation called s-stable cluster, for modeling stable clusters. First, we identify the structural properties associated with the models, and investigate the computational complexity of these problems. Next, we develop mathematical programming techniques for the optimization problems introduced, and apply them in presenting effective solution approaches to the problems. We present integer programming formulations for the optimization problems of interest, and provide a detailed study of the associated polytopes. Particularly, we develop valid inequalities and identify different classes of facets for the polytopes. Exact solution approaches developed for solving the problems include simple branch and bound, branch and cut, and combinatorial branch and bound algorithms. In addition, we introduce many preprocessing techniques and heuristics to enhance their performance. The presented algorithms are tested computationally on a number of graph instances, that include social networks and random graphs, to study the capability of the proposed solution methods. As a fitting conclusion to this work, we propose new techniques to get easily computable and strong upper bounds for the maximum clique problem. We investigate k-core and its stronger variant k-core/2-club in this light, and present minimization problems to get an upper bound on the maximization problems. Simple linear programming relaxations are developed and strengthened by valid inequalities, which are then compared with some standard relaxations from the literature. We present a detailed study of our computational results on a number of benchmark instances to test the effectiveness of our technique for getting good upper bounds

A Direct Algorithm for the Type Interference in the Rank 2 Fragment of the Second--Order λ-Calculus

We study the problem of type inference for a family of polymorphic type disciplines containing the power of Core-ML. This family comprises all levels of the stratification of the second-order lambda-calculus by "rank" of types. We show that typability is an undecidable problem at every rank k ≥ 3 of this stratification. While it was already known that typability is decidable at rank ≤ 2, no direct and easy-to-implement algorithm was available. To design such an algorithm, we develop a new notion of reduction and show how to use it to reduce the problem of typability at rank 2 to the problem of acyclic semi-unification. A by-product of our analysis is the publication of a simple solution procedure for acyclic semi-unification

Computing Stable Coalitions: Approximation Algorithms for Reward Sharing

Consider a setting where selfish agents are to be assigned to coalitions or projects from a fixed set P. Each project k is characterized by a valuation function; v_k(S) is the value generated by a set S of agents working on project k. We study the following classic problem in this setting: "how should the agents divide the value that they collectively create?". One traditional approach in cooperative game theory is to study core stability with the implicit assumption that there are infinite copies of one project, and agents can partition themselves into any number of coalitions. In contrast, we consider a model with a finite number of non-identical projects; this makes computing both high-welfare solutions and core payments highly non-trivial. The main contribution of this paper is a black-box mechanism that reduces the problem of computing a near-optimal core stable solution to the purely algorithmic problem of welfare maximization; we apply this to compute an approximately core stable solution that extracts one-fourth of the optimal social welfare for the class of subadditive valuations. We also show much stronger results for several popular sub-classes: anonymous, fractionally subadditive, and submodular valuations, as well as provide new approximation algorithms for welfare maximization with anonymous functions. Finally, we establish a connection between our setting and the well-studied simultaneous auctions with item bidding; we adapt our results to compute approximate pure Nash equilibria for these auctions.Comment: Under Revie