Barrier Frank-Wolfe for Marginal Inference

We introduce a globally-convergent algorithm for optimizing the tree-reweighted (TRW) variational objective over the marginal polytope. The algorithm is based on the conditional gradient method (Frank-Wolfe) and moves pseudomarginals within the marginal polytope through repeated maximum a posteriori (MAP) calls. This modular structure enables us to leverage black-box MAP solvers (both exact and approximate) for variational inference, and obtains more accurate results than tree-reweighted algorithms that optimize over the local consistency relaxation. Theoretically, we bound the sub-optimality for the proposed algorithm despite the TRW objective having unbounded gradients at the boundary of the marginal polytope. Empirically, we demonstrate the increased quality of results found by tightening the relaxation over the marginal polytope as well as the spanning tree polytope on synthetic and real-world instances.Comment: 25 pages, 12 figures, To appear in Neural Information Processing Systems (NIPS) 2015, Corrected reference and cleaned up bibliograph

New complexity results and algorithms for min-max-min robust combinatorial optimization

In this work we investigate the min-max-min robust optimization problem applied to combinatorial problems with uncertain cost-vectors which are contained in a convex uncertainty set. The idea of the approach is to calculate a set of k feasible solutions which are worst-case optimal if in each possible scenario the best of the k solutions would be implemented. It is known that the min-max-min robust problem can be solved efficiently if k is at least the dimension of the problem, while it is theoretically and computationally hard if k is small. While both cases are well studied in the literature nothing is known about the intermediate case, namely if k is smaller than but close to the dimension of the problem. We approach this open question and show that for a selection of combinatorial problems the min-max-min problem can be solved exactly and approximately in polynomial time if some problem specific values are fixed. Furthermore we approach a second open question and present the first implementable algorithm with oracle-pseudopolynomial runtime for the case that k is at least the dimension of the problem. The algorithm is based on a projected subgradient method where the projection problem is solved by the classical Frank-Wolfe algorithm. Additionally we derive a branch & bound method to solve the min-max-min problem for arbitrary values of k and perform tests on knapsack and shortest path instances. The experiments show that despite its theoretical impact the projected subgradient method cannot compete with an already existing method. On the other hand the performance of the branch & bound method scales very well with the number of solutions. Thus we are able to solve instances where k is above some small threshold very efficiently

An Active Set Algorithm for Robust Combinatorial Optimization Based on Separation Oracles

We address combinatorial optimization problems with uncertain coefficients varying over ellipsoidal uncertainty sets. The robust counterpart of such a problem can be rewritten as a second-oder cone program (SOCP) with integrality constraints. We propose a branch-and-bound algorithm where dual bounds are computed by means of an active set algorithm. The latter is applied to the Lagrangian dual of the continuous relaxation, where the feasible set of the combinatorial problem is supposed to be given by a separation oracle. The method benefits from the closed form solution of the active set subproblems and from a smart update of pseudo-inverse matrices. We present numerical experiments on randomly generated instances and on instances from different combinatorial problems, including the shortest path and the traveling salesman problem, showing that our new algorithm consistently outperforms the state-of-the art mixed-integer SOCP solver of Gurobi

A combined simulated annealing and TABU search strategy to solve network design problem with two classes of users

A methodology to solve a transportation network design problem (TCNDP) with two classes of users (passenger cars and trucks) is developed. Given an existing highway system, with a capital investment budget constraint, the methodology selects the best links to be expanded by an extra lane by considering one of three types of traffic operations: exclusive for passenger cars, exclusive for trucks, and for both passenger cars and trucks such that the network total user equilibrium (UE) travel time is minimized. The problem is formulated as an NP-hard combinatorial nonlinear integer programming problem. The classical branch and bound methodology for the integer programming problem is very inefficient in solving this computationally hard problem. A combined simulated annealing and tabu search strategy (SA-TABU), was developed which is shown to perform in a robust and efficient manner in solving five networks ranging from 36 to 332 links. A comprehensive heuristic evaluation function (HEF), a core for the heuristic search strategy, was developed which can be adjusted to the characteristics of the problem and the search strategy used. It is composed of three elements: the link volume to capacity ratio, the historical contribution of the link to the objective function, and a random variable which resembles the error term of the HEF. The principal characteristics of the SA-TABU are the following: HEF, Markov chain length, “temperature” dropping rate and the tabu list length. Sensitivity analysis was conducted in identifying the best parameter values of the main components of the SA-TABU. Sufficiently “good” solutions were found in all the problems within a rather short computational time. The solution results suggest that in most of the scenarios, the shared lane option, passenger cars and trucks, was found to be the most favored selection. Expanding approximately 10% of the links, results in a very high percentage improvement ranging from 73% to 97% for the five test networks

RNN training along locally optimal trajectoriesvia Frank-Wolfe algorithm

We propose a novel and efficient training method for RNNs by iteratively seeking a local minima on the loss surface within a small region, and leverage this directional vector for the update, in an outer-loop. We propose to utilize the Frank-Wolfe (FW) algorithm in this context. Although, FW implicitly involves normalized gradients, which can lead to a slow convergence rate, we develop a novel RNN training method that, surprisingly, even with the additional cost, the overall training cost is empirically observed to be lower than backpropagation. Our method leads to a new Frank-Wolfe method, that is in essence an SGD algorithm with a restart scheme. We prove that under certain conditions our algorithm has a sublinear convergence rate of O (1/ϵ) for ϵ error. We then conduct empirical experiments on several benchmark datasets including those that exhibit long-term dependencies, and show significant performance improvement. We also experiment with deep RNN architectures and show efficient training performance. Finally, we demonstrate that our training method is robust to noisy data.https://doi.org/10.1109/icpr48806.2021.9412188Accepted manuscrip

Modelling and solution methods for portfolio optimisation

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University, 16/01/2004.In this thesis modelling and solution methods for portfolio optimisation are presented. The investigations reported in this thesis extend the Markowitz mean-variance model to the domain of quadratic mixed integer programming (QMIP) models which are 'NP-hard' discrete optimisation problems. In addition to the modelling extensions a number of challenging aspects of solution algorithms are considered. The relative performances of sparse simplex (SSX) as well as the interior point method (IPM) are studied in detail. In particular, the roles of 'warmstart' and dual simplex are highlighted as applied to the construction of the efficient frontier which requires processing a family of problems; that is, the portfolio planning model stated in a parametric form. The method of solving QMIP models using the branch and bound algorithm is first developed; this is followed up by heuristics which improve the performance of the (discrete) solution algorithm. Some properties of the efficient frontier with discrete constraints are considered and a method of computing the discrete efficient frontier (DEF) efficiently is proposed. The computational investigation considers the efficiency and effectiveness in respect of the scale up properties of the proposed algorithm. The extensions of the real world models and the proposed solution algorithms make contribution as new knowledge