Swarm-Based Optimization with Random Descent

We extend our study of the swarm-based gradient descent method for non-convex optimization, [Lu, Tadmor & Zenginoglu, arXiv:2211.17157], to allow random descent directions. We recall that the swarm-based approach consists of a swarm of agents, each identified with a position, ${\mathbf x}$, and mass, $m$. The key is the transfer of mass from high ground to low(-est) ground. The mass of an agent dictates its step size: lighter agents take larger steps. In this paper, the essential new feature is the choice of direction: rather than restricting the swarm to march in the steepest gradient descent, we let agents proceed in randomly chosen directions centered around -- but otherwise different from -- the gradient direction. The random search secures the descent property while at the same time, enabling greater exploration of ambient space. Convergence analysis and benchmark optimizations demonstrate the effectiveness of the swarm-based random descent method as a multi-dimensional global optimizer

Linearly convergent adjoint free solution of least squares problems by random descent

We consider the problem of solving linear least squares problems in a framework where only evaluations of the linear map are possible. We derive randomized methods that do not need any other matrix operations than forward evaluations, especially no evaluation of the adjoint map is needed. Our method is motivated by the simple observation that one can get an unbiased estimate of the application of the adjoint. We show convergence of the method and then derive a more efficient method that uses an exact linesearch. This method, called random descent, resembles known methods in other context and has the randomized coordinate descent method as special case. We provide convergence analysis of the random descent method emphasizing the dependence on the underlying distribution of the random vectors. Furthermore we investigate the applicability of the method in the context of ill-posed inverse problems and show that the method can have beneficial properties when the unknown solution is rough. We illustrate the theoretical findings in numerical examples. One particular result is that the random descent method actually outperforms established transposed-free methods (TFQMR and CGS) in examples

Heuristic pattern search for bound constrained minimax problems

This paper presents a pattern search algorithm and its hybridization with a random descent search for solving bound constrained minimax problems. The herein proposed heuristic pattern search method combines the Hooke and Jeeves (HJ) pattern and exploratory moves with a randomly generated approxi- mate descent direction. Two versions of the heuristic algorithm have been applied to several benchmark minimax problems and compared with the original HJ pat- tern search algorithm

That rare random descent : an approach to the cantos of sylvia plath

The intent of the following thesis is to establish an approach to the body of Sylvia Plath\u27s poetry, to provide some insight into an artistic career which lasted less than ten years. [...] The approach I establish, then, takes the poems in their chronological order. I have attempted an understanding of her mautre work, thus, on its own terms, terms Plath herself established, subsequently broke from and re-styled, so that I might dispel the stultifying effect her suicide has had on our critical interpretation

Compressed sensing of data with a known distribution

Compressed sensing is a technique for recovering an unknown sparse signal from a small number of linear measurements. When the measurement matrix is random, the number of measurements required for perfect recovery exhibits a phase transition: there is a threshold on the number of measurements after which the probability of exact recovery quickly goes from very small to very large. In this work we are able to reduce this threshold by incorporating statistical information about the data we wish to recover. Our algorithm works by minimizing a suitably weighted $\ell_1$-norm, where the weights are chosen so that the expected statistical dimension of the corresponding descent cone is minimized. We also provide new discrete-geometry-based Monte Carlo algorithms for computing intrinsic volumes of such descent cones, allowing us to bound the failure probability of our methods.Comment: 22 pages, 7 figures. New colorblind safe figures. Sections 3 and 4 completely rewritten. Minor typos fixe

Perceptron learning with random coordinate descent

A perceptron is a linear threshold classifier that separates examples with a hyperplane. It is perhaps the simplest learning model that is used standalone. In this paper, we propose a family of random coordinate descent algorithms for perceptron learning on binary classification problems. Unlike most perceptron learning algorithms which require smooth cost functions, our algorithms directly minimize the training error, and usually achieve the lowest training error compared with other algorithms. The algorithms are also computational efficient. Such advantages make them favorable for both standalone use and ensemble learning, on problems that are not linearly separable. Experiments show that our algorithms work very well with AdaBoost, and achieve the lowest test errors for half of the datasets

A Note on Shortest Developments

De Vrijer has presented a proof of the finite developments theorem which, in addition to showing that all developments are finite, gives an effective reduction strategy computing longest developments as well as a simple formula computing their length. We show that by applying a rather simple and intuitive principle of duality to de Vrijer's approach one arrives at a proof that some developments are finite which in addition yields an effective reduction strategy computing shortest developments as well as a simple formula computing their length. The duality fails for general beta-reduction. Our results simplify previous work by Khasidashvili

Probabilistic Rewriting: On Normalization, Termination, and Unique Normal Forms

While a mature body of work supports the study of rewriting systems, even infinitary ones, abstract tools for Probabilistic Rewriting are still limited. Here, we investigate questions such as uniqueness of the result (unique limit distribution) and we develop a set of proof techniques to analyze and compare reduction strategies. The goal is to have tools to support the operational analysis of probabilistic calculi (such as probabilistic lambda-calculi) whose evaluation is also non-deterministic, in the sense that different reductions are possible. In particular, we investigate how the behavior of different rewrite sequences starting from the same term compare w.r.t. normal forms, and propose a robust analogue of the notion of "unique normal form". Our approach is that of Abstract Rewrite Systems, i.e. we search for general properties of probabilistic rewriting, which hold independently of the specific structure of the objects.Comment: Extended version of the paper in FSCD 2019, International Conference on Formal Structures for Computation and Deductio