Near-Optimal Target Learning With Stochastic Binary Signals

We study learning in a noisy bisection model: specifically, Bayesian algorithms to learn a target value V given access only to noisy realizations of whether V is less than or greater than a threshold theta. At step t = 0, 1, 2, ..., the learner sets threshold theta t and observes a noisy realization of sign(V - theta t). After T steps, the goal is to output an estimate V^ which is within an eta-tolerance of V . This problem has been studied, predominantly in environments with a fixed error probability q < 1/2 for the noisy realization of sign(V - theta t). In practice, it is often the case that q can approach 1/2, especially as theta -> V, and there is little known when this happens. We give a pseudo-Bayesian algorithm which provably converges to V. When the true prior matches our algorithm's Gaussian prior, we show near-optimal expected performance. Our methods extend to the general multiple-threshold setting where the observation noisily indicates which of k >= 2 regions V belongs to

Thompson sampling guided stochastic searching on the line for deceptive environments with applications to root-finding problems

publishedVersio

A novel technique for stochastic root-finding: Enhancing the search with adaptive d-ary search

The most fundamental problem encountered in the field of stochastic optimization and control, is the Stochastic Root Finding (SRF) problem where the task is to locate (or in the context of control, to move towards), an unknown point x* for which g(x*)=0 for a given function g that can only be observed in the presence of noise. The vast majority of the state-of-the-art solutions to the SRF problem in both stochastic optimization and control involve the theory of stochastic approximation. The premise of the latter family of algorithms is to operate by means of so-called “small-step” processes that explore the search space in a conservative manner. Instead of relying on the well-established stochastic approximation theory, we rather advocate a radically different approach that resorts to principles akin to those used in noisy Bisection Search. In deterministic Bisection Search, the main question is to determine a point x* located on a line by querying an Oracle about whether the optimal point x* lies to the left or right of a point x. A noisy version of this problem, in which the Oracle gives the correct response with probability p, was studied by Waeber and his colleagues. Inspired by the pioneering work of these authors, and its potential application in stochastic optimization and control, we map the SRF problem to a noisy Bisection Search problem where we solely use the sign information of the samples in order to guide the search. Our solution recursively shrinks the search space by, at least, a factor of 2d3 at each epoch, where d ≥ 2 is a user-defined parameter of the algorithm. Our scheme is based, in part, on the Continuous Point Location with Adaptive d-ary Search (CPL–AdS), originally presented by Oommen and his co-authors. The solution to the CPL–AdS, however, is not applicable here because of the inherent asymmetry of the SRF problem. Our solution invokes a CPL–AdS-like solution to partition the search interval into d sub-intervals, evaluates the location of the unknown root x* with respect to these sub-intervals using Learning Automata, and prunes the search space in each iteration by eliminating at least one partition. Our scheme, the CPL–AdS algorithm for SRF, denoted as SRF–AdS, is shown to converge to the unknown root x* with an arbitrary large degree of accuracy, i.e., with a probability as close to unity as desired.acceptedVersionnivå

Models and algorithms for multi-agent search problems

The problem of searching for objects of interest occurs in important applications ranging from rescue, security, transportation, to medicine. With the increasing use of autonomous vehicles as search platforms, there is a need for fast algorithms that can generate search plans for multiple agents in response to new information. In this dissertation, we develop new techniques for automated generation of search plans for different classes of search problems. First, we study the problem of searching for a stationary object in a discrete search space with multiple agents where each agent can access only a subset of the search space. In these problems, agents can fail to detect an object when inspecting a location. We show that when the probabilities of detection only depend on the locations, this problem can be reformulated as a minimum cost network optimization problem, and develop a fast specialized algorithm for the solution. We prove that our algorithm finds the optimal solution in finite time, and has worst-case computation performance that is faster than general minimum cost flow algorithms. We then generalize it to the case where the probabilities of detection depend on the agents and the locations, and propose a greedy algorithm that is 1/2-approximate. Second, we study the problem of searching for a moving object in a discrete search space with multiple agents where each agent can access only a subset of a discrete search space at any time and agents can fail to detect objects when searching a location at a given time. We provide necessary conditions for an optimal search plan, extending prior results in search theory. For the case where the probabilities of detection depend on the locations and the time periods, we develop a forward-backward iterative algorithm based on coordinate descent techniques to obtain solutions. To avoid local optimum, we derive a convex relaxation of the dynamic search problem and show this can be solved optimally using coordinate descent techniques. The solutions of the relaxed problem are used to provide random starting conditions for the iterative algorithm. We also address the problem where the probabilities of detection depend on the agents as well as the locations and the time periods, and show that a greedy-style algorithm is 1/2-approximate. Third, we study problems when multiple objects of interest being searched are physically scattered among locations on a graph and the agents are subject to motion constraints captured by the graph edges as well as budget constraints. We model such problem as an orienteering problem, when searching with a single agent, or a team orienteering problem, when searching with multiple agents. We develop novel real-time efficient algorithms for both problems. Fourth, we investigate classes of continuous-region multi-agent adaptive search problems as stochastic control problems with imperfect information. We allow the agent measurement errors to be either correlated or independent across agents. The structure of these problems, with objectives related to information entropy, allows for a complete characterization of the optimal strategies and the optimal cost. We derive a lower bound on the performance of the minimum mean-square error estimator, and provide upper bounds on the estimation error for special cases. For agents with independent errors, we show that the optimal sensing strategies can be obtained in terms of the solution of decoupled scalar convex optimization problems, followed by a joint region selection procedure. We further consider search of multiple objects and provide an explicit construction for adaptively determining the sensing actions

Towards Thompson Sampling for Complex Bayesian Reasoning

Paper III, IV, and VI are not available as a part of the dissertation due to the copyright.Thompson Sampling (TS) is a state-of-art algorithm for bandit problems set in a Bayesian framework. Both the theoretical foundation and the empirical efficiency of TS is wellexplored for plain bandit problems. However, the Bayesian underpinning of TS means that TS could potentially be applied to other, more complex, problems as well, beyond the bandit problem, if suitable Bayesian structures can be found. The objective of this thesis is the development and analysis of TS-based schemes for more complex optimization problems, founded on Bayesian reasoning. We address several complex optimization problems where the previous state-of-art relies on a relatively myopic perspective on the problem. These includes stochastic searching on the line, the Goore game, the knapsack problem, travel time estimation, and equipartitioning. Instead of employing Bayesian reasoning to obtain a solution, they rely on carefully engineered rules. In all brevity, we recast each of these optimization problems in a Bayesian framework, introducing dedicated TS based solution schemes. For all of the addressed problems, the results show that besides being more effective, the TS based approaches we introduce are also capable of solving more adverse versions of the problems, such as dealing with stochastic liars.publishedVersio