Performance Guarantee of a Sub-Optimal Policy for a Discrete Markov Decision Process and Its Application to a Robotic Surveillance Problem

This dissertation deals with the development and analysis of sub-optimal decision algorithms for a collection of robots that assist a remotely located operator in perimeter surveillance. The operator is tasked with the classification of incursions across the perimeter. Whenever there is an incursion into the perimeter, a nearby Unattended Ground Sensor (UGS) signals an alert. A robot services the alert by visiting the alert location, collecting evidence in the form of video imagery, and transmitting it to the operator. The accuracy of operator's classification depends on the volume and freshness of information gathered and provided by the robots at locations where incursions occur. There are two competing needs for a robot: it needs to spend adequate time at an alert location to collect evidence for aiding the operator in accurate classification but it also needs to service other alerts as soon as possible, so that the evidence collected is relevant. The control problem is to determine the optimal amount of time a robot must spend servicing an alert. The incursions are stochastic and their statistics are assumed to be known. The control problem may be posed as a Markov Decision Problem (MDP). Dynamic Programming(DP) provides the optimal policy to the MDP. However, because of the "curse of dimensionality" of DP, finding the optimal policy is not practical in many applications. For a perimeter surveillance problem with two robots and five UGS locations, the number of states is of the order of billions. Approximate Dynamic Programming (ADP) via Linear Programming (LP) provides a way to approximate the value function and derive sub-optimal strategies. Using state partitioning and ADP, this dissertation provides different LP formulations for upper and lower bounds to the value function of the MDP, and shows the relationship between LPs and MDP. The novel features of this dissertation are (1) the derivation of a tractable lower bound via LP and state partitioning, (2) the construction of a sub-optimal policy whose performance exceeds the lower bound, and (3) the derivation of an upper bound using a non-linear programming formuation. The upper and lower bounds provides approximation ratio to the value function. Finally, illustrative perimeter surveillance examples corroborate the results derived in this dissertation

QoS Provision for Wireless Sensor Networks

Wireless sensor network is a fast growing area of research, receiving attention not only within the computer science and electrical engineering communities, but also in relation to network optimization, scheduling, risk and reliability analysis within industrial and system engineering. The availability of micro-sensors and low-power wireless communications will enable the deployment of densely distributed sensor/actuator networks. And an integration of such system plays critical roles in many facets of human life ranging from intelligent assistants in hospitals to manufacturing process, to rescue agents in large scale disaster response, to sensor networks tracking environment phenomena, and others. The sensor nodes will perform significant signal processing, computation, and network self-configuration to achieve scalable, secure, robust and long-lived networks. More specifically, sensor nodes will do local processing to reduce energy costs, and key exchanges to ensure robust communications. These requirements pose interesting challenges for networking research. The most important technical challenge arises from the development of an integrated system which is 1)energy efficient because the system must be long-lived and operate without manual intervention, 2)reliable for data communication and robust to attackers because information security and system robustness are important in sensitive applications, such as military. Based on the above challenges, this dissertation provides Quality of Service (QoS) implementation and evaluation for the wireless sensor networks. It includes the following 3 modules, 1) energy-efficient routing, 2) energy-efficient coverage, 3). communication security. Energy-efficient routing combines the features of minimum energy consumption routing protocols with minimum computational cost routing protocols. Energy-efficient coverage provides on-demand sensing and measurement. Information security needs a security key exchange scheme to ensure reliable and robust communication links. QoS evaluation metrics and results are presented based on the above requirements

Adaptive policies for perimeter surveillance problems

We consider the problem of sequentially choosing observation regions along a line, with an aim of maximising the detection of events of interest. Such a problem may arise when monitoring the movements of endangered or migratory species, detecting crossings of a border, policing activities at sea, and in many other settings. In each case, the key operational challenge is to learn an allocation of surveillance resources which maximises successful detection of events of interest. We present a combinatorial multi-armed bandit model with Poisson rewards and a novel filtered feedback mechanism - arising from the failure to detect certain intrusions - where reward distributions are dependent on the actions selected. Our solution method is an upper confidence bound approach and we derive upper and lower bounds on its expected performance. We prove that the gap between these bounds is of constant order, and demonstrate empirically that our approach is more reliable in simulated problems than competing algorithms

A stochastic mixed integer programming approach to wildfire management systems

Wildfires have become more destructive and are seriously threatening societies and our ecosystems throughout the world. Once a wildfire escapes from its initial suppression attack, it can easily develop into a destructive huge fire that can result in significant loss of lives and resources. Some human-caused wildfires may be prevented; however, most nature-caused wildfires cannot. Consequently, wildfire suppression and contain- ment becomes fundamentally important; but suppressing and containing wildfires is costly. Since the budget and resources for wildfire management are constrained in reality, it is imperative to make important decisions such that the total cost and damage associated with the wildfire is minimized while wildfire containment effectiveness is maximized. To achieve this objective, wildfire attack-bases should be optimally located such that any wildfire is suppressed within the effective attack range from some bases. In addition, the optimal fire-fighting resources should be deployed to the wildfire location such that it is efficiently suppressed from an economic perspective. The two main uncertain/stochastic factors in wildfire management problems are fire occurrence frequency and fire growth characteristics. In this thesis two models for wildfire management planning are proposed. The first model is a strategic model for the optimal location of wildfire-attack bases under uncertainty in fire occurrence. The second model is a tactical model for the optimal deployment of fire-fighting resources under uncertainty in fire growth. A stochastic mixed-integer programming approach is proposed in order to take into account the uncertainty in the problem data and to allow for robust wildfire management decisions under uncertainty. For computational results, the tactical decision model is numerically experimented by two different approaches to provide the more efficient method for solving the model

Poisson process bandits:Sequential models and algorithms for maximising the detection of point data

In numerous settings in areas as diverse as security, ecology, astronomy, and logistics, it is desirable to optimally deploy a limited resource to observe events, which may be modelled as point data arising according to a Non-homogeneous Poisson process. Increasingly, thanks to developments in mobile and adaptive technologies, it is possible to update a deployment of such resource and gather feedback on the quality of multiple actions. Such a capability presents the opportunity to learn, and with it a classic problem in operations research and machine learning - the explorationexploitation dilemma. To perform optimally, how should investigative choices which explore the value of poorly understood actions and optimising choices which choose actions known to be of a high value be balanced? Effective techniques exist to resolve this dilemma in simpler settings, but the Poisson process data brings new challenges. In this thesis, effective solution methods for the problem of sequentially deploying resource are developed, via a combination of efficient inference schemes, bespoke optimisation approaches, and advanced sequential decision-making strategies. Furthermore, extensive theoretical work provides strong guarantees on the performance of the proposed solution methods and an understanding of the challenges of this problem and more complex extensions. In particular, Upper Confidence Bound and Thompson Sampling (TS) approaches are derived for combinatorial and continuum-armed bandit versions of the problem, with accompanying analysis displaying that the regret of the approaches is of optimal order. A broader understanding of the performance of TS based on non-parametric models for smooth reward functions is developed, and new posterior contraction results for the Gaussian Cox Process, a popular Bayesian non-parametric model of point data, are derived. These results point to effective strategies for more challenging variants of the event detection problem, and more generally advance the understanding of bandit decision-making with complex data structures

Constrained Task Assignment and Scheduling on Networks of Arbitrary Topology.

This dissertation develops a framework to address centralized and distributed constrained task assignment and task scheduling problems. This framework is used to prove properties of these problems that can be exploited, develop effective solution algorithms, and to prove important properties such as correctness, completeness and optimality. The centralized task assignment and task scheduling problem treated here is expressed as a vehicle routing problem with the goal of optimizing mission time subject to mission constraints on task precedence and agent capability. The algorithm developed to solve this problem is able to coordinate vehicle (agent) timing for task completion. This class of problems is NP-hard and analytical guarantees on solution quality are often unavailable. This dissertation develops a technique for determining solution quality that can be used on a large class of problems and does not rely on traditional analytical guarantees. For distributed problems several agents must communicate to collectively solve a distributed task assignment and task scheduling problem. The distributed task assignment and task scheduling algorithms developed here allow for the optimization of constrained military missions in situations where the communication network may be incomplete and only locally known. Two problems are developed. The distributed task assignment problem incorporates communication constraints that must be satisfied; this is the Communication-Constrained Distributed Assignment Problem. A novel distributed assignment algorithm, the Stochastic Bidding Algorithm, solves this problem. The algorithm is correct, probabilistically complete, and has linear average-case time complexity. The distributed task scheduling problem addressed here is to minimize mission time subject to arbitrary predicate mission constraints; this is the Minimum-time Arbitrarily-constrained Distributed Scheduling Problem. The Optimal Distributed Non-sequential Backtracking Algorithm solves this problem. The algorithm is correct, complete, outputs time optimal schedules, and has low average-case time complexity. Separation of the task assignment and task scheduling problems is exploited here to ameliorate the effects of an incomplete communication network. The mission-modeling conditions that allow this and the benefits gained are discussed in detail. It is shown that the distributed task assignment and task scheduling algorithms developed here can operate concurrently and maintain their correctness, completeness, and optimality properties.Ph.D.Aerospace EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/91527/1/jpjack_1.pd