Linear programming-based solution methods for constrained partially observable Markov decision processes

Constrained partially observable Markov decision processes (CPOMDPs) have been used to model various real-world phenomena. However, they are notoriously difficult to solve to optimality, and there exist only a few approximation methods for obtaining high-quality solutions. In this study, grid-based approximations are used in combination with linear programming (LP) models to generate approximate policies for CPOMDPs. A detailed numerical study is conducted with six CPOMDP problem instances considering both their finite and infinite horizon formulations. The quality of approximation algorithms for solving unconstrained POMDP problems is established through a comparative analysis with exact solution methods. Then, the performance of the LP-based CPOMDP solution approaches for varying budget levels is evaluated. Finally, the flexibility of LP-based approaches is demonstrated by applying deterministic policy constraints, and a detailed investigation into their impact on rewards and CPU run time is provided. For most of the finite horizon problems, deterministic policy constraints are found to have little impact on expected reward, but they introduce a significant increase to CPU run time. For infinite horizon problems, the reverse is observed: deterministic policies tend to yield lower expected total rewards than their stochastic counterparts, but the impact of deterministic constraints on CPU run time is negligible in this case. Overall, these results demonstrate that LP models can effectively generate approximate policies for both finite and infinite horizon problems while providing the flexibility to incorporate various additional constraints into the underlying model.Comment: 42 pages, 8 figure

Hidden Markov Model Based Visual Perception Filtering in Robotic Soccer

Autonomous robots can initiate their mission plans only after gathering sufficient information about the environment. Therefore reliable perception information plays a major role in the overall success of an autonomous robot. The Hidden Markov Model based post-perception filtering module proposed in this paper aims to identify and remove spurious perception information in a given perception sequence using the generic meta-pose definition. This method allows representing uncertainty in more abstract terms compared to the common physical representations. Our experiments with the four legged AIBO robot indicated that the proposed module improved perception and localization performance significantly

Enhancing the construction safety training by using virtual environment : V-SAFE

Construction is one of the most high-risk industries in the world. The safety records of the construction sector report that, construction workers are approximately over three times more likely to be exposed to serious accidents comparing to other industries. In addition to these injuries and fatalities, work-related accidents also cause financial damage, conflictual cases and pecuniary penalties for the construction companies. Therefore, the significance of the safety management process has been increasing in the construction industry. However, since the majority of the construction activities are complex and require collaboration between the workers, the provision of the safety has become one of the challenging tasks. So, the behavior-based skills of the workers play a crucial role in the safety management. Traditional safety training methods have been merely focusing on information-based techniques such as lectures, videos, written materials, etc. On the other hand, previous research has indicated that adequate safety training should also involve behavioral modeling and hands-on training, together with traditional learning methods. Due to the nature of the construction projects, hands-on training in the construction field is not practicable. In this sense, using virtual environments is an effective method that enables a safe environment for the users without being exposed to adverse effects of the failed tasks. Thus, virtual environments allow visual simulation that is helpful for the improvement of the trainees’ behavior-based skills. Therefore, virtual environments provide an important opportunity to advance the level of safety training. The main aim of the study is to describe the developed virtual safety training environment called V-SAFE (Virtual Safety Analysis For Engineering applications), which involves methods to simulate, and visualize construction operation scenarios. V-SAFE is based on the Unreal game engine for the visualization of the environment, and USARSim is used for the high-fidelity simulation of the robot behavior and environment mapping. V-SAFE is projected to establish a base to identify construction-specific safety risks and to improve the behavior-based skills of the construction project participants. In brief, V-SAFE has high potential to improve the risk recognition capability, and situational awareness of the construction managers, workers, safety managers, field engineers. So, V-SAFE could be beneficial for the construction organizations aim to advance the effectiveness of the safety training.Non UBCUnreviewedFacultyOthe

A multi-objective constrained POMDP model for breast cancer screening

Breast cancer is a common and deadly disease, but it is often curable when diagnosed early. While most countries have large-scale screening programs, there is no consensus on a single globally accepted policy for breast cancer screening. The complex nature of the disease; limited availability of screening methods such as mammography, magnetic resonance imaging (MRI), and ultrasound screening; and public health policies all factor into the development of screening policies. Resource availability concerns necessitate the design of policies which conform to a budget, a problem which can be modelled as a constrained partially observable Markov decision process (CPOMDP). In this study, we propose a multi-objective CPOMDP model for breast cancer screening with two objectives: minimize the lifetime risk of dying due to breast cancer and maximize the quality-adjusted life years. Additionally, we consider an expanded action space which allows for screening methods beyond mammography. Each action has a unique impact on quality-adjusted life years and lifetime risk, as well as a unique cost. Our results reveal the Pareto frontier of optimal solutions for average and high risk patients at different budget levels, which can be used by decision makers to set policies in practice