State-Augmentation Transformations for Risk-Sensitive Markov Decision Processes

Markov decision processes (MDPs) provide a mathematical framework for modeling sequential decision making (SDM) where system evolution and reward are partly under the control of a decision maker and partly random. MDPs have been widely adopted in numerous fields, such as finance, robotics, manufacturing, and control systems. For stochastic control problems, MDPs serve as the underlying models in dynamic programming and reinforcement learning (RL) algorithms. In this thesis, we study risk estimation in MDPs, where the variability of random rewards is taken into account. First, we categorize the reward into four classes: deterministic/stochastic and state-/transition-based. Though numerous of theoretical methods are designed for MDPs or Markov processes with a deterministic (and state-based) reward, many practical problems are naturally modeled by processes with stochastic (and transition-based) reward. When the optimality criterion refers to the risk-neutral expectation of a (discount) total reward, we can use a model (reward) simplification to bridge the gap. However, when the criterion is risk-sensitive, a model simplification will change the risk value. For preserving the risks, we address that most, if not all, the inherent risk measures depend on the reward sequence (Rt). In order to bridge the gap between theoretical methods and practical problems with respect to risk-sensitive criteria, we propose a state-augmentation transformation (SAT). Four cases are thoroughly studied in which different forms of SAT should be implemented for risk preservation. In numerical experiments, we compare the results from the model simplifications and the SAT, and illustrate that, i). the model simplifications change (Rt) as well as return (or total reward) distributions; and ii). the proposed SAT transforms processes with complicated rewards, such as stochastic and transition-based rewards, into ones with deterministic state-based rewards, with intact (Rt). Second, we consider constrained risk-sensitive SDM problems in dynamic environments. Unlike other studies, we simultaneously consider the three factors—constraint, risk, and dynamic environment. We propose a scheme to generate a synthetic dataset for training an approximator. The reasons for not using historical data are two-fold. The first reason refers to information incompleteness. Historical data usually contains no information on criterion parameters (which risk objective and constraint(s) are concerned) and (or) the optimal policy (usually just an action for each item of data), and in many cases, even the information on environmental parameters (such as all involved costs) is incomplete. The second reason is about optimality. The decision makers might prefer an easy-to-use policy than an optimal one, which is hard to determine whether the preferred policy is optimal (such as an EOQ policy), since the practical problems could be different from the theoretical model diversely and subtly. Therefore, we propose to evaluate or estimate risk measures with RL methods and train an approximator, such as neural network, with a synthetic dataset. A numerical experiment validates the proposed scheme. The contributions of this study are three-fold. First, for risk evaluation in different cases, we propose the SAT theorem and corollaries to enable theoretical methods to solve practical problems with a preserved (Rt). Second, we estimate three risk measures with return variance as examples to illustrate the difference between the results from the SAT and the model simplification. Third, we present a scheme for constrained, risk-sensitive SDM problems in a dynamic environment with an inventory control example

Deterministic and Random Isogeometric Analysis of Fluid Flow in Unsaturated Soils

The main objective of this research is to use IGA as an efficient and robust alternative for numerical simulation of unsaturated seepage problems. Moreover, this research develops an IGA-based probabilistic framework that can properly account for the variability of soil hydraulic properties in the simulations. In the first part, IGA is used in a deterministic framework to solve a head-based form of Richards’ equation. It is shown that IGA is able to properly simulate changes in pore pressure at the soils interface. In the second part of this research, a new probabilistic framework, named random IGA (RIGA), is developed. A joint lognormal distribution function is used with IGA to perform Monte Carlo simulations. The results depict the statistical outputs relating to seepage quantities and pore water pressure. It is shown that pore water pressure, flow rate, etc. change considerably with respect to standard deviation and correlation of the model parameters

Strategies to address structural issues in hydrologic models

This thesis explores approaches for improving predictions of conceptual hydrologic models by addressing their structural issues, which typically arise from simplifications that occur when modelling complex hydrologic processes. The thesis documents the research undertaken to develop a generic and defensible modelling strategy. Hydrologic modelling is particularly challenging due to non-stationarity and limitations of data and in the methods used to analyse uncertainties. The thesis proposes two broad categories of methods to address these issues: (1) state uncertainty estimation and (2) model structure modification. A new Bayesian framework for estimating state uncertainty in hydrologic models was developed in Chapter 2, which was demonstrated in a synthetic study and a real-world case in the Bates catchment, Western Australia. This provided the theoretical underpinnings for Chapter 3, which presented a more practical approach named State and Parameter Uncertainty Estimation (SPUE). SPUE was applied to 46 catchments in Australia using the rainfall runoff model, GR4J. SPUE outperformed the classical approach of solely estimating uncertainty in hydrologic parameters based on validation fit to observed streamflow, reliability, and precision metrics. Chapter 4 explored a different approach, where direct modifications to the model structure were made to a river reach model, creating the River Bed/Bank Storage (RBS) model. This had the added advantage of more accurately representing the actual hydrological systems and their dynamic response to changing environmental conditions. In Chapter 5, the RBS model was further improved by combining it with SPUE, which resulted in better reliability and improved probability distributions. Overall, the thesis demonstrated that a comprehensive uncertainty formulation is essential for more accurate predictions of hydrologic models, and the use of both state uncertainty estimation and model structure modification methods can significantly enhance model performance

Approximations and inference of dynamics on networks

The study of dynamics on networks is a subject area that has wide-reaching applications in areas such as epidemic outbreaks, rumour spreading, and innovation diffusion. In this thesis I look at how to both approximate and infer these dynamics. Specifically, I first explore mean-field approximations for SIS epidemic dynamics. I outline several established approximations of varying complexity, before investigating how their accuracy depends on the network and dynamical parameters. Next, I use a method called approximate lumping to coarse-grain SIS dynamics, and I show how this method allows us to derive mean-field approximations directly from the full master equation description, rather than via ad hoc moment closures, as is common. Finally, I consider inference of network dynamic parameters on multilayer networks. I focus on a case study of SIS dynamics occurring on a two-layer network, where the dynamics on one of the layers is unobserved or “hidden”. My goal is to estimate the SIS parameters, assuming I only have data about the events occurring on the visible layer. To do this I develop several simpler approximate models of the dynamics which have tractable likelihoods, and then use Markov chain Monte Carlo routines to infer the most likely parameters for these approximate dynamics

Merging Data Sources to Predict Remaining Useful Life – An Automated Method to Identify Prognostic Parameters

The ultimate goal of most prognostic systems is accurate prediction of the remaining useful life (RUL) of individual systems or components based on their use and performance. This class of prognostic algorithms is termed Degradation-Based, or Type III Prognostics. As equipment degrades, measured parameters of the system tend to change; these sensed measurements, or appropriate transformations thereof, may be used to characterize degradation. Traditionally, individual-based prognostic methods use a measure of degradation to make RUL estimates. Degradation measures may include sensed measurements, such as temperature or vibration level, or inferred measurements, such as model residuals or physics-based model predictions. Often, it is beneficial to combine several measures of degradation into a single parameter. Selection of an appropriate parameter is key for making useful individual-based RUL estimates, but methods to aid in this selection are absent in the literature. This dissertation introduces a set of metrics which characterize the suitability of a prognostic parameter. Parameter features such as trendability, monotonicity, and prognosability can be used to compare candidate prognostic parameters to determine which is most useful for individual-based prognosis. Trendability indicates the degree to which the parameters of a population of systems have the same underlying shape. Monotonicity characterizes the underlying positive or negative trend of the parameter. Finally, prognosability gives a measure of the variance in the critical failure value of a population of systems. By quantifying these features for a given parameter, the metrics can be used with any traditional optimization technique, such as Genetic Algorithms, to identify the optimal parameter for a given system. An appropriate parameter may be used with a General Path Model (GPM) approach to make RUL estimates for specific systems or components. A dynamic Bayesian updating methodology is introduced to incorporate prior information in the GPM methodology. The proposed methods are illustrated with two applications: first, to the simulated turbofan engine data provided in the 2008 Prognostics and Health Management Conference Prognostics Challenge and, second, to data collected in a laboratory milling equipment wear experiment. The automated system was shown to identify appropriate parameters in both situations and facilitate Type III prognostic model development

Analysis of Models for Epidemiologic and Survival Data

Mortality statistics are useful tools for public-health statisticians, actuaries and policy makers to study health status of populations in communities and to make plans in health care systems. Several statistical models and methods of parameter estimation have been proposed. In this thesis, we review some benchmark mortality models and propose three alternative statistical models for both epidemiologic data and survival data. For epidemiologic data, we propose two statistical models, a Smoothed Segmented Lee-Carter model and a Smoothed Segmented Poisson Log-bilinear model. The models are modifications of the Lee-Carter (1992) model which combine an age segmented Lee-Carter parameterization with spline smoothed period effects within each age segment. With different period effects across age groups, the two models are fitted by maximizing respectively a penalized least squares criterion and a penalized Poisson likelihood. The new methods are applied to the 1971-2006 public-use mortality data sets released by the National Center for Health Statistics (NCHS). Mortality rates for three leading causes of death, heart diseases, cancer and accidents, are studied. For survival data, we propose a phase type model having features of mixtures, multiple stages or hits and a trapping state. Two parameter estimation techniques studied are a direct numerical method and an EM algorithm. Since phase type model parameters are known to be difficult to estimate, we study in detail the performance of our parameter estimation techniques by reference to the Fisher Information matrix. An alternative way to produce a Fisher Information matrix for an EM parameter estimation is also provided. The proposed model and the best available parameter estimation techniques are applied to a large SEER 1992-2002 breast cancer dataset

ISIPTA'07: Proceedings of the Fifth International Symposium on Imprecise Probability: Theories and Applications

B

An overview of clustering methods with guidelines for application in mental health research

Cluster analyzes have been widely used in mental health research to decompose inter-individual heterogeneity by identifying more homogeneous subgroups of individuals. However, despite advances in new algorithms and increasing popularity, there is little guidance on model choice, analytical framework and reporting requirements. In this paper, we aimed to address this gap by introducing the philosophy, design, advantages/disadvantages and implementation of major algorithms that are particularly relevant in mental health research. Extensions of basic models, such as kernel methods, deep learning, semi-supervised clustering, and clustering ensembles are subsequently introduced. How to choose algorithms to address common issues as well as methods for pre-clustering data processing, clustering evaluation and validation are then discussed. Importantly, we also provide general guidance on clustering workflow and reporting requirements. To facilitate the implementation of different algorithms, we provide information on R functions and librarie

Enhanced understanding of protein glycosylation in CHO cells through computational tools and experimentation

Chinese hamster ovary (CHO) cells are the workhorse of the multibillion-dollar biopharmaceuticals industry. They have been extensively harnessed for recombinant protein synthesis, as they exhibit high titres and human-like post translational modifications (PTM), such as protein N-linked glycosylation. More specifically, N-linked glycosylation is a crucial PTM that includes the addition of an oligosaccharide in the backbone of the protein and strongly affects therapeutic efficacy and immunogenicity. In addition, the Quality by Design (QbD) paradigm that is broadly applied in academic research, necessitates a comprehensive understanding of the underlying biological relationships between the process parameters and the product quality attributes. To that end, computational tools have been vastly employed to elucidate cellular functions and predict the effect of process parameters on cell growth, product synthesis and quality. This thesis reports several advancements in the use of mathematical models for describing and optimizing bioprocesses. Firstly, a kinetic mathematical model describing CHO cell growth, metabolism, antibody synthesis and N-linked glycosylation was proposed, in order to capture the effect of galactose and uridine supplementation on cell growth and monoclonal antibody (mAb) glycosylation. Subsequently, the model was utilized to optimize galactosylation, a desired quality attribute of therapeutic mAbs. Following the QbD paradigm for ensuring product titre and quality, the kinetic model was subsequently used to identify an in silico Design Space (DS) that was also experimentally verified. An elaborate parameter estimation methodology was also developed in order to adapt the existing model to data from a newly introduced CHO cell line, without altering model structure. In an effort to reduce the burden of parameter estimation, the N-linked glycosylation submodel was replaced with an artificial neural network that was used as a standalone machine learning algorithm to predict the effect of feeding alterations and genetic engineering on the glycan distribution of several therapeutic proteins. In addition, a hybrid model configuration (HyGlycoM) incorporating the ANN-glycosylation model was also formulated to link extracellular process conditions to glycan distribution. The latter was found to outperform its fully kinetic equivalent when compared to experimental data. Finally, a comprehensive investigation of mAb galactosylation bottlenecks was carried out. Five fed-batch experiments with different concentrations of galactose and uridine supplemented throughout the culturing period, were carried out and were found to present similar mAb galactosylation. In order to identify the bottlenecks that limit galactosylation, further experimental analysis, including the investigation of glycans microheterogeneity of CHO host cell proteins (HCPs), was conducted. The experimental results were used to parameterize a novel and significant extension of the kinetic glycosylation model that simultaneously describes the N-linked glycosylation of both HCPs and the mAb product. Flux balance analysis was also used to analyse carbon and nitrogen metabolism using the experimental amino acid concentration profiles. In addition to the expression levels of the beta-1,4-galactosyltransferase enzyme, constraints imposed by the transport of the galactosylation sugar donor in the Golgi compartments and the consumption of resources towards HCPs glycosylation, were found to considerably influence mAb galactosylation.Open Acces