A New Optimal Stepsize For Approximate Dynamic Programming

Approximate dynamic programming (ADP) has proven itself in a wide range of applications spanning large-scale transportation problems, health care, revenue management, and energy systems. The design of effective ADP algorithms has many dimensions, but one crucial factor is the stepsize rule used to update a value function approximation. Many operations research applications are computationally intensive, and it is important to obtain good results quickly. Furthermore, the most popular stepsize formulas use tunable parameters and can produce very poor results if tuned improperly. We derive a new stepsize rule that optimizes the prediction error in order to improve the short-term performance of an ADP algorithm. With only one, relatively insensitive tunable parameter, the new rule adapts to the level of noise in the problem and produces faster convergence in numerical experiments.Comment: Matlab files are included with the paper sourc

Markov Decision Processes

The theory of Markov Decision Processes is the theory of controlled Markov chains. Its origins can be traced back to R. Bellman and L. Shapley in the 1950\u27s. During the decades of the last century this theory has grown dramatically. It has found applications in various areas like e.g. computer science, engineering, operations research, biology and economics. In this article we give a short introduction to parts of this theory. We treat Markov Decision Processes with finite and infinite time horizon where we will restrict the presentation to the so-called (generalized) negative case. Solution algorithms like Howard\u27s policy improvement and linear programming are also explained. Various examples show the application of the theory. We treat stochastic linear-quadratic control problems, bandit problems and dividend pay-out problems

A matter of timing : A modelling-based investigation of the dynamic behaviour of reproductive hormones in girls and women

Hypothalamus-hypofyse-gonade aksen er en del av det kvinnelige endokrine systemet, og regulerer evnen til reproduksjon. Hormoner produsert og utskilt fra tre kjertler (hypotalamus, hypofysen, eggstokkene) påvirker hverandre via tilbakemeldingsinteraksjoner, som er nødvendige for å etablere en regelmessig menstruasjonssyklus hos kvinner. Matematiske modeller som forutsier utviklingen av slike hormonkonsentrasjoner og modning av eggstokkfollikler er nyttige verktøy for å forstå menstruasjonssyklusens dynamiske oppførsel. Slike modeller kan for eksempel hjelpe oss med å undersøke patologiske tilstander som endometriose og polycystisk ovariesyndrom. Videre kan de brukes til systematiske undersøkelser av effekten av medikamenter på det kvinnelige endokrine systemet. Derfor kan vi potensielt bruke slike menstruasjonsyklusmodeller som kliniske beslutningsstøttessystemer. Vi trenger modeller som forutsier hormonkonsentrasjoner sammen med modningen av eggstokkfollikler hos enkeltindivider gjennom påfølgende sykluser. Dette for å kunne simulere hormonelle behandlinger som stimulerer vekst av eggstokkfolliklene (eggstokkstimuleringsprotokoller). Her legger jeg fram et forslag til en matematisk menstruasjonsyklusmodell og viser modellens evne til å forutsi resultatet av eggstokkstimuleringsprotokoller. For å kalibrere denne typen modell trenges individuelle tidsseriedata. Innsamling av slike data er tidskrevende, og forutsetter høy grad av engasjement fra deltakerne i studien. Det er derfor viktig å finne brukbare datatyper som er mindre tid- og ressurskrevende å samle inn, og som likevel kan brukes til modellkalibrering. En type data som er enklere å samle inn er tversnittdata. I denne avhandlingen har jeg utviklet en prosedyre for å bruke tversnittpopulasjonsdata i modellens kalibreringsprosess, og viser hvordan en modell kalibrert med tversnittdata kan brukes til å forutsi individuelle resultater ved oppdatering av en del av modellens parametere. I tillegg til det vitenskapelige bidraget, håper jeg at avhandlingen min skaper oppmerksomhet rundt viktigheten av forskning på kvinners reproduktive helse, og at avhandlingen underbygger verdien av matematiske modeller i forskning på kvinnehelse.The hypothalamic-pituitary-gonadal axis (HPG axis), a part of the human endocrine system, regulates the female reproductive function. Feedback interactions between hormones secreted from the glands forming the HPG axis are essential for establishing a regular menstrual cycle. Mathematical models predicting the time evolution of hormone concentrations and the maturation of ovarian follicles are useful tools for understanding the dynamic behaviour of the menstrual cycle. Such models can, for example, help us to investigate pathological conditions, such as endometriosis or Polycystic Ovary Syndrome. Furthermore, they can be used to systematically study the effects of drugs on the endocrine system. In doing so, menstrual cycle models could potentially be integrated into clinical routines as clinical decision support systems. For the simulation-based investigation of hormonal treatments aiming to stimulate the growth of ovarian follicles (Controlled Ovarian Stimulation (COS)), we need models that predict hormone concentrations and the maturation of ovarian follicles in biological units throughout consecutive cycles. Here, I propose such a mechanistic menstrual cycle model. I also demonstrate its capability to predict the outcome of COS. Individual time series data is usually used to calibrate mechanistic models having clinical implications. Collecting these data, however, is time-consuming and requires a high commitment from study participants. Therefore, integrating different data sets into the model calibration process is of interest. One type of data that is often more feasible to collect than individual time series is cross-sectional data. As part of my thesis, I developed a workflow based on Bayesian updating to integrate cross-sectional data into the model calibration process. I demonstrate the workflow using a mechanistic model describing the time evolution of reproductive hormones during puberty in girls. Exemplary, I show that a model calibrated with cross-sectional data can be used to predict individual dynamics after updating a subset of model parameters. In addition to the scientific contributions of this thesis, I hope that it creates attention for the importance of research in the area of women's reproductive health and underpins the value of mathematical modelling for this field.Doktorgradsavhandlin

Dynamic cash management models

Classical cash management models concern how an organisation should maintain their (liquid) cash balances in order to meet cash demands over time. In these models the balance can be increased or decreased to offset penalties for not being able to meet a cash demand or the opportunity cost of holding too much cash, respectively. The external source from which this money comes from or is sent to is not explicitly modelled but is assumed to be available at all times. In this thesis we contribute to the cash management problem by discussing three novel cash management models. To begin with, we include a second asset to the cash management model and assume the cash inflows are generated from this asset. We formulate this problem as a discrete Markov decision process (MDP) and solve it by the classic backward iteration method. We show that the optimal cash policy for this model possesses the two-threshold two-target form. Moreover we observe that the agent should take a ‘safer’ cash policy when the company has a balanced cash inflows and outflows. Then we introduce loan opportunities to the model. In this problem, we allow the agent taking loans from financial intermediates. We assume there is one type of unsecured loan with fixed interest rate and the manager can take this loan repeatedly once his previous debt is paid off. We also solve this model via the discrete MDP approach. Moreover we propose a heuristic for this problem based on the policy improvement which is shown to perform strongly in our experiments. At last, we consider an agent managing a cash account and a number of as- sets accounts. Hence both cash policies and asset allocation policies are studied simultaneously. Moreover we assume the agent wishes to pursuit the net profits while controlling the risk associated with his management strategies. We solve this model using a separable Piecewise linear approximate dynamic programming approach. We also provide a heuristic based on the myopic greedy algorithm and the discrete MDP approach as benchmarks. The numerical experiments show that the PWL ADP outperforms the heuristic in terms of objective values and takes significantly less solution time comparing with the discrete MDP

A longitudinal study of the experiences and psychological well-being of Indian surrogates

Study question: What is the psychological well-being of Indian surrogates during and after the surrogacy pregnancy? Summary answer: Surrogates were similar to a matched group of expectant mothers on anxiety and stress. However, they scored higher on depression during and after pregnancy. What is known already: The recent ban on trans-national commercial surrogacy in India has led to urgent policy discussions regarding surrogacy. Whilst previous studies have reported the motivations and experiences of Indian surrogates no studies have systematically examined the psychological well-being of Indian surrogates, especially from a longitudinal perspective. Previous research has shown that Indian surrogates are motivated by financial payment and may face criticism from their family and community due to negative social stigma attached to surrogacy. Indian surrogates often recruited by agencies and mainly live together in a “surrogacy house.” Study design, size, duration: A longitudinal study was conducted comparing surrogates to a matched group of expectant mothers over two time points: (a) during pregnancy (Phase1: 50 surrogates, 70 expectant mothers) and (b) 4–6 months after delivery (Phase 2: 45 surrogates, 49 expectant mothers). The Surrogates were recruited from a fertility clinic in Mumbai and the matched comparison group was recruited from four public hospitals in Mumbai and Delhi. Data collection was completed over 2 years. Participants/materials, setting, methods: Surrogates and expectant mothers were aged between 23 and 36 years. All participants were from a low socio-economic background and had left school before 12–13 years of age. In-depth faceto-face semi-structured interviews and a psychological questionnaire assessing anxiety, stress and depression were administered in Hindi to both groups. Interviews took place in a private setting. Audio recordings of surrogate interviews were later translated and transcribed into English. Main results and the role of chance: Stress and anxiety levels did not significantly differ between the two groups for both phases of the study. For depression, surrogates were found to be significantly more depressed than expectant mothers at phase 1 (p = 0.012) and phase 2 (p = 0.017). Within the surrogacy group, stress and depression did not change during and after pregnancy. However, a non-significant trend was found showing that anxiety decreased after delivery (p = 0.086). No participants reported being coerced into surrogacy, however nearly all kept it a secret from their wider family and community and hence did not face criticism. Surrogates lived at the surrogate house for different durations. During pregnancy, 66% (N = 33/50) reported their experiences of the surrogate house as positive, 24% (N = 12/50) as negative and 10% (N = 5/50) as neutral. After delivery, most surrogates (66%, N = 30/45) reported their experiences of surrogacy to be positive, with the remainder viewing it as neutral (28%) or negative (4%). In addition, most (66%, N = 30/45) reported that they had felt “socially supported and loved” during the surrogacy arrangement by friends in the surrogate hostel, clinic staff or family. Most surrogates did not meet the intending parents (49%, N = 22/45) or the resultant child (75%, N = 34/45). Limitations, reasons for caution: Since the surrogates were recruited from only one clinic, the findings may not be representative of all Indian surrogates. Some were lost to follow-up which may have produced sampling bias. Wider implications of the findings: This is the first study to examine the psychological well-being of surrogates in India. This research is of relevance to current policy discussions in India regarding legislation on surrogacy. Moreover, the findings are of relevance to clinicians, counselors and other professionals involved in surrogacy. Trial registration number: N/A