BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations

Real-world planning problems\unicode{x2014}including autonomous driving and sustainable energy applications like carbon storage and resource exploration\unicode{x2014}have recently been modeled as partially observable Markov decision processes (POMDPs) and solved using approximate methods. To solve high-dimensional POMDPs in practice, state-of-the-art methods use online planning with problem-specific heuristics to reduce planning horizons and make the problems tractable. Algorithms that learn approximations to replace heuristics have recently found success in large-scale problems in the fully observable domain. The key insight is the combination of online Monte Carlo tree search with offline neural network approximations of the optimal policy and value function. In this work, we bring this insight to partially observed domains and propose BetaZero, a belief-state planning algorithm for POMDPs. BetaZero learns offline approximations based on accurate belief models to enable online decision making in long-horizon problems. We address several challenges inherent in large-scale partially observable domains; namely challenges of transitioning in stochastic environments, prioritizing action branching with limited search budget, and representing beliefs as input to the network. We apply BetaZero to various well-established benchmark POMDPs found in the literature. As a real-world case study, we test BetaZero on the high-dimensional geological problem of critical mineral exploration. Experiments show that BetaZero outperforms state-of-the-art POMDP solvers on a variety of tasks.Comment: 20 page

Changing the script: a typology of Dutch theatre manuscripts in the Southern Low Countries, and the interaction between manuscript and print (Seventeenth–Eighteenth centuries)

NWO016.Veni.195.371Medieval and Early Modern Studie

Antitumour and antiangiogenic effects of Aplidin® in the 5TMM syngeneic models of multiple myeloma

Aplidin® is an antitumour drug, currently undergoing phase II evaluation in different haematological and solid tumours. In this study, we analysed the antimyeloma effects of Aplidin in the syngeneic 5T33MM model, which is representable for the human disease. In vitro, Aplidin inhibited 5T33MMvv DNA synthesis with an IC50 of 3.87 nM. On cell-cycle progression, the drug induced an arrest in transition from G0/G1 to S phase, while Western blot showed a decreased cyclin D1 and CDK4 expression. Furthermore, Aplidin induced apoptosis by lowering the mitochondrial membrane potential, by inducing cytochrome c release and by activating caspase-9 and caspase-3. For the in vivo experiment, 5T33MM-injected C57Bl/KaLwRij mice were intraperitoneally treated with vehicle or Aplidin (90 μg kg−1 daily). Chronic treatment with Aplidin was well tolerated and reduced serum paraprotein concentration by 42% (P<0.001), while BM invasion with myeloma cells was decreased by 35% (P<0.001). Aplidin also reduced the myeloma-associated angiogenesis to basal values. This antiangiogenic effect was confirmed in vitro and explained by inhibition of endothelial cell proliferation and vessel formation. These data indicate that Aplidin is well tolerated in vivo and its antitumour and antiangiogenic effects support the use of the drug in multiple myeloma

Extramedullary disease in multiple myeloma: a systematic literature review

Extramedullary involvement (or extramedullary disease, EMD) represents an aggressive form of multiple myeloma (MM), characterized by the ability of a clone and/or subclone to thrive and grow independent of the bone marrow microenvironment. Several different definitions of EMD have been used in the published literature. We advocate that true EMD is restricted to soft-tissue plasmacytomas that arise due to hematogenous spread and have no contact with bony structures. Typical sites of EMD vary according to the phase of MM. At diagnosis, EMD is typically found in skin and soft tissues; at relapse, typical sites involved include liver, kidneys, lymph nodes, central nervous system (CNS), breast, pleura, and pericardium. The reported incidence of EMD varies considerably, and differences in diagnostic approach between studies are likely to contribute to this variability. In patients with newly diagnosed MM, the reported incidence ranges from 0.5% to 4.8%, while in relapsed/refractory MM the reported incidence is 3.4 to 14%. Available data demonstrate that the prognosis is poor, and considerably worse than for MM without soft-tissue plasmacytomas. Among patients with plasmacytomas, those with EMD have poorer outcomes than those with paraskeletal involvement. CNS involvement is rare, but prognosis is even more dismal than for EMD in other locations, particularly if there is leptomeningeal involvement. Available data on treatment outcomes for EMD are derived almost entirely from retrospective studies. Some agents and combinations have shown a degree of efficacy but, as would be expected, this is less than in MM patients with no extramedullary involvement. The paucity of prospective studies makes it difficult to justify strong recommendations for any treatment approach. Prospective data from patients with clearly defined EMD are important for the optimal evaluation of treatment outcomes

A pattern-search-based inverse method

Uncertainty in model predictions is caused to a large extent by the uncertainty in model parameters, while the identification of model parameters is demanding because of the inherent heterogeneity of the aquifer. A variety of inverse methods has been proposed for parameter identification. In this paper we present a novel inverse method to constrain the model parameters (hydraulic conductivities) to the observed state data (hydraulic heads). In the method proposed we build a conditioning pattern consisting of simulated model parameters and observed flow data. The unknown parameter values are simulated by pattern searching through an ensemble of realizations rather than optimizing an objective function. The model parameters do not necessarily follow a multi-Gaussian distribution, and the nonlinear relationship between the parameter and the response is captured by the multipoint pattern matching. The algorithm is evaluated in two synthetic bimodal aquifers. The proposed method is able to reproduce the main structure of the reference fields, and the performance of the updated model in predicting flow and transport is improved compared with that of the prior model.The authors gratefully acknowledge the financial support from the Ministry of Science and Innovation, project CGL2011-23295. The first author also acknowledges the scholarship provided by the China Scholarship Council (CSC [2007] 3020). The authors would like to thank Gregoire Mariethoz (University of New South Wales) and Philippe Renard (University of Neuchatel) for their enthusiastic help in answering questions about the direct sampling algorithm. Gregoire Mariethoz and two anonymous reviewers are also thanked for their comments during the reviewing process, which helped improving the final paper.Zhou ., H.; Gómez-Hernández, JJ.; Li ., L. (2012). A pattern-search-based inverse method. Water Resources Research. 48(3):1-17. https://doi.org/10.1029/2011WR011195S117483Alcolea, A., & Renard, P. (2010). Blocking Moving Window algorithm: Conditioning multiple-point simulations to hydrogeological data. Water Resources Research, 46(8). doi:10.1029/2009wr007943Alcolea, A., Carrera, J., & Medina, A. (2006). Pilot points method incorporating prior information for solving the groundwater flow inverse problem. Advances in Water Resources, 29(11), 1678-1689. doi:10.1016/j.advwatres.2005.12.009Arpat, G. B., & Caers, J. (2007). Conditional Simulation with Patterns. Mathematical Geology, 39(2), 177-203. doi:10.1007/s11004-006-9075-3Caers , J. 2002 Geostatistical history matching under training-image based geological model constraintsCaers, J. (2003). Efficient gradual deformation using a streamline-based proxy method. Journal of Petroleum Science and Engineering, 39(1-2), 57-83. doi:10.1016/s0920-4105(03)00040-8Caers, J., & Hoffman, T. (2006). The Probability Perturbation Method: A New Look at Bayesian Inverse Modeling. Mathematical Geology, 38(1), 81-100. doi:10.1007/s11004-005-9005-9Capilla, J. E., & Llopis-Albert, C. (2009). Gradual conditioning of non-Gaussian transmissivity fields to flow and mass transport data: 1. Theory. Journal of Hydrology, 371(1-4), 66-74. doi:10.1016/j.jhydrol.2009.03.015Carrera, J., & Neuman, S. P. (1986). Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information. Water Resources Research, 22(2), 199-210. doi:10.1029/wr022i002p00199Chen, Y., & Zhang, D. (2006). Data assimilation for transient flow in geologic formations via ensemble Kalman filter. Advances in Water Resources, 29(8), 1107-1122. doi:10.1016/j.advwatres.2005.09.007Christiansen, L., Binning, P. J., Rosbjerg, D., Andersen, O. B., & Bauer-Gottwein, P. (2011). Using time-lapse gravity for groundwater model calibration: An application to alluvial aquifer storage. Water Resources Research, 47(6). doi:10.1029/2010wr009859De Marsily, G., Delay, F., Gonçalvès, J., Renard, P., Teles, V., & Violette, S. (2005). Dealing with spatial heterogeneity. Hydrogeology Journal, 13(1), 161-183. doi:10.1007/s10040-004-0432-3Deutsch, C. V., & Tran, T. T. (2002). FLUVSIM: a program for object-based stochastic modeling of fluvial depositional systems. Computers & Geosciences, 28(4), 525-535. doi:10.1016/s0098-3004(01)00075-9Dubuisson, M.-P., & Jain, A. K. (s. f.). A modified Hausdorff distance for object matching. Proceedings of 12th International Conference on Pattern Recognition. doi:10.1109/icpr.1994.576361Emsellem, Y., & De Marsily, G. (1971). An Automatic Solution for the Inverse Problem. Water Resources Research, 7(5), 1264-1283. doi:10.1029/wr007i005p01264Evensen, G. (2003). The Ensemble Kalman Filter: theoretical formulation and practical implementation. Ocean Dynamics, 53(4), 343-367. doi:10.1007/s10236-003-0036-9Falivene, O., Cabello, P., Arbués, P., Muñoz, J. A., & Cabrera, L. (2009). A geostatistical algorithm to reproduce lateral gradual facies transitions: Description and implementation. Computers & Geosciences, 35(8), 1642-1651. doi:10.1016/j.cageo.2008.12.003Fernàndez-Garcia, D., Illangasekare, T. H., & Rajaram, H. (2005). Differences in the scale dependence of dispersivity and retardation factors estimated from forced-gradient and uniform flow tracer tests in three-dimensional physically and chemically heterogeneous porous media. Water Resources Research, 41(3). doi:10.1029/2004wr003125Feyen, L., & Caers, J. (2006). Quantifying geological uncertainty for flow and transport modeling in multi-modal heterogeneous formations. Advances in Water Resources, 29(6), 912-929. doi:10.1016/j.advwatres.2005.08.002Fu, J., & Gómez-Hernández, J. J. (2008). A Blocking Markov Chain Monte Carlo Method for Inverse Stochastic Hydrogeological Modeling. Mathematical Geosciences, 41(2), 105-128. doi:10.1007/s11004-008-9206-0Jaime Gómez-Hernánez, J., Sahuquillo, A., & Capilla, J. (1997). Stochastic simulation of transmissivity fields conditional to both transmissivity and piezometric data—I. Theory. Journal of Hydrology, 203(1-4), 162-174. doi:10.1016/s0022-1694(97)00098-xGuardiano, F. B., & Srivastava, R. M. (1993). Multivariate Geostatistics: Beyond Bivariate Moments. Geostatistics Tróia ’92, 133-144. doi:10.1007/978-94-011-1739-5_12Harbaugh , A. W. E. R. Banta M. C. Hill M. G. McDonald 2000 MODFLOW-2000, the U.S. Geological Survey modular ground-water model-User guide to modularization concepts and the ground-water flow process Reston, Va.Hendricks Franssen, H. J., & Kinzelbach, W. (2008). Real-time groundwater flow modeling with the Ensemble Kalman Filter: Joint estimation of states and parameters and the filter inbreeding problem. Water Resources Research, 44(9). doi:10.1029/2007wr006505Franssen, H.-J. H., Gómez-Hernández, J., & Sahuquillo, A. (2003). Coupled inverse modelling of groundwater flow and mass transport and the worth of concentration data. Journal of Hydrology, 281(4), 281-295. doi:10.1016/s0022-1694(03)00191-4Henrion, V., Caumon, G., & Cherpeau, N. (2010). ODSIM: An Object-Distance Simulation Method for Conditioning Complex Natural Structures. Mathematical Geosciences, 42(8), 911-924. doi:10.1007/s11004-010-9299-0Hoeksema, R. J., & Kitanidis, P. K. (1984). An Application of the Geostatistical Approach to the Inverse Problem in Two-Dimensional Groundwater Modeling. Water Resources Research, 20(7), 1003-1020. doi:10.1029/wr020i007p01003Hoeksema, R. J., & Kitanidis, P. K. (1985). Analysis of the Spatial Structure of Properties of Selected Aquifers. Water Resources Research, 21(4), 563-572. doi:10.1029/wr021i004p00563Hu, L. Y. (2000). Mathematical Geology, 32(1), 87-108. doi:10.1023/a:1007506918588Hu, L. Y., & Chugunova, T. (2008). Multiple-point geostatistics for modeling subsurface heterogeneity: A comprehensive review. Water Resources Research, 44(11). doi:10.1029/2008wr006993Huysmans, M., & Dassargues, A. (2009). Application of multiple-point geostatistics on modelling groundwater flow and transport in a cross-bedded aquifer (Belgium). Hydrogeology Journal, 17(8), 1901-1911. doi:10.1007/s10040-009-0495-2Jafarpour, B., & Khodabakhshi, M. (2011). A Probability Conditioning Method (PCM) for Nonlinear Flow Data Integration into Multipoint Statistical Facies Simulation. Mathematical Geosciences, 43(2), 133-164. doi:10.1007/s11004-011-9316-yJournel, A., & Zhang, T. (2006). The Necessity of a Multiple-Point Prior Model. Mathematical Geology, 38(5), 591-610. doi:10.1007/s11004-006-9031-2Kerrou, J., Renard, P., Hendricks Franssen, H.-J., & Lunati, I. (2008). Issues in characterizing heterogeneity and connectivity in non-multiGaussian media. Advances in Water Resources, 31(1), 147-159. doi:10.1016/j.advwatres.2007.07.002Kwa, Franz, M. O., & Scholkopf, B. (2005). Iterative kernel principal component analysis for image modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(9), 1351-1366. doi:10.1109/tpami.2005.181Kitanidis, P. K. (2007). On stochastic inverse modeling. Geophysical Monograph Series, 19-30. doi:10.1029/171gm04Kitanidis, P. K., & Vomvoris, E. G. (1983). A geostatistical approach to the inverse problem in groundwater modeling (steady state) and one-dimensional simulations. Water Resources Research, 19(3), 677-690. doi:10.1029/wr019i003p00677Li, L., Zhou, H., Hendricks Franssen, H. J., & Gómez-Hernández, J. J. (2011). Groundwater flow inverse modeling in non-MultiGaussian media: performance assessment of the normal-score Ensemble Kalman Filter. Hydrology and Earth System Sciences Discussions, 8(4), 6749-6788. doi:10.5194/hessd-8-6749-2011Li, L., Zhou, H., & Gómez-Hernández, J. J. (2011). Transport upscaling using multi-rate mass transfer in three-dimensional highly heterogeneous porous media. Advances in Water Resources, 34(4), 478-489. doi:10.1016/j.advwatres.2011.01.001Li, L., Zhou, H., & Gómez-Hernández, J. J. (2011). A comparative study of three-dimensional hydraulic conductivity upscaling at the macro-dispersion experiment (MADE) site, Columbus Air Force Base, Mississippi (USA). Journal of Hydrology, 404(3-4), 278-293. doi:10.1016/j.jhydrol.2011.05.001Mariethoz, G., Renard, P., & Straubhaar, J. (2010). The Direct Sampling method to perform multiple-point geostatistical simulations. Water Resources Research, 46(11). doi:10.1029/2008wr007621Mariethoz, G., Renard, P., & Caers, J. (2010). Bayesian inverse problem and optimization with iterative spatial resampling. Water Resources Research, 46(11). doi:10.1029/2010wr009274Neuman, S. P. (1973). Calibration of distributed parameter groundwater flow models viewed as a multiple-objective decision process under uncertainty. Water Resources Research, 9(4), 1006-1021. doi:10.1029/wr009i004p01006Oliver, D. S., Cunha, L. B., & Reynolds, A. C. (1997). Markov chain Monte Carlo methods for conditioning a permeability field to pressure data. Mathematical Geology, 29(1), 61-91. doi:10.1007/bf02769620RamaRao, B. S., LaVenue, A. M., De Marsily, G., & Marietta, M. G. (1995). Pilot Point Methodology for Automated Calibration of an Ensemble of conditionally Simulated Transmissivity Fields: 1. Theory and Computational Experiments. Water Resources Research, 31(3), 475-493. doi:10.1029/94wr02258Rubin, Y., Chen, X., Murakami, H., & Hahn, M. (2010). A Bayesian approach for inverse modeling, data assimilation, and conditional simulation of spatial random fields. Water Resources Research, 46(10). doi:10.1029/2009wr008799Strebelle, S. (2002). Mathematical Geology, 34(1), 1-21. doi:10.1023/a:1014009426274Suzuki, S., & Caers, J. (2008). A Distance-based Prior Model Parameterization for Constraining Solutions of Spatial Inverse Problems. Mathematical Geosciences, 40(4), 445-469. doi:10.1007/s11004-008-9154-8Wen, X. H., Capilla, J. E., Deutsch, C. V., Gómez-Hernández, J. J., & Cullick, A. S. (1999). A program to create permeability fields that honor single-phase flow rate and pressure data. Computers & Geosciences, 25(3), 217-230. doi:10.1016/s0098-3004(98)00126-5Zhang, T., Switzer, P., & Journel, A. (2006). Filter-Based Classification of Training Image Patterns for Spatial Simulation. Mathematical Geology, 38(1), 63-80. doi:10.1007/s11004-005-9004-xZhou, H., Gómez-Hernández, J. J., Hendricks Franssen, H.-J., & Li, L. (2011). An approach to handling non-Gaussianity of parameters and state variables in ensemble Kalman filtering. Advances in Water Resources, 34(7), 844-864. doi:10.1016/j.advwatres.2011.04.014Zhou, H., Li, L., Hendricks Franssen, H.-J., & Gómez-Hernández, J. J. (2011). Pattern Recognition in a Bimodal Aquifer Using the Normal-Score Ensemble Kalman Filter. Mathematical Geosciences, 44(2), 169-185. doi:10.1007/s11004-011-9372-

Identifying nurses' rewards: a qualitative categorization study in Belgium

BACKGROUND: Rewards are important in attracting, motivating and retaining the most qualified employees, and nurses are no exception to this rule. This makes the establishment of an efficient reward system for nurses a true challenge for every hospital manager. A reward does not necessarily have a financial connotation: non-financial rewards may matter too, or may even be more important. Therefore, the present study examines nurses' reward perceptions, in order to identify potential reward options. METHODS: To answer the research question "What do nurses consider a reward and how can these rewards be categorized?", 20 in-depth semi-structured interviews with nurses were conducted and analysed using discourse and content analyses. In addition, the respondents received a list of 34 rewards (derived from the literature) and were asked to indicate the extent to which they perceived each of them to be rewarding. RESULTS: Discourse analysis revealed three major reward categories: financial, non-financial and psychological, each containing different subcategories. In general, nurses more often mentioned financial rewards spontaneously in the interview, compared to non-financial and psychological rewards. The questionnaire results did not, however, indicate a significant difference in the rewarding potential of these three categories. Both the qualitative and quantitative data revealed that a number of psychological and non-financial rewards were important for nurses in addition to their monthly pay and other remunerations. In particular, appreciation for their work by others, compliments from others, presents from others and contact with patients were highly valued. Moreover, some demographical variables influenced the reward perceptions. Younger and less experienced nurses considered promotion possibilities as more rewarding than the older and more senior ones. The latter valued job security and working for a hospital with a good reputation higher than their younger and more junior colleagues. CONCLUSION: When trying to establish an efficient reward system for nurses, hospital managers should not concentrate on the financial reward possibilities alone. They also ought to consider non-financial and psychological rewards (in combination with financial rewards), since nurses value these as well and they may lead to a more personalized reward system

Geostatistical analysis of an experimental stratigraphy

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/95276/1/wrcr10212.pd