Dynamically optimal treatment allocation using Reinforcement Learning

Devising guidance on how to assign individuals to treatment is an important goal in empirical research. In practice, individuals often arrive sequentially, and the planner faces various constraints such as limited budget/capacity, or borrowing constraints, or the need to place people in a queue. For instance, a governmental body may receive a budget outlay at the beginning of a year, and it may need to decide how best to allocate resources within the year to individuals who arrive sequentially. In this and other examples involving inter-temporal trade-offs, previous work on devising optimal policy rules in a static context is either not applicable, or sub-optimal. Here we show how one can use offline observational data to estimate an optimal policy rule that maximizes expected welfare in this dynamic context. We allow the class of policy rules to be restricted for legal, ethical or incentive compatibility reasons. The problem is equivalent to one of optimal control under a constrained policy class, and we exploit recent developments in Reinforcement Learning (RL) to propose an algorithm to solve this. The algorithm is easily implementable with speedups achieved through multiple RL agents learning in parallel processes. We also characterize the statistical regret from using our estimated policy rule by casting the evolution of the value function under each policy in a Partial Differential Equation (PDE) form and using the theory of viscosity solutions to PDEs. We find that the policy regret decays at a $n^{-1/2}$ rate in most examples; this is the same rate as in the static case.Comment: 67 page

Synthesis and DNA-binding of Metallocyclic Architectures

A new family of cationic N-heterocyclic ligand derivatives was prepared and characterised. Among these compounds are halide salts of the dications [Y(spacer)Y]2+, each of which comprise two N heterocyclic donor groups (Y = 4,4′-bipy, pyz, apyz, apym) linked by a conformationally flexible spacer such as (CH2)n, α,α′-xylylene, 2,6-lutidylene or thiabicyclo[3.3.1]nonane-2,6 diyl. The diquaternary halide salts were converted to NO3- and PF6- salts, and interaction of these bridging ligands with labile palladium(II) and platinum(II) precursors afforded several multinuclear complexes. Bis(4,4′-bipyridinium) dications were incorporated into the dinuclear macrocycles [M2(2,2′ bipy)2{4,4′ bipy(CH2)n4,4′-bipy}2]8+ (M = Pd, Pt; n = 4, 6), cis [Pd2Cl4{4,4′ bipy(CH2)34,4′-bipy}2]4+, [Pt2(dppp)2{4,4′-bipy(1,2-xylylene)4,4′-bipy}2]8+ and cis-[Pt2Cl4{4,4′-bipy(1,2-xylylene)4,4′-bipy}2]4+. While bis(pyrazinium) analogues were unreactive towards the palladium(II) and platinum(II) precursors, the doubly deprotonated bis(3 aminopyrazinium) and bis(2 aminopyrimidinium) derivatives served as charge-neutral quadruply-bridging ligands in the complexes [Pt4(2,2′ bipy)4{apyz(CH2)6apyz–2H}2]8+ and [Pt4(2,2′ bipy)4{apym(CH2)5apym–2H}2]8+, both of which feature Pt(II). Pt(II) interactions. Larger species formed when the diamine O,O′-bis(2-aminoethyl)octadeca(ethylene glycol) (PEGda) was treated with cis dinitratopalladium(II) and platinum(II) precursors. The resulting complexes [M(N,N)(PEGda)]2+ (M = Pd, Pt; N,N = 2,2′-bipy, en, tmeda) possessed great size (62 membered chelate rings) and aqueous solubility. DNA-binding studies were conducted with selected complexes in order to investigate the types of interactions these species might participate in. Equimolar mixtures containing either the 16mer duplex DNA D2 or the single strand D2a and palladium(II)/platinum(II) complexes were prepared and analysed by negative-ion ESI MS. Studies of D2/Pd(II) mixtures suggested extensive fragmentation was occuring, and the use of [Pd(tmeda)(PEGda)]2+ and [Pd2(2,2′-bipy)2{4,4′-bipy(CH2)44,4′-bipy}2]8+ resulted in D2 adducts of [Pd(tmeda)]2+ and [4,4′-bipy(CH2)44,4′-bipy]2+, respectively. Decomposition also occurred when D2a was used, although 1 : 1 adducts were observed with [Pd(tmeda)(PEGda)]2+, [Pd(2,2′ bipy)(PEGda)]2+ and [Pd2(2,2′-bipy)2{4,4′-bipy(CH2)44,4′-bipy}2]8+. The low intensities of these adducts indicated that they are unstable towards ESI MS. Analogous ESI-MS experiments using platinum(II) derivatives were performed and, in contrast to those with palladium(II), indicated that the complexes remained largely intact. ESI-MS analysis of D2/Pt(II) mixtures allowed for the detection of 1 : 1 D2 adducts of [Pt(en)(PEGda)]2+, [Pt(tmeda)(PEGda)]2+ and [Pt2(2,2′-bipy)2{4,4′-bipy(CH2)44,4′-bipy}2]8+. Intensities of the adduct ions suggested the greater charge and aryl surface area allow the dinuclear species to bind D2 most strongly. Both [Pt(2,2′-bipy)(Mebipy)2]4+ and [Pt(2,2′ bipy)(NH3)2]2+ gave rise to 1 : 2 adducts of D2, although the latter was found to be a weaker binder, perhaps owing to its lower charge. Data obtained using 1 : 5 (D2 : complex) mixtures were consistent with the results above and suggested that D2 can bind more molecules of daunomycin than any of the platinum(II) species. Analyses of D2a/Pt(II) mixtures gave results similar to those obtained with D2, although fragmentation was more pronounced, indicating that the nucleobases in D2a play more significant roles in mediating decomposition than those in D2, in which they are paired in a complementary manner. Investigations into the effects of selected platinum(II) complexes on the thermal denaturation of calf-thymus DNA (CT-DNA) in solution were conducted. Both [Pt2(2,2′ bipy)2{4,4′-bipy(CH2)64,4′-bipy}2]8+ and [Pt(2,2′-bipy)(Mebipy)2]4+ greatly stabilised CT-DNA, most likely by intercalation. In contrast, [Pt(tmeda)(PEGda)]2+ and [Pt(en)(PEGda)]2+ (as well as PEGda) caused negligible changes in melting temperature (∆Tm), suggesting that these interact weakly with CT-DNA. Data for [Pt(2,2′ bipy)(PEGda)]2+ and [Pt(2,2′-bipy)(NH3)2]2+ indicated that these species perhaps intercalate CT-DNA, with similar ∆Tm values for both complexes implying that PEGda does not play a major role in binding. While findings from ESI-MS experiments were similar to those from the thermal denaturation experiments, discrepancies between results from the two methods could be found. In particular, fragmentation of cyclic species during ESI-MS caused the binding strength of the species to be underestimated when this method was employed

Identifying Key Factors Affecting Translational Landslides in Part of the Yakima Fold and Thrust Belt, Washington State

Washington has one of the fastest growth rates in the nation, and unfortunately also is among the most landslide-prone states. With increased population density and urban sprawl, the need for landslide hazard assessment grows. On the Columbia Plateau in central Washington, the smooth rigid, inclined surfaces of the Columbia River Basalts (CRBs) with loose sediment layers between them induces landslides of large blocks of bedrock. These hazards remain poorly understood, but their significance was heightened by the 2017 Rattlesnake Hills Landslide that currently threatens a community and an interstate highway south of Yakima, Washington. I propose that the strongest influences on translational landslides within the folded CRBs and sediment layers are dip slope angle, sediment composition and hillslope base removal. Using ArcGIS 10.5 mapping software, my research combines previous mapping efforts, the Washington State Landslide Inventory, and the Washington DNR Lidar Portal to evaluate landslide factors. Having conducted field work and sampled material from two sedimentary members in the area with the highest frequency of translational landslides in my study area, I compare grain size, hydraulic conductivity, dip slope angle, and the apparent dissection and debuttressing of overlying basalt members to identify relative significance of these factors. I aim to better understand where and why translational landslides are likely to develop to asses associated risk and avoid hazardous conditions of area prone to translational landslides

Wettbewerb und Kooperation in der Nutzung von Absolventenstudien in der Schweiz

Im Beitrag wird die Nutzung resp. Nicht-Nutzung von Absolventenstudien durch Hochschulen in der Schweiz betrachtet. Das Schweizer Bundesamt für Statistik führt zweijährlich eine nationale Absolventenbefragung durch, zu der alle Absolventinnen und Absolventen aller Schweizer Hochschulen eingeladen werden. Hochschulen können der Umfrage eigene Fragen hinzufügen und den gesamten Datensatz analysieren. Inwiefern bietet diese Ausgangslage Gelegenheiten für Zusammenarbeit unter, aber auch Wettbewerb zwischen den Hochschulen? Dieser Frage geht der Beitrag nach. Die Ergebnisse basieren auf einer qualitativen Befragung der Hochschulen und auf unseren eigenen institutionellen Erfahrungen. Sie zeigen den Wunsch nach Zusammenarbeit auf der methodischen Ebene, während in der Nutzung der Daten Wettbewerbsdenken sichtbar wir

Wissenschaft in der Vertrauenskrise? : Plädoyer für eine gemeinwohlorientierte Wissenschaft

Fake news, Faktenignoranz und politische Unterdrückung der Zirkulation von Wissen – die Wissenschaft muss angesichts dieser Herausforderungen ihre Rolle neu definieren. Soll sie sich in die Politik einmischen und mehr Verantwortung für die Lösung gesellschaftlicher Probleme übernehmen? Wird sie glaubwürdiger, indem sie ihre Werte transparent macht? Oder sind solche Massnahmen sogar kontraproduktiv, um einem Vertrauensverlust in die Wissenschaft entgegenzuwirken? In jedem Fall sollte Wissenschaft sich dem Gemeinwohl verpflichten, öffentliche Diskurse anstoßen und zu gesellschaftlicher Orientierung beitragen

A new early occurrence of Cervidae in North America from the Miocene-Pliocene Ellensburg Formation in Washington, USA

A new fossil cervid from the Craig’s Hill locality of the Miocene-Pliocene Ellensburg Formation in the State of Washington, USA, may be one of the oldest fossil deer yet found in North America, underlying a date of 4.9 Ma ± 0.1 Ma. This mandible fragment with m2, m3, and associated p2 has a size that does not distinguish it from Bretzia pseudalces, Odocoileus hemionus, or Capreolus constantini, and distinguishes it from Eocoileus gentryorum and Odocoileus lucasi only in having a thinner p2. A strong paraconid on the p2, and ectostylids and cingulids on the m2 and m3 link it most strongly with Bretzia pseudalces, but these are also intermittently present in Odocoileus hemionus. Because of the close geographic and temporal proximity of Bretzia pseudalces in the Ringold Formation, and its shared morphology with Bretzia, we are tentatively assigning this specimen to cf. Bretzia

REndo: Internal Instrumental Variables to Address Endogeneity

Endogeneity is a common problem in any causal analysis. It arises when the independence assumption between an explanatory variable and the error in a statistical model is violated. The causes of endogeneity are manifold and include response bias in surveys, omission of important explanatory variables, or simultaneity between explanatory and response variables. Instrumental variable estimation provides a possible solution. However, valid and strong external instruments are difficult to find. Consequently, internal instrumental variable approaches have been proposed to correct for endogeneity without relying on external instruments. The R package REndo implements various internal instrumental variable approaches, i.e., latent instrumental variables estimation (Ebbes, Wedel, Boeckenholt, and Steerneman 2005), higher moments estimation (Lewbel 1997), heteroscedastic error estimation (Lewbel 2012), joint estimation using copula (Park and Gupta 2012) and multilevel generalized method of moments estimation (Kim and Frees 2007). Package usage is illustrated on simulated and real-world data

REndo: Internal Instrumental Variables to Address Endogeneity

Endogeneity is a common problem in any causal analysis. It arises when the independence assumption between an explanatory variable and the error in a statistical model is violated. The causes of endogeneity are manifold and include response bias in surveys, omission of important explanatory variables, or simultaneity between explanatory and response variables. Instrumental variable estimation provides a possible solution. However, valid and strong external instruments are difficult to find. Consequently, internal instrumental variable approaches have been proposed to correct for endogeneity without relying on external instruments. The R package REndo implements various internal instrumental variable approaches, i.e., latent instrumental variables estimation (Ebbes, Wedel, Boeckenholt, and Steerneman 2005), higher moments estimation (Lewbel 1997), heteroscedastic error estimation (Lewbel 2012), joint estimation using copula (Park and Gupta 2012) and multilevel generalized method of moments estimation (Kim and Frees 2007). Package usage is illustrated on simulated and real-world data

Essays in applied microeconomics and microeconometrics

First, I investigate the change in hate crime targeting race or religion after the Brexit vote. My results reveal a substantial and transitory increase in such hate crime following the vote. The focus of my analysis is the considerable spatial heterogeneity of this increase. Areas with a greater increase in hate crime are characterized by both a greater immigrant share and higher income proxies. Issues of multiple hypothesis testing and model selection limit the use of classic methods; therefore, I apply and adapt recent machine learning methods to uncover patterns in the spatial heterogeneity. I then focus on the question how to utilize data from randomized control trials to obtain an optimal dynamic treatment rule. Consider a situation wherein individuals arrive sequentially - for example when becoming unemployed - to a social planner. Once each individual arrives, the planner decides instantaneously on a treatment assignment - for example job training - while taking into account the characteristics of the individual and the remaining capacity to offer training. In order to decide optimally, expectations over the dynamic process of unemployment patterns are required. Reinforcement learning methods can be used to solve this dynamic optimization problem and the resulting algorithm has a number of desirable properties. Finally, I study the creation of not-for-profit firms. Reputation is key for high-quality producers when quality is only observed after the time of purchase. For companies that potentially enter several markets, I show that the concern for reputation affects both the optimal organizational form and the decision which markets to enter. Specifically, a market with poor customers that would be ignored in isolation can be served for signaling purposes. The optimal organizational forms in that case are a not-for-profit firm used for signaling in the “market for the poor” and an associated for-profit firm in the “market for the rich”