Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies

Reinforcement learning (RL) has shown great promise with algorithms learning in environments with large state and action spaces purely from scalar reward signals. A crucial challenge for current deep RL algorithms is that they require a tremendous amount of environment interactions for learning. This can be infeasible in situations where such interactions are expensive; such as in robotics. Offline RL algorithms try to address this issue by bootstrapping the learning process from existing logged data without needing to interact with the environment from the very beginning. While online RL algorithms are typically evaluated as a function of the number of environment interactions, there exists no single established protocol for evaluating offline RL methods.In this paper, we propose a sequential approach to evaluate offline RL algorithms as a function of the training set size and thus by their data efficiency. Sequential evaluation provides valuable insights into the data efficiency of the learning process and the robustness of algorithms to distribution changes in the dataset while also harmonizing the visualization of the offline and online learning phases. Our approach is generally applicable and easy to implement. We compare several existing offline RL algorithms using this approach and present insights from a variety of tasks and offline datasets.Comment: TMLR 202

Versatile Confocal Raman Imaging Microscope Built from Off-the-Shelf Opto-Mechanical Components

Confocal Raman microscopic (CRM) imaging has evolved to become a key tool for spatially resolved, compositional analysis and imaging, down to the μm-scale, and nowadays one may choose between numerous commercial instruments. That notwithstanding, situations may arise which exclude the use of a commercial instrument, e.g., if the analysis involves toxic or radioactive samples/environments; one may not wish to render an expensive instrument unusable for other uses, due to contamination. Therefore, custom-designed CRM instrumentation—being adaptable to hazardous conditions and providing operational flexibility—may be beneficial. Here, we describe a CRM setup, which is constructed nearly in its entirety from off-the-shelf optomechanical and optical components. The original aim was to develop a CRM suitable for the investigation of samples exposed to tritium. For increased flexibility, the CRM system incorporates optical fiber coupling to both the Raman excitation laser and the spectrometer. Lateral raster scans and axial profiling of samples are facilitated by the use of a motorized xyz-translation assembly. Besides the description of the construction and alignment of the CRM system, we also provide (i) the experimental evaluation of system performance (such as, e.g., spatial resolution) and (ii) examples of Raman raster maps and axial profiles of selected thin-film samples (such as, e.g., graphene sheets)

Accurate reference gas mixtures containing tritiated molecules: Their production and raman‐based analysis

Highly accurate, quantitative analyses of mixtures of hydrogen isotopologues—both the stable species, H$_{2}$, D$_{2}$2, and HD, and the radioactive species, T$_{2}$, HT, and DT—are of great importance in fields as diverse as deuterium–tritium fusion, neutrino mass measurements using tritium β-decay, or for photonuclear experiments in which hydrogen–deuterium targets are used. In this publication we describe a production, handling, and analysis facility capable of fabricating well-defined gas samples, which may contain any of the stable and radioactive hydrogen isotopologues, with sub-percent accuracy for the relative species concentrations. The production is based on precise manometric gas mixing of H$_{2}$, D$_{2}$, and T$_{2}$. The heteronuclear isotopologues HD, HT, and DT are generated via controlled, in-line catalytic reaction or by β-induced self-equilibration, respectively. The analysis was carried out using an in-line intensity- and wavelength-calibrated Raman spectroscopy system. This allows for continuous monitoring of the composition of the circulating gas during the self-equilibration or catalytic evolution phases. During all procedures, effects, such as exchange reactions with wall materials, were considered with care. Together with measurement statistics, these and other systematic effects were included in the determination of composition uncertainties of the generated reference gas samples. Measurement and calibration accuracy at the level of 1% was achieved

The Generation and Analysis of Tritium-substituted Methane

An unavoidable category of molecular species in large-scale tritium applications, such as nuclear fusion, are tritium-substituted hydrocarbons; these form by radiochemical reactions in the presence of (circulating) tritium and carbon (mainly from the steel of vessels and tubing). Tritiumsubstituted methane species, CQ$_4$ (with Q = H , D , T), are often the precursor for higher-order reaction chains, and thus are of particular interest. Here we describe the controlled production of CQ$_4$ carried out in the CAPER facility of the Tritium Laboratory Karlsruhe (TLK), exploiting catalytic reactions and species-enrichment via the CAPER-integral permeator. CQ4 was generated in substantial quantity (>1000 cm$^3$ at ~850 mbar, with CQ$_4$ - content of up to ~20 %). These samples were analyzed using laser Raman and mass spectrometry, to determine the relative isotopologue composition and to trace the generation of tritiated chain-hydrocarbons.Comment: 10 pages, 4 figures. This article has been accepted for publication in Fusion Science and Technology, published by Taylor & Franci

A combined analysis of short-baseline neutrino experiments in the (3+1) and (3+2) sterile neutrino oscillation hypotheses

We investigate adding two sterile neutrinos to resolve the apparent tension existing between short-baseline neutrino oscillation results and CPT-conserving, four-neutrino oscillation models. For both (3+1) and (3+2) models, the level of statistical compatibility between the combined dataset from the null short-baseline experiments Bugey, CHOOZ, CCFR84, CDHS, KARMEN, and NOMAD, on the one hand; and the LSND dataset, on the other, is computed. A combined analysis of all seven short-baseline experiments, including LSND, is also performed, to obtain the favored regions in neutrino mass and mixing parameter space for both models. Finally, four statistical tests to compare the (3+1) and the (3+2) hypotheses are discussed. All tests show that (3+2) models fit the existing short-baseline data significantly better than (3+1) models.Comment: 16 pages, 15 figures. Added NOMAD data to the analysis, one statistical test, and two figures. References and text added. Version submitted to PR

The KATRIN Pre-Spectrometer at reduced Filter Energy

The KArlsruhe TRItium Neutrino experiment, KATRIN, will determine the mass of the electron neutrino with a sensitivity of 0.2 eV (90% C.L.) via a measurement of the beta-spectrum of gaseous tritium near its endpoint of E_0 =18.57 keV. An ultra-low background of about b = 10 mHz is among the requirements to reach this sensitivity. In the KATRIN main beam-line two spectrometers of MAC-E filter type are used in a tandem configuration. This setup, however, produces a Penning trap which could lead to increased background. We have performed test measurements showing that the filter energy of the pre-spectrometer can be reduced by several keV in order to diminish this trap. These measurements were analyzed with the help of a complex computer simulation, modeling multiple electron reflections both from the detector and the photoelectric electron source used in our test setup.Comment: 22 pages, 12 figure

Mapping the EQ-5D index by UPDRS and PDQ-8 in patients with Parkinson’s disease

Background: Clinical studies employ the Unified Parkinson’s Disease Rating Scale (UPDRS) to measure the severity of Parkinson’s disease. Evaluations often fail to consider the health-related quality of life (HrQoL) or apply disease-specific instruments. Health-economic studies normally use estimates of utilities to calculate quality-adjusted life years. We aimed to develop an estimation algorithm for EuroQol- 5 dimensions (EQ-5D)-based utilities from the clinical UPDRS or disease-specific HrQoL data in the absence of original utilities estimates. Methods: Linear and fractional polynomial regression analyses were performed with data from a study of Parkinson’s disease patients (n=138) to predict the EQ-5D index values from UPDRS and Parkinson’s disease questionnaire eight dimensions (PDQ-8) data. German and European weights were used to calculate the EQ-5D index. The models were compared by R2, the root mean square error (RMS), the Bayesian information criterion, and Pregibon’s link test. Three independent data sets validated the models. Results: The regression analyses resulted in a single best prediction model (R2: 0.713 and 0.684, RMS: 0.139 and 13.78 for indices with German and European weights, respectively) consisting of UPDRS subscores II, III, IVa-c as predictors. When the PDQ-8 items were utilised as independent variables, the model resulted in an R2 of 0.60 and 0.67. The independent data confirmed the prediction models. Conclusion: The best results were obtained from a model consisting of UPDRS subscores II, III, IVa-c. Although a good model fit was observed, primary EQ-5D data are always preferable. Further validation of the prediction algorithm within large, independent studies is necessary prior to its generalised use

First observation of tritium adsorption on graphene

In this work, we report on the first-ever studies of graphene exposed to tritium gas in a controlled environment. The single layer graphene on $\textrm{SiO}_2$/Si substrate was exposed to 400 mbar of $\textrm{T}_2$ for a total time of ~55 h. The resistivity of the graphene sample was measured in-situ during tritium exposure using the Van der Pauw method. After the exposure, the samples were scanned with a confocal Raman microscope to study the effect of tritium on the graphene structure as well as the homogeneity of spectral modifications. We found that the sheet resistance increases by three orders of magnitude during the exposure. By Raman microscopy, we demonstrate that the graphene film can withstand the bombardment from the beta-decay of tritium, and primary and secondary ions. Additionally, the Raman spectra after tritium exposure are comparable with previously observed results in hydrogen-loading experiments carried out by other groups. By thermal annealing we could demonstrate, using Raman spectral analysis, that the structural changes were partially reversible. Considering all observations, we conclude that the graphene film was at least partially tritiated during the tritium exposure.Comment: Submitted to Nanoscale Advances (RSC), 14 pages, 4 figure

Improved Upper Limit on the Neutrino Mass from a Direct Kinematic Method by KATRIN.

We report on the neutrino mass measurement result from the first four-week science run of the Karlsruhe Tritium Neutrino experiment KATRIN in spring 2019. Beta-decay electrons from a high-purity gaseous molecular tritium source are energy analyzed by a high-resolution MAC-E filter. A fit of the integrated electron spectrum over a narrow interval around the kinematic end point at 18.57 keV gives an effective neutrino mass square value of (-1.0_{-1.1}^{+0.9})  eV^{2}. From this, we derive an upper limit of 1.1 eV (90% confidence level) on the absolute mass scale of neutrinos. This value coincides with the KATRIN sensitivity. It improves upon previous mass limits from kinematic measurements by almost a factor of 2 and provides model-independent input to cosmological studies of structure formation