Model-free preference-based reinforcement learning

Specifying a numeric reward function for reinforcement learning typically requires a lot of hand-tuning from a human expert. In contrast, preference-based reinforcement learning (PBRL) utilizes only pairwise comparisons between trajectories as a feedback signal, which are often more intuitive to specify. Currently available approaches to PBRL for control problems with continuous state/action spaces require a known or estimated model, which is often not available and hard to learn. In this paper, we integrate preference-based estimation of the reward function into a model-free reinforcement learning (RL) algorithm, resulting in a model-free PBRL algorithm. Our new algorithm is based on Relative Entropy Policy Search (REPS), enabling us to utilize stochastic policies and to directly control the greediness of the policy update. REPS decreases exploration of the policy slowly by limiting the relative entropy of the policy update, which ensures that the algorithm is provided with a versatile set of trajectories, and consequently with informative preferences. The preference-based estimation is computed using a sample-based Bayesian method, which can also estimate the uncertainty of the utility. Additionally, we also compare to a linear solvable approximation, based on inverse RL. We show that both approaches perform favourably to the current state-of-the-art. The overall result is an algorithm that can learn non-parametric continuous action policies from a small number of preferences

"You have reached your destination" : a single trial EEG classification study

Studies have established that it is possible to differentiate between the brain's responses to observing correct and incorrect movements in navigation tasks. Furthermore, these classifications can be used as feedback for a learning-based BCI, to allow real or virtual robots to find quasi-optimal routes to a target. However, when navigating it is important not only to know we are moving in the right direction toward a target, but also to know when we have reached it. We asked participants to observe a virtual robot performing a 1-dimensional navigation task. We recorded EEG and then performed neurophysiological analysis on the responses to two classes of correct movements: those that moved closer to the target but did not reach it, and those that did reach the target. Further, we used a stepwise linear classifier on time-domain features to differentiate the classes on a single-trial basis. A second data set was also used to further test this single-trial classification. We found that the amplitude of the P300 was significantly greater in cases where the movement reached the target. Interestingly, we were able to classify the EEG signals evoked when observing the two classes of correct movements against each other with mean overall accuracy of 66.5 and 68.0% for the two data sets, with greater than chance levels of accuracy achieved for all participants. As a proof of concept, we have shown that it is possible to classify the EEG responses in observing these different correct movements against each other using single-trial EEG. This could be used as part of a learning-based BCI and opens a new door toward a more autonomous BCI navigation system

Efavirenz-induced urolithiasis

We describe the first case of efavirenz-induced urolithiasis in a 47-year-old HIV-positive patient. Urinary obstruction led to pyelonephritis and septic shock, requiring emergency ureteral catheterisation. The subsequent clinical course was favourable, allowing the patient's discharge on day5. A 7mm, radio-translucent, non-crystalline, beige stone was extracted during catheterisation. Stone analysis by Fourier transform infrared spectrometry, liquid chromatography and mass spectrometry revealed a stone composed of efavirenz (EFV) metabolites M4, M5, M8 (as described by Mutlib et al. in 1999) and approximately 50% of unspecified proteins. EFV is a non-nucleoside reverse transcriptase inhibitor introduced to European markets in 1999. It is principally metabolised by cytochrome P450 3A4 and 2B6. Of the dose, 14-34% is excreted in the urine, 1% as unchanged drug. The patient had been taking 600mg EFV per day for 3years. As EFV-induced urolithiasis has not been reported so far, we would like to draw the attention of the medical community to this potentially severe complicatio

Characterization of rectangular copper wire forming properties and derivation of control concepts for the kinematic bending of hairpin coils

As a result of the continuously growing demand for electric vehicles, innovative production technologies must be developed to fulfill the high automotive requirements for productivity and quality in the manufacturing of electric drives. By providing advantages regarding the degree of automation, the productivity as well as the attainable filling factors in comparison to established round wire winding technologies, the hairpin technology shows a high potential for meeting the requested specifications but also technological weaknesses, especially concerning the process reliability. The referring production process of stators is normally based on the spatial forming of open, hairpin-shaped coils of enameled flat copper wire as well as subsequent joining and contacting processes. Consequently, the hairpin coils represent the elementary components of the process chain and can be either shaped by robust tool-bound or flexible kinematic bending processes that enable the shaping of different contours at moderate tool costs. In this paper, the essential mechanical forming and product properties of flat copper wires with different dimensions and insulation coatings are characterized by means of uniaxial tensile tests as well as metallographic analyses of the material structure, at first. Subsequently, the identified forming properties are correlated to the applied manufacturing processes drawing, rolling as well as continuous extruding and considered as limits of possible material variations. To evaluate the effect of fluctuating wire qualities on the robustness of kinematic hairpin bending processes, the fabrication tolerances are analyzed by finite element simulations, using the example of elementary kinematic bending operations and modeled changes of the material properties. Based on the knowledge of material-based process tolerances, different control concepts for the kinematic bending of hairpin coils are derived and compared based on technical as well as economic aspects

A precursor state to unconventional superconductivity in CeIrIn5{_5}

We present sensitive measurements of the Hall effect and magnetoresistance in CeIrIn${_5}$ down to temperatures of 50 mK and magnetic fields up to 15 T. The presence of a low temperature coherent Kondo state is established. Deviations from Kohler's rule and a quadratic temperature dependence of the cotangent of the Hall angle are reminiscent of properties observed in the high temperature superconducting cuprates. The most striking observation pertains to the presence of a \textit{precursor} state--characterized by a change in the Hall mobility--that appears to precede the superconductivity in this material, in similarity to the pseudogap in the cuprate high $T_c$ superconductors.Comment: 4 figure

Predictors of Social Physique Anxiety in Elite Female Youth Athletes

The purpose of this study was to examine predictors of social physique anxiety (SPA). SPA, self-esteem, body-esteem, public body consciousness (PBC) and percent body fat (%BF) were assessed with elite female youth athletes (N = 68) competing in either figure skating, soccer or gymnastics. Stepwise multiple regression analyses, controlling for BF%, accounted for 59% of the variance in SPA. Self-esteem entered first, and BF%, followed by body-esteem and PBC. The psychological variables accounted for 57% of the variance with self-esteem contributing the most (R square change = 45%). Contrary to previous research, BF% did not significantly contribute to SPA. Additionally, a MANOVA and follow- up ANOVA and Scheffe\u27s tests revealed significant sport differences among SPA, self-esteem, and body-esteem

High spin polarization in the ferromagnetic filled skutterudites KFe4Sb12 and NaFe4Sb12

The spin polarization of ferromagnetic alkali-metal iron antimonides KFe4Sb12 and NaFe4Sb12 is studied by point-contact Andreev reflection using superconducting Nb and Pb tips. From these measurements an intrinsic transport spin polarization Pt of 67% and 60% for the K and Na compound, respectively, is inferred which establishes these materials as a new class of highly spin polarized ferromagnets. The results are in accord with band structure calculations within the local spin density approximation (LSDA) that predict nearly 100% spin polarization in the density of states. We discuss the impact of calculated Fermi velocities and spin fluctuations on Pt.Comment: Pdf file with fi

First-order structural transition in the magnetically ordered phase of Fe1.13Te

Specific heat, resistivity, magnetic susceptibility, linear thermal expansion (LTE), and high-resolution synchrotron X-ray powder diffraction investigations of single crystals Fe1+yTe (0.06 < y < 0.15) reveal a splitting of a single, first-order transition for y 0.12. Most strikingly, all measurements on identical samples Fe1.13Te consistently indicate that, upon cooling, the magnetic transition at T_N precedes the first-order structural transition at a lower temperature T_s. The structural transition in turn coincides with a change in the character of the magnetic structure. The LTE measurements along the crystallographic c-axis displays a small distortion close to T_N due to a lattice striction as a consequence of magnetic ordering, and a much larger change at T_s. The lattice symmetry changes, however, only below T_s as indicated by powder X-ray diffraction. This behavior is in stark contrast to the sequence in which the phase transitions occur in Fe pnictides.Comment: 6 page