Extreme State Aggregation Beyond MDPs

We consider a Reinforcement Learning setup where an agent interacts with an environment in observation-reward-action cycles without any (esp.\ MDP) assumptions on the environment. State aggregation and more generally feature reinforcement learning is concerned with mapping histories/raw-states to reduced/aggregated states. The idea behind both is that the resulting reduced process (approximately) forms a small stationary finite-state MDP, which can then be efficiently solved or learnt. We considerably generalize existing aggregation results by showing that even if the reduced process is not an MDP, the (q-)value functions and (optimal) policies of an associated MDP with same state-space size solve the original problem, as long as the solution can approximately be represented as a function of the reduced states. This implies an upper bound on the required state space size that holds uniformly for all RL problems. It may also explain why RL algorithms designed for MDPs sometimes perform well beyond MDPs.Comment: 28 LaTeX pages. 8 Theorem

Not All Children with Cystic Fibrosis Have Abnormal Esophageal Neutralization during Chemical Clearance of Acid Reflux.

PurposeAcid neutralization during chemical clearance is significantly prolonged in children with cystic fibrosis, compared to symptomatic children without cystic fibrosis. The absence of available reference values impeded identification of abnormal findings within individual patients with and without cystic fibrosis. The present study aimed to test the hypothesis that significantly more children with cystic fibrosis have acid neutralization durations during chemical clearance that fall outside the physiological range.MethodsPublished reference value for acid neutralization duration during chemical clearance (determined using combined impedance/pH monitoring) was used to assess esophageal acid neutralization efficiency during chemical clearance in 16 children with cystic fibrosis (3 to <18 years) and 16 age-matched children without cystic fibrosis.ResultsDuration of acid neutralization during chemical clearance exceeded the upper end of the physiological range in 9 of 16 (56.3%) children with and in 3 of 16 (18.8%) children without cystic fibrosis (p=0.0412). The likelihood ratio for duration indicated that children with cystic fibrosis are 2.1-times more likely to have abnormal acid neutralization during chemical clearance, and children with abnormal acid neutralization during chemical clearance are 1.5-times more likely to have cystic fibrosis.ConclusionSignificantly more (but not all) children with cystic fibrosis have abnormally prolonged esophageal clearance of acid. Children with cystic fibrosis are more likely to have abnormal acid neutralization during chemical clearance. Additional studies involving larger sample sizes are needed to address the importance of genotype, esophageal motility, composition and volume of saliva, and gastric acidity on acid neutralization efficiency in cystic fibrosis children

The Complexity of Graph-Based Reductions for Reachability in Markov Decision Processes

We study the never-worse relation (NWR) for Markov decision processes with an infinite-horizon reachability objective. A state q is never worse than a state p if the maximal probability of reaching the target set of states from p is at most the same value from q, regard- less of the probabilities labelling the transitions. Extremal-probability states, end components, and essential states are all special cases of the equivalence relation induced by the NWR. Using the NWR, states in the same equivalence class can be collapsed. Then, actions leading to sub- optimal states can be removed. We show the natural decision problem associated to computing the NWR is coNP-complete. Finally, we ex- tend a previously known incomplete polynomial-time iterative algorithm to under-approximate the NWR

The anti-glucocorticoid receptor antibody clone 5E4: raising awareness of unspecific antibody binding

Unspecific antibody binding takes a significant toll on researchers in the form of both the economic burden and the disappointed hopes of promising new therapeutic targets. Despite recent initiatives promoting antibody validation, a uniform approach addressing this issue has not yet been developed. Here, we demonstrate that the anti-glucocorticoid receptor (GR) antibody clone 5E4 predominantly targets two different proteins of approximately the same size, namely AMP deaminase 2 (AMPD2) and transcription intermediary factor 1-beta (TRIM28). This paper is intended to generate awareness of unspecific binding of well-established reagents and advocate the use of more rigorous verification methods to improve antibody quality in the future

Hindrance of the excitation of the Hoyle state and the ghost of the 22+2^+_2 state in 12^{12}C

While the Hoyle state (the isoscalar $0^+_2$ excitation at 7.65 MeV in $^{12}$C) has been observed in almost all the electron and $\alpha$ inelastic scattering experiments, the second $2^+$ excited state of $^{12}$C at $E_{\rm x}\approx 10$ MeV, believed to be an excitation of the Hoyle state, has not been clearly observed in these measurements excepting the high-precision \aap experiments at $E_\alpha=240$ and 386 MeV. Given the (spin and isospin zero) $\alpha$-particle as a good probe for the nuclear isoscalar excitations, it remains a puzzle why the peak of the $2^+_2$ state could not be clearly identified in the measured \aap spectra. To investigate this effect, we have performed a microscopic folding model analysis of the \ac scattering data at 240 and 386 MeV in both the Distorted Wave Born Approximation (DWBA) and coupled-channel (CC) formalism, using the nuclear transition densities given by the antisymmetrized molecular dynamics (AMD) approach and a complex CDM3Y6 density dependent interaction. Although AMD predicts a very weak transition strength for the direct $(0^+_1\to 2^+_2)$ excitation, our detailed analysis has shown evidence that a weak \emph{ghost} of the $2^+_2$ state could be identified in the 240 MeV \aap data for the $0^+_3$ state at 10.3 MeV, when the CC effects by the indirect excitation of the $2^+_2$ state are taken into account. Based on the same AMD structure input and preliminary \aap data at 386 MeV, we have estimated relative contributions from the $2^+_2$ and $0^+_3$ states to the excitation of $^{12}$C at $E_{\rm x}\approx 10$ MeV as well as possible contamination by $3^-_1$ state.Comment: Accepted for publication in Phys. Lett.

Gastroesophageal reflux in cystic fibrosis across the age spectrum.

Background:Scientific advances have improved longevity in cystic fibrosis (CF) patients and many of these patients can expect to experience age-related gastrointestinal co-morbidities. We aimed to assess the extent to which age might impact gastroesophageal reflux (GER) in patients with CF. Methods:Our esophageal pH-multichannel intraluminal impedance monitoring database was searched for tracings belonging to CF patients ≥2 years old without prior fundoplication and not taking anti-reflux medications immediately prior (within 7 days) and during the study. Tracings were retrospectively analyzed; Impedance and pH variables were evaluated with respect to age and pulmonary function. Results:Twenty-eight patients were enrolled; 16 children (3.1-17.7 years) and 12 adults (18.2-48.9 years). Among pH probe parameters, correlation analysis showed DeMeester score (P=0.011) and number of acid reflux events lasting >5 minutes (P=0.047) to be significantly correlated with age. Age was not significantly correlated with any of the impedance parameters. Age was negatively correlated with baseline impedance (BI) in the distal esophagus (r=-0.424, P=0.023) and BI was negatively correlated with several pH parameters, including reflux index (r=-0.553, P=0.002), number of total acid reflux events (r=-0.576, P=0.001), number of acid reflux events lasting >5 minutes (r=-0.534, P=0.003), and DeMeester score (r=-0.510, P=0.006). Pulmonary function (percent predicted forced expiratory volume in one minute; ppFEV1) was negatively correlated with age (r=-0.494, P=0.0007). The interaction of age and ppFEV1 and any of the reflux parameters, however, was not significant (P>0.05); the strongest evidence for an interaction was found for the number of acid reflux events reaching the proximal esophagus, but this interaction still did not reach statistical significance (P=0.070). Conclusions:In a small cohort, we found evidence that age may be associated with increased acid exposure and that both age and increased acid exposure are associated with reduced BI in the distal esophagus. The negative relationship between pulmonary function and age in our cohort is not related to GER. This pilot study supports the need for esophageal assessment and treatment of GER as standard components of clinical care for an aging CF population

Teratoma formation of human embryonic stem cells in three-dimensional perfusion culture bioreactors

Teratoma formation in mice is today the most stringent test for pluripotency that is available for human pluripotent cells, as chimera formation and tetraploid complementation cannot be performed with human cells. The teratoma assay could also be applied for assessing the safety of human pluripotent cell-derived cell populations intended for therapeutic applications. In our study we examined the spontaneous differentiation behaviour of human embryonic stem cells (hESCs) in a perfused 3D multi-compartment bioreactor system and compared it with differentiation of hESCs and human induced pluripotent cells (hiPSCs) cultured in vitro as embryoid bodies and in vivo in an experimental mouse model of teratoma formation. Results from biochemical, histological/immunohistological and ultrastuctural analyses revealed that hESCs cultured in bioreactors formed tissue-like structures containing derivatives of all three germ layers. Comparison with embryoid bodies and the teratomas revealed a high degree of similarity of the tissues formed in the bioreactor to these in the teratomas at the histological as well as transcriptional level, as detected by comparative whole-genome RNA expression profiling. The 3D culture system represents a novel in vitro model that permits stable long-term cultivation, spontaneous multi-lineage differentiation and tissue formation of pluripotent cells that is comparable to in vivo differentiation. Such a model is of interest, e.g. for the development of novel cell differentiation strategies. In addition, the 3D in vitro model could be used for teratoma studies and pluripotency assays in a fully defined, controlled environment, alternatively to in vivo mouse models. Copyright (c) 2012 John Wiley & Sons, Ltd