Extreme State Aggregation Beyond MDPs

We consider a Reinforcement Learning setup where an agent interacts with an environment in observation-reward-action cycles without any (esp.\ MDP) assumptions on the environment. State aggregation and more generally feature reinforcement learning is concerned with mapping histories/raw-states to reduced/aggregated states. The idea behind both is that the resulting reduced process (approximately) forms a small stationary finite-state MDP, which can then be efficiently solved or learnt. We considerably generalize existing aggregation results by showing that even if the reduced process is not an MDP, the (q-)value functions and (optimal) policies of an associated MDP with same state-space size solve the original problem, as long as the solution can approximately be represented as a function of the reduced states. This implies an upper bound on the required state space size that holds uniformly for all RL problems. It may also explain why RL algorithms designed for MDPs sometimes perform well beyond MDPs.Comment: 28 LaTeX pages. 8 Theorem

Detailed Kinetics of the Direct Allo-Response in Human Liver Transplant Recipients: New Insights from an Optimized Assay

Conventional assays for quantification of allo-reactive T-cell precursor frequencies (PF) are relatively insensitive. We present a robust assay for quantification of PF of T-cells with direct donor-specificity, and establish the kinetics of circulating donor-specific T cells after liver transplantation (LTx). B cells from donor splenocytes were differentiated into professional antigen-presenting cells by CD40-engagement (CD40-B cells). CFSE-labelled PBMC from LTx-recipients obtained before and at several time points after LTx, were stimulated with donor-derived or 3rd party CD40-B cells. PF of donor-specific T cells were calculated from CFSE-dilution patterns, and intracellular IFN-γ was determined after re-stimulation with CD40-B cells. Compared to splenocytes, stimulations with CD40-B cells resulted in 3 to 5-fold higher responding T-cell PF. Memory and naïve T-cell subsets responded equally to allogeneic CD40-B cell stimulation. Donor-specific CD4+ and CD8+ T-cell PF ranged from 0.5 to 19% (median: 5.2%). One week after LTx, PF of circulating donor-specific CD4+ and CD8+ T cells increased significantly, while only a minor increase in numbers of T cells reacting to 3rd party allo-antigens was observed. One year after LTx numbers of CD4+ and CD8+ T cells reacting to donor antigens, as well as those reacting to 3rd party allo-antigens, were slightly lower compared to pre-transplant values. Moreover, CD4+ and CD8+ T cells responding to donor-derived, as well as those reacting to 3rd party CD40-B cells, produced less IFN-γ. In conclusion, our alternative approach enables detection of allo-reactive human T cells at high frequencies, and after application we conclude that donor-specific T-cell PF increase immediately after LTx. However, no evidence for a specific loss of circulating T-cells recognizing donor allo-antigens via the direct pathway up to 1 year after LTx was obtained, underscoring the relative insensitiveness of previous assays

The Ankyrin Repeats and DHHC S-acyl Transferase Domain of AKR1 Act Independently to Regulate Switching from Vegetative to Mating States in Yeast

Signal transduction from G-protein coupled receptors to MAPK cascades through heterotrimeric G-proteins has been described for many eukaryotic systems. One of the best-characterised examples is the yeast pheromone response pathway, which is negatively regulated by AKR1. AKR1-like proteins are present in all eukaryotes and contain a DHHC domain and six ankyrin repeats. Whilst the DHHC domain dependant S-acyl transferase (palmitoyl transferase) function of AKR1 is well documented it is not known whether the ankyrin repeats are also required for this activity. Here we show that the ankyrin repeats of AKR1 are required for full suppression of the yeast pheromone response pathway, by sequestration of the Gβγ dimer, and act independently of AKR1 S-acylation function. Importantly, the functions provided by the AKR1 ankyrin repeats and DHHC domain are not required on the same molecule to fully restore WT phenotypes and function. We also show that AKR1 molecules are S-acylated at locations other than the DHHC cysteine, increasing the abundance of AKR1 in the cell. Our results have important consequences for studies of AKR1 function, including recent attempts to characterise S-acylation enzymology and kinetics. Proteins similar to AKR1 are found in all eukaryotes and our results have broad implications for future work on these proteins and the control of switching between Gβγ regulated pathways

A Model of Ischemia-Induced Neuroblast Activation in the Adult Subventricular Zone

We have developed a rat brain organotypic culture model, in which tissue slices contain cortex-subventricular zone-striatum regions, to model neuroblast activity in response to in vitro ischemia. Neuroblast activation has been described in terms of two main parameters, proliferation and migration from the subventricular zone into the injured cortex. We observed distinct phases of neuroblast activation as is known to occur after in vivo ischemia. Thus, immediately after oxygen/glucose deprivation (6–24 hours), neuroblasts reduce their proliferative and migratory activity, whereas, at longer time points after the insult (2 to 5 days), they start to proliferate and migrate into the damaged cortex. Antagonism of ionotropic receptors for extracellular ATP during and after the insult unmasks an early activation of neuroblasts in the subventricular zone, which responded with a rapid and intense migration of neuroblasts into the damaged cortex (within 24 hours). The process is further enhanced by elevating the production of the chemoattractant SDf-1α and may also be boosted by blocking the activation of microglia. This organotypic model which we have developed is an excellent in vitro system to study neurogenesis after ischemia and other neurodegenerative diseases. Its application has revealed a SOS response to oxygen/glucose deprivation, which is inhibited by unfavorable conditions due to the ischemic environment. Finally, experimental quantifications have allowed us to elaborate a mathematical model to describe neuroblast activation and to develop a computer simulation which should have promising applications for the screening of drug candidates for novel therapies of ischemia-related pathologies

Connecting Network Properties of Rapidly Disseminating Epizoonotics

To effectively control the geographical dissemination of infectious diseases, their properties need to be determined. To test that rapid microbial dispersal requires not only susceptible hosts but also a pre-existing, connecting network, we explored constructs meant to reveal the network properties associated with disease spread, which included the road structure.Using geo-temporal data collected from epizoonotics in which all hosts were susceptible (mammals infected by Foot-and-mouth disease virus, Uruguay, 2001; birds infected by Avian Influenza virus H5N1, Nigeria, 2006), two models were compared: 1) 'connectivity', a model that integrated bio-physical concepts (the agent's transmission cycle, road topology) into indicators designed to measure networks ('nodes' or infected sites with short- and long-range links), and 2) 'contacts', which focused on infected individuals but did not assess connectivity.THE CONNECTIVITY MODEL SHOWED FIVE NETWORK PROPERTIES: 1) spatial aggregation of cases (disease clusters), 2) links among similar 'nodes' (assortativity), 3) simultaneous activation of similar nodes (synchronicity), 4) disease flows moving from highly to poorly connected nodes (directionality), and 5) a few nodes accounting for most cases (a "20:80" pattern). In both epizoonotics, 1) not all primary cases were connected but at least one primary case was connected, 2) highly connected, small areas (nodes) accounted for most cases, 3) several classes of nodes were distinguished, and 4) the contact model, which assumed all primary cases were identical, captured half the number of cases identified by the connectivity model. When assessed together, the synchronicity and directionality properties explained when and where an infectious disease spreads.Geo-temporal constructs of Network Theory's nodes and links were retrospectively validated in rapidly disseminating infectious diseases. They distinguished classes of cases, nodes, and networks, generating information usable to revise theory and optimize control measures. Prospective studies that consider pre-outbreak predictors, such as connecting networks, are recommended

Uterine Epithelial Cell Regulation of DC-SIGN Expression Inhibits Transmitted/Founder HIV-1 Trans Infection by Immature Dendritic Cells

Sexual transmission accounts for the majority of HIV-1 infections. In over 75% of cases, infection is initiated by a single variant (transmitted/founder virus). However, the determinants of virus selection during transmission are unknown. Host cell-cell interactions in the mucosa may be critical in regulating susceptibility to infection. We hypothesized in this study that specific immune modulators secreted by uterine epithelial cells modulate susceptibility of dendritic cells (DC) to infection with HIV-1.Here we report that uterine epithelial cell secretions (i.e. conditioned medium, CM) decreased DC-SIGN expression on immature dendritic cells via a transforming growth factor beta (TGF-β) mechanism. Further, CM inhibited dendritic cell-mediated trans infection of HIV-1 expressing envelope proteins of prototypic reference. Similarly, CM inhibited trans infection of HIV-1 constructs expressing envelopes of transmitted/founder viruses, variants that are selected during sexual transmission. In contrast, whereas recombinant TGF- β1 inhibited trans infection of prototypic reference HIV-1 by dendritic cells, TGF-β1 had a minimal effect on trans infection of transmitted/founder variants irrespective of the reporter system used to measure trans infection.Our results provide the first direct evidence for uterine epithelial cell regulation of dendritic cell transmission of infection with reference and transmitted/founder HIV-1 variants. These findings have immediate implications for designing strategies to prevent sexual transmission of HIV-1

Phylogenomic analysis of the Chlamydomonas genome unmasks proteins potentially involved in photosynthetic function and regulation

Chlamydomonas reinhardtii, a unicellular green alga, has been exploited as a reference organism for identifying proteins and activities associated with the photosynthetic apparatus and the functioning of chloroplasts. Recently, the full genome sequence of Chlamydomonas was generated and a set of gene models, representing all genes on the genome, was developed. Using these gene models, and gene models developed for the genomes of other organisms, a phylogenomic, comparative analysis was performed to identify proteins encoded on the Chlamydomonas genome which were likely involved in chloroplast functions (or specifically associated with the green algal lineage); this set of proteins has been designated the GreenCut. Further analyses of those GreenCut proteins with uncharacterized functions and the generation of mutant strains aberrant for these proteins are beginning to unmask new layers of functionality/regulation that are integrated into the workings of the photosynthetic apparatus

A parsimonious cognitive architecture for human-computer interactive musical free improvisation

This paper presents some of the historical and theoretical foundations for a new cognitive architecture for human-computer interactive musical free improvisation. The architecture is parsimonious in that it has no access to musical knowledge and no domain-general subsystems, such as memory or representational abilities. The paper first describes some of the features and limitations of the architecture. It then illustrates how this architecture draws on insights from cybernetics, artificial life, artificial intelligence and ecological theory by situating it within a historical context. The context presented consists of a few key developments in the history of biologically-inspired robotics, followed by an indication of how they connect to James Gibson’s ecological theory. Finally, it describes how a recent approach to musicology informed by ecological theory bears on an implementation of this architecture

Predicting epidemic thresholds on complex networks: Limitations of mean-field approaches

Over the last decade considerable research effort has been invested in an attempt to understand the dynamics of viruses as they spread through complex networks, be they the networks in human population, computers or otherwise. The efforts have contributed to an understanding of epidemic behavior in random networks, but were generally unable to accommodate specific nonrandom features of the network's actual topology. Recently, though still in the context of the mean field theory, . Chakrabarti et al. (2008) proposed a model that intended to take into account the graph's specific topology and solve a longstanding problem regarding epidemic thresholds in both random and nonrandom networks. Here we review previous theoretical work dealing with this problem (usually based on mean field approximations) and show with several relevant and concrete counter examples that results to date breakdown for nonrandom topologies

Potential for Oilseed Sunflowers in the United States

The rising prominence of sunflower oil in world edible oil markets has stimulated increased interest in expanded U.S. production. U.S. acreage devoted to oilseed sunflowers has expanded rapidly, with over 600,000 acres grown in 1972. Production has been concentrated in the southern Cotton Belt States and in the Red River Valley area of Minnesota and North Dakota. This report examines recent trends in domestic fats and oils markets and the possible place of sunflower oil in these markets. Estimates are developed of the yield per acre and price required of sunflowerseed to make it competitive with established crops in the two areas. Also, estimates are made of the costs and profitability of processing sunflowerseed in southern screw-press mills