Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Mammalian gene expression variability is explained by underlying cell state.

Gene expression variability in mammalian systems plays an important role in physiological and pathophysiological conditions. This variability can come from differential regulation related to cell state (extrinsic) and allele-specific transcriptional bursting (intrinsic). Yet, the relative contribution of these two distinct sources is unknown. Here, we exploit the qualitative difference in the patterns of covariance between these two sources to quantify their relative contributions to expression variance in mammalian cells. Using multiplexed error robust RNA fluorescent in situ hybridization (MERFISH), we measured the multivariate gene expression distribution of 150 genes related to Ca2+ signaling coupled with the dynamic Ca2+ response of live cells to ATP. We show that after controlling for cellular phenotypic states such as size, cell cycle stage, and Ca2+ response to ATP, the remaining variability is effectively at the Poisson limit for most genes. These findings demonstrate that the majority of expression variability results from cell state differences and that the contribution of transcriptional bursting is relatively minimal

Growth, Economic Development and Structural Transition in Small Vulnerable States

Small states, Micro-states, Growth, Vulnerability, Development, Policy strategies

Habit Learning by Naive Macaques Is Marked by Response Sharpening of Striatal Neurons Representing the Cost and Outcome of Acquired Action Sequences

Over a century of scientific work has focused on defining the factors motivating behavioral learning. Observations in animals and humans trained on a wide range of tasks support reinforcement learning (RL) algorithms as accounting for the learning. Still unknown, however, are the signals that drive learning in naive, untrained subjects. Here, we capitalized on a sequential saccade task in which macaque monkeys acquired repetitive scanning sequences without instruction. We found that spike activity in the caudate nucleus after each trial corresponded to an integrated cost-benefit signal that was highly correlated with the degree of naturalistic untutored learning by the monkeys. Across learning, neurons encoding both cost and outcome gradually acquired increasingly sharp phasic trial-end responses that paralleled the development of the habit-like, repetitive saccade sequences. Our findings demonstrate an integrated cost-benefit signal by which RL and its neural correlates could drive naturalistic behaviors in freely behaving primates. Video Abstract: Feedback about the costs and benefits of our actions is an essential part of how we learn. Desrochers et al. show that neurons in the striatum of monkeys develop combined cost-benefit signals marking movement sequences that they acquire without explicit training.National Institutes of Health (U.S.) (Grant R01 EY012848)National Institutes of Health (U.S.) (Grant R01 NS025529)United States. Defense Advanced Research Projects Agency (Grant NBCHC070105)United States. Office of Naval Research (Grant N00014-07-1-0903

Habit Learning by Naive Macaques Is Marked by Response Sharpening of Striatal Neurons Representing the Cost and Outcome of Acquired Action Sequences

SummaryOver a century of scientific work has focused on defining the factors motivating behavioral learning. Observations in animals and humans trained on a wide range of tasks support reinforcement learning (RL) algorithms as accounting for the learning. Still unknown, however, are the signals that drive learning in naive, untrained subjects. Here, we capitalized on a sequential saccade task in which macaque monkeys acquired repetitive scanning sequences without instruction. We found that spike activity in the caudate nucleus after each trial corresponded to an integrated cost-benefit signal that was highly correlated with the degree of naturalistic untutored learning by the monkeys. Across learning, neurons encoding both cost and outcome gradually acquired increasingly sharp phasic trial-end responses that paralleled the development of the habit-like, repetitive saccade sequences. Our findings demonstrate an integrated cost-benefit signal by which RL and its neural correlates could drive naturalistic behaviors in freely behaving primates.Video Abstrac

The total assessment profile, volume 2

Appendices are presented which include discussions of interest formulas, factors in regionalization, parametric modeling of discounted benefit-sacrifice streams, engineering economic calculations, and product innovation. For Volume 1, see

Topics in Evolutionary Ecology

46 pages, 1 article*Topics in Evolutionary Ecology* (Levin, Simon A.; Castillo-Chavez, Carlos) 46 page

Disentangling the genetic and morphological structure of Patella candei complex in Macaronesia (NE Atlantic)

The uptake of natural living resources for human consumption has triggered serious changes in the balance of ecosystems. In the archipelagos of Macaronesia (NE Atlantic), limpets have been extensively exploited probably since islands were first colonized. This has led to profound consequences in the dynamics of rocky shore communities. The Patella candei complex includes various subspecies of limpets that are ascribed to a particular archipelago and has been the focus of several taxonomic surveys without much agreement. Under a conservational perspective, we apply morphometric and genetic analyses to test subspecies boundaries in P. candei and to evaluate its current population connectivity throughout Macaronesia (Azores, Madeira, and Canaries). A highly significant genetic break between archipelagos following isolation by distance was detected (FST = 0.369, p < .001). Contrastingly, significant genetic differentiation among islands (i.e., Azores) was absent possibly indicating ongoing gene flow via larval exchange between populations. Significant shell-shape differences among archipelagos were also detected using both distance-based and geometric morphometric analyses. Adaptive processes associated with niche differentiation and strong barriers to gene flow among archipelagos may be the mechanisms underlying P. candei diversification in Macaronesia. Under the very probable assumption that populations of P. candei from each archipelago are geographically and/or ecologically isolated populations, the various subspecies within the P. candei complex may be best thought of as true species using the denomination: P. candei in Selvagens, Patella gomesii in Azores, Patella ordinaria in Madeira, and Patella crenata for Canaries. This would be in agreement with stock delimitation and units of conservation of P. candei sensu latu along Macaronesia

Eye movements in the wild : Oculomotor control, gaze behavior & frames of reference

Understanding the brain's capacity to encode complex visual information from a scene and to transform it into a coherent perception of 3D space and into well-coordinated motor commands are among the outstanding questions in the study of integrative brain function. Eye movement methodologies have allowed us to begin addressing these questions in increasingly naturalistic tasks, where eye and body movements are ubiquitous and, therefore, the applicability of most traditional neuroscience methods restricted. This review explores foundational issues in (1) how oculomotor and motor control in lab experiments extrapolates into more complex settings and (2) how real-world gaze behavior in turn decomposes into more elementary eye movement patterns. We review the received typology of oculomotor patterns in laboratory tasks, and how they map onto naturalistic gaze behavior (or not). We discuss the multiple coordinate systems needed to represent visual gaze strategies, how the choice of reference frame affects the description of eye movements, and the related but conceptually distinct issue of coordinate transformations between internal representations within the brain.Peer reviewe