Temporal-Difference Reinforcement Learning with Distributed Representations

Temporal-difference (TD) algorithms have been proposed as models of reinforcement learning (RL). We examine two issues of distributed representation in these TD algorithms: distributed representations of belief and distributed discounting factors. Distributed representation of belief allows the believed state of the world to distribute across sets of equivalent states. Distributed exponential discounting factors produce hyperbolic discounting in the behavior of the agent itself. We examine these issues in the context of a TD RL model in which state-belief is distributed over a set of exponentially-discounting “micro-Agents”, each of which has a separate discounting factor (γ). Each µAgent maintains an independent hypothesis about the state of the world, and a separate value-estimate of taking actions within that hypothesized state. The overall agent thus instantiates a flexible representation of an evolving world-state. As with other TD models, the value-error (δ) signal within the model matches dopamine signals recorded from animals in standard conditioning reward-paradigms. The distributed representation of belief provides an explanation for the decrease in dopamine at the conditioned stimulus seen in overtrained animals, for the differences between trace and delay conditioning, and for transient bursts of dopamine seen at movement initiation. Because each µAgent also includes its own exponential discounting factor, the overall agent shows hyperbolic discounting, consistent with behavioral experiments

Genetic variants linked to education predict longevity

Educational attainment is associated with many health outcomes, including longevity. It is also known to be substantially heritable. Here, we used data from three large genetic epidemiology cohort studies (Generation Scotland, n = ∼17,000; UK Biobank, n = ∼115,000; and the Estonian Biobank, n = ∼6,000) to test whether education-linked genetic variants can predict lifespan length. We did so by using cohort members’ polygenic profile score for education to predict their parents’ longevity. Across the three cohorts, meta-analysis showed that a 1 SD higher polygenic education score was associated with ∼2.7% lower mortality risk for both mothers (total ndeaths = 79,702) and ∼2.4% lower risk for fathers (total ndeaths = 97,630). On average, the parents of offspring in the upper third of the polygenic score distribution lived 0.55 y longer compared with those of offspring in the lower third. Overall, these results indicate that the genetic contributions to educational attainment are useful in the prediction of human longevity.</p

The molecular genetic architecture of economic and political preferences

Preferences are fundamental building blocks in all models of economic and political behavior. We study a new sample of comprehensively genotyped subjects with data on economic and political preferences and educational attainment. We use dense single nucleotide polymorphism (SNP) data to estimate the proportion of variation in these traits explained by common SNPs and to conduct genome-wide association study (GWAS) and prediction analyses. The pattern of results is consistent with findings for other complex traits. First, the estimated fraction of phenotypic variation that could, in principle, be explained by dense SNP arrays is around one-half of the narrow heritability estimated using twin and family samples. The molecular-genetic–based heritability estimates, therefore, partially corroborate evidence of significant heritability from behavior genetic studies. Second, our analyses suggest that these traits have a polygenic architecture, with the heritable variation explained by many genes with small effects. Our results suggest that most published genetic association studies with economic and political traits are dramatically underpowered, which implies a high false discovery rate. These results convey a cautionary message for whether, how, and how soon molecular genetic data can contribute to, and potentially transform, research in social science. We propose some constructive responses to the inferential challenges posed by the small explanatory power of individual SNPs

Uncertain survival and time discounting: intertemporal consumption plans for family trusts

Intertemporal choice, Family extinction, Hyperbolic discounting, G0, D91, D81,

Social security and conflict within the family

Social security, Savings, Bargaining, H55, D13, D91,

Risk and time preferences: linking experimental and household survey data from Vietnam

We conducted experiments in Vietnamese villages to determine the predictors of risk and time preferences. In villages with higher mean income, people are less loss-averse and more patient. Household income is correlated with patience but not with risk. We expand measurements of risk and time preferences beyond expected utility and exponential discounting, replacing those models with prospect theory and a three-parameter hyperbolic discounting model. Comparable risk parameter estimates have been found for Chinese farmers, using our method