Reinforcement Learning With Parsimonious Computation and a Forgetting Process

Decision-making is assumed to be supported by model-free and model-based systems: the model-free system is based purely on experience, while the model-based system uses a cognitive map of the environment and is more accurate. The recently developed multistep decision-making task and its computational model can dissociate the contributions of the two systems and have been used widely. This study used this task and model to understand our value-based learning process and tested alternative algorithms for the model-free and model-based learning systems. The task used in this study had a deterministic transition structure, and the degree of use of this structure in learning is estimated as the relative contribution of the model-based system to choices. We obtained data from 29 participants and fitted them with various computational models that differ in the model-free and model-based assumptions. The results of model comparison and parameter estimation showed that the participants update the value of action sequences and not each action. Additionally, the model fit was improved substantially by assuming that the learning mechanism includes a forgetting process, where the values of unselected options change to a certain default value over time. We also examined the relationships between the estimated parameters and psychopathology and other traits measured by self-reported questionnaires, and the results suggested that the difference in model assumptions can change the conclusion. In particular, inclusion of the forgetting process in the computational models had a strong impact on estimation of the weighting parameter of the model-free and model-based systems

An attenuated vaccinia vaccine encoding the severe acute respiratory syndrome coronavirus-2 spike protein elicits broad and durable immune responses, and protects cynomolgus macaques and human angiotensin-converting enzyme 2 transgenic mice from severe acute respiratory syndrome coronavirus-2 and its variants

As long as the coronavirus disease-2019 (COVID-19) pandemic continues, new variants of severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) with altered antigenicity will emerge. The development of vaccines that elicit robust, broad, and durable protection against SARS-CoV-2 variants is urgently required. We have developed a vaccine consisting of the attenuated vaccinia virus Dairen-I (DIs) strain platform carrying the SARS-CoV-2 S gene (rDIs-S). rDIs-S induced neutralizing antibody and T-lymphocyte responses in cynomolgus macaques and human angiotensin-converting enzyme 2 (hACE2) transgenic mice, and the mouse model showed broad protection against SARS-CoV-2 isolates ranging from the early-pandemic strain (WK-521) to the recent Omicron BA.1 variant (TY38-873). Using a tandem mass tag (TMT)-based quantitative proteomic analysis of lung homogenates from hACE2 transgenic mice, we found that, among mice subjected to challenge infection with WK-521, vaccination with rDIs-S prevented protein expression related to the severe pathogenic effects of SARS-CoV-2 infection (tissue destruction, inflammation, coagulation, fibrosis, and angiogenesis) and restored protein expression related to immune responses (antigen presentation and cellular response to stress). Furthermore, long-term studies in mice showed that vaccination with rDIs-S maintains S protein-specific antibody titers for at least 6 months after a first vaccination. Thus, rDIs-S appears to provide broad and durable protective immunity against SARS-CoV-2, including current variants such as Omicron BA.1 and possibly future variants

Predictors of Enhancing Human Physical Attractiveness: Data from 93 Countries

People across the world and throughout history have gone to great lengths to enhance their physical appearance. Evolutionary psychologists and ethologists have largely attempted to explain this phenomenon via mating preferences and strategies. Here, we test one of the most popular evolutionary hypotheses for beauty-enhancing behaviors, drawn from mating market and parasite stress perspectives, in a large cross-cultural sample. We also test hypotheses drawn from other influential and non-mutually exclusive theoretical frameworks, from biosocial role theory to a cultural media perspective. Survey data from 93,158 human participants across 93 countries provide evidence that behaviors such as applying makeup or using other cosmetics, hair grooming, clothing style, caring for body hygiene, and exercising or following a specific diet for the specific purpose of improving ones physical attractiveness, are universal. Indeed, 99% of participants reported spending \u3e10 min a day performing beauty-enhancing behaviors. The results largely support evolutionary hypotheses: more time was spent enhancing beauty by women (almost 4 h a day, on average) than by men (3.6 h a day), by the youngest participants (and contrary to predictions, also the oldest), by those with a relatively more severe history of infectious diseases, and by participants currently dating compared to those in established relationships. The strongest predictor of attractiveness-enhancing behaviors was social media usage. Other predictors, in order of effect size, included adhering to traditional gender roles, residing in countries with less gender equality, considering oneself as highly attractive or, conversely, highly unattractive, TV watching time, higher socioeconomic status, right-wing political beliefs, a lower level of education, and personal individualistic attitudes. This study provides novel insight into universal beauty-enhancing behaviors by unifying evolutionary theory with several other complementary perspectives

Revisiting the importance of model fitting for model-based fMRI: It does matter in computational psychiatry.

Computational modeling has been applied for data analysis in psychology, neuroscience, and psychiatry. One of its important uses is to infer the latent variables underlying behavior by which researchers can evaluate corresponding neural, physiological, or behavioral measures. This feature is especially crucial for computational psychiatry, in which altered computational processes underlying mental disorders are of interest. For instance, several studies employing model-based fMRI-a method for identifying brain regions correlated with latent variables-have shown that patients with mental disorders (e.g., depression) exhibit diminished neural responses to reward prediction errors (RPEs), which are the differences between experienced and predicted rewards. Such model-based analysis has the drawback that the parameter estimates and inference of latent variables are not necessarily correct-rather, they usually contain some errors. A previous study theoretically and empirically showed that the error in model-fitting does not necessarily cause a serious error in model-based fMRI. However, the study did not deal with certain situations relevant to psychiatry, such as group comparisons between patients and healthy controls. We developed a theoretical framework to explore such situations. We demonstrate that the parameter-misspecification can critically affect the results of group comparison. We demonstrate that even if the RPE response in patients is completely intact, a spurious difference to healthy controls is observable. Such a situation occurs when the ground-truth learning rate differs between groups but a common learning rate is used, as per previous studies. Furthermore, even if the parameters are appropriately fitted to individual participants, spurious group differences in RPE responses are observable when the model lacks a component that differs between groups. These results highlight the importance of appropriate model-fitting and the need for caution when interpreting the results of model-based fMRI

Examinations of Biases by Model Misspecification and Parameter Reliability of Reinforcement Learning Models

Reinforcement learning models have the potential to clarify meaningful individual differences in the decision-making process. This study focused on two aspects regarding the nature of a reinforcement learning model and its parameters: the problems of model misspecification and reliability. Online participants, N=453, completed self-report measures and a probabilistic learning task twice 1.5 months apart, and data from the task were fitted using several reinforcement learning models. To address the problem of model misspecification, we compared the models with and without the influence of choice history, or perseveration. Results showed that the lack of a perseveration term in the model led to a decrease in learning rates for win and loss outcomes, with slightly different influences depending on outcome volatility, and increases in inverse temperature. We also conducted simulations to examine the mechanism of the observed biases and revealed that failure to incorporate perseveration directly affected the estimation bias in the learning rate and indirectly affected that in inverse temperature. Furthermore, in both model fittings and model simulations, the lack of perseveration caused win-stay probability underestimation and loss-shift probability overestimation. We also assessed the parameter reliability. Test-retest reliabilities were poor (learning rates) to moderate (inverse temperature and perseveration magnitude). A learning effect was noted in the inverse temperature and perseveration magnitude parameters, showing an increment of the estimates in the second session. We discuss possible misinterpretations of results and limitations considering the estimation biases and parameter reliability

Does the reliability of computational models truly improve with hierarchical modeling? Some recommendations and considerations for the assessment of model parameter reliability

Computational modeling of behavior is increasingly being adopted as a standard methodology in psychology, cognitive neuroscience, and computational psychiatry. This approach involves estimating parameters in a computational (or cognitive) model that represents the computational processes of the underlying behavior. In this approach, the reliability of the parameter estimates is an important issue. The use of hierarchical (Bayesian) approaches, which place a prior on each model parameter of the individual participants, is thought to improve the reliability of the parameters. However, the characteristics of reliability in parameter estimates, especially when individual-level priors are assumed, as in hierarchical models, have not yet been fully discussed. Furthermore, the suitability of different reliability measures for assessing parameter reliability is not thoroughly understood. In this study, we conduct a systematic examination of these issues through theoretical analysis and numerical simulations, focusing specifically on reinforcement learning models. We note that the heterogeneity in the estimation precision of individual parameters, particularly with priors, can skew reliability measures toward individuals with higher precision. We further note that there are two factors that reduce reliability, namely, estimation error and intersession variation in the true parameters, and we discuss how to evaluate these factors separately. Based on the considerations of this study, we present several recommendations and cautions for assessing the reliability of the model parameters

Brain-immune interaction accompanying odor-evoked autobiographic memory.

The phenomenon in which a certain smell evokes a specific memory is known as the Proust phenomenon. Odor-evoked autobiographic memories are more emotional than those elicited by other sensory stimuli. The results of our previous study indicated that odor-evoked autobiographic memory accompanied by positive emotions has remarkable effects on various psychological and physiological activities, including the secretion of cytokines, which are immune-signaling molecules that modulate systemic inflammation. In this study, we aimed to clarify the neural substrates associated with the interaction between odor-evoked autobiographic memory and peripheral circulating cytokines. We recruited healthy male and female volunteers and investigated the association between brain responses and the concentration of several cytokines in the plasma by using positron emission tomography (PET) recordings when an autographic memory was evoked in participants by asking them to smell an odor that was nostalgic to them. Participants experienced positive emotions and autobiographic memories when nostalgic odors were presented to them. The levels of peripheral proinflammatory cytokines, such as the tumor necrosis factor-α (TNF-α) and interferon-γ (IFN-γ), were significantly reduced after experiencing odor-evoked autobiographic memory. Subtraction analysis of PET images indicated that the medial orbitofrontal cortex (mOFC) and precuneus/posterior cingulate cortex (PCC) were significantly activated during experiences of odor-evoked autobiographic memory. Furthermore, a correlation analysis indicated that activities of the mOFC and precuneus/PCC were negatively correlated with IFN-γ concentration. These results indicate that the neural networks including the precuneus/PCC and mOFC might regulate the secretion of peripheral proinflammatory cytokines during the experience of odor-evoked autobiographic memories accompanied with positive emotions

GATA6 coordinates cross-talk between BMP10 and oxidative stress axis in pulmonary arterial hypertension

Pulmonary arterial hypertension (PAH) is a life-threatening condition characterized by a progressive increase in pulmonary vascular resistance leading to right ventricular failure and often death. Here we report that deficiency of transcription factor GATA6 is a shared pathological feature of PA endothelial (PAEC) and smooth muscle cells (PASMC) in human PAH and experimental PH, which is responsible for maintenance of hyper-proliferative cellular phenotypes, pulmonary vascular remodeling and pulmonary hypertension. We further show that GATA6 acts as a transcription factor and direct positive regulator of anti-oxidant enzymes, and its deficiency in PAH/PH pulmonary vascular cells induces oxidative stress and mitochondrial dysfunction. We demonstrate that GATA6 is regulated by the BMP10/BMP receptors axis and its loss in PAECs and PASMC in PAH supports BMPR deficiency. In addition, we have established that GATA6-deficient PAEC, acting in a paracrine manner, increase proliferation and induce other pathological changes in PASMC, supporting the importance of GATA6 in pulmonary vascular cell communication. Treatment with dimethyl fumarate resolved oxidative stress and BMPR deficiency, reversed hemodynamic changes caused by endothelial Gata6 loss in mice, and inhibited proliferation and induced apoptosis in human PAH PASMC, strongly suggesting that targeting GATA6 deficiency may provide a therapeutic advance for patients with PAH

Correlation between peripheral cytokine levels and neuroimaging findings.

<p>(A) Scatter plot showing the correlation between the beta value of the mOFC cluster in the Proust condition and the beta value of the precuneus/PCC cluster in the Proust condition. (B) Scatter plot showing the correlation between the beta value of the precuneus/PCC cluster in the Proust condition and IFN-γ concentration after smelling a nostalgic odor that evoked an autobiographic memory. (C) Scatter plot showing the correlation between the beta value of the mOFC cluster in the Proust condition and IFN-γ concentration after smelling a nostalgic odor that evoked an autobiographic memory.</p