Reinforcement Learning With Parsimonious Computation and a Forgetting Process

Decision-making is assumed to be supported by model-free and model-based systems: the model-free system is based purely on experience, while the model-based system uses a cognitive map of the environment and is more accurate. The recently developed multistep decision-making task and its computational model can dissociate the contributions of the two systems and have been used widely. This study used this task and model to understand our value-based learning process and tested alternative algorithms for the model-free and model-based learning systems. The task used in this study had a deterministic transition structure, and the degree of use of this structure in learning is estimated as the relative contribution of the model-based system to choices. We obtained data from 29 participants and fitted them with various computational models that differ in the model-free and model-based assumptions. The results of model comparison and parameter estimation showed that the participants update the value of action sequences and not each action. Additionally, the model fit was improved substantially by assuming that the learning mechanism includes a forgetting process, where the values of unselected options change to a certain default value over time. We also examined the relationships between the estimated parameters and psychopathology and other traits measured by self-reported questionnaires, and the results suggested that the difference in model assumptions can change the conclusion. In particular, inclusion of the forgetting process in the computational models had a strong impact on estimation of the weighting parameter of the model-free and model-based systems

Complex sequencing rules of birdsong can be explained by simple hidden Markov processes

Complex sequencing rules observed in birdsongs provide an opportunity to investigate the neural mechanism for generating complex sequential behaviors. To relate the findings from studying birdsongs to other sequential behaviors, it is crucial to characterize the statistical properties of the sequencing rules in birdsongs. However, the properties of the sequencing rules in birdsongs have not yet been fully addressed. In this study, we investigate the statistical propertiesof the complex birdsong of the Bengalese finch (Lonchura striata var. domestica). Based on manual-annotated syllable sequences, we first show that there are significant higher-order context dependencies in Bengalese finch songs, that is, which syllable appears next depends on more than one previous syllable. This property is shared with other complex sequential behaviors. We then analyze acoustic features of the song and show that higher-order context dependencies can be explained using first-order hidden state transition dynamics with redundant hidden states. This model corresponds to hidden Markov models (HMMs), well known statistical models with a large range of application for time series modeling. The song annotation with these models with first-order hidden state dynamics agreed well with manual annotation, the score was comparable to that of a second-order HMM, and surpassed the zeroth-order model (the Gaussian mixture model (GMM)), which does not use context information. Our results imply that the hierarchical representation with hidden state dynamics may underlie the neural implementation for generating complex sequences with higher-order dependencies

Consolidated bioprocessing of corn cob-derived hemicellulose: engineered industrial Saccharomyces cerevisiae as efficient whole cell biocatalysts

Background Consolidated bioprocessing, which combines saccharolytic and fermentative abilities in a single microorganism, is receiving increased attention to decrease environmental and economic costs in lignocellulosic biorefineries. Nevertheless, the economic viability of lignocellulosic ethanol is also dependent of an efficient utilization of the hemicellulosic fraction, which contains xylose as a major component in concentrations that can reach up to 40% of the total biomass in hardwoods and agricultural residues. This major bottleneck is mainly due to the necessity of chemical/enzymatic treatments to hydrolyze hemicellulose into fermentable sugars and to the fact that xylose is not readily consumed by Saccharomyces cerevisiaethe most used organism for large-scale ethanol production. In this work, industrial S. cerevisiae strains, presenting robust traits such as thermotolerance and improved resistance to inhibitors, were evaluated as hosts for the cell-surface display of hemicellulolytic enzymes and optimized xylose assimilation, aiming at the development of whole-cell biocatalysts for consolidated bioprocessing of corn cob-derived hemicellulose. Results These modifications allowed the direct production of ethanol from non-detoxified hemicellulosic liquor obtained by hydrothermal pretreatment of corn cob, reaching an ethanol titer of 11.1 g/L corresponding to a yield of 0.328 g/g of potential xylose and glucose, without the need for external hydrolytic catalysts. Also, consolidated bioprocessing of pretreated corn cob was found to be more efficient for hemicellulosic ethanol production than simultaneous saccharification and fermentation with addition of commercial hemicellulases. Conclusions These results show the potential of industrial S. cerevisiae strains for the design of whole-cell biocatalysts and paves the way for the development of more efficient consolidated bioprocesses for lignocellulosic biomass valorization, further decreasing environmental and economic costs.This work has been carried out at the Biomass and Bioenergy Research Infrastructure (BBRI)-LISBOA-01-0145-FEDER-022059, supported by Operational Programme for Competitiveness and Internationalization (PORTUGAL2020), by Lisbon Portugal Regional Operational Programme (Lisboa 2020) and by North Portugal Regional Operational Programme (Norte 2020) under the Portugal 2020 Partnership Agreement, through the European Regional Development Fund (ERDF) and has been supported by the Portuguese Foundation for Science and Technology (FCT) under the scope of the strategic funding of UIDB/04469/2020, the “Contrato-Programa” UIDB/04050/2020, the MIT-Portugal Program (Ph.D. Grant PD/BD/128247/2016 to Joana T. Cunha) and through Project FatVal (POCI-01-0145-FEDER-032506) and BioTecNorte operation (NORTE-01-0145-FEDER-000004) funded by the European Regional Development Fund under the scope of Norte2020 - Programa Operacional Regional do Norte.info:eu-repo/semantics/publishedVersio

Revisiting the importance of model fitting for model-based fMRI: It does matter in computational psychiatry.

Computational modeling has been applied for data analysis in psychology, neuroscience, and psychiatry. One of its important uses is to infer the latent variables underlying behavior by which researchers can evaluate corresponding neural, physiological, or behavioral measures. This feature is especially crucial for computational psychiatry, in which altered computational processes underlying mental disorders are of interest. For instance, several studies employing model-based fMRI-a method for identifying brain regions correlated with latent variables-have shown that patients with mental disorders (e.g., depression) exhibit diminished neural responses to reward prediction errors (RPEs), which are the differences between experienced and predicted rewards. Such model-based analysis has the drawback that the parameter estimates and inference of latent variables are not necessarily correct-rather, they usually contain some errors. A previous study theoretically and empirically showed that the error in model-fitting does not necessarily cause a serious error in model-based fMRI. However, the study did not deal with certain situations relevant to psychiatry, such as group comparisons between patients and healthy controls. We developed a theoretical framework to explore such situations. We demonstrate that the parameter-misspecification can critically affect the results of group comparison. We demonstrate that even if the RPE response in patients is completely intact, a spurious difference to healthy controls is observable. Such a situation occurs when the ground-truth learning rate differs between groups but a common learning rate is used, as per previous studies. Furthermore, even if the parameters are appropriately fitted to individual participants, spurious group differences in RPE responses are observable when the model lacks a component that differs between groups. These results highlight the importance of appropriate model-fitting and the need for caution when interpreting the results of model-based fMRI