Continuous Reinforcement Learning-based Dynamic Difficulty Adjustment in a Visual Working Memory Game

Dynamic Difficulty Adjustment (DDA) is a viable approach to enhance a player's experience in video games. Recently, Reinforcement Learning (RL) methods have been employed for DDA in non-competitive games; nevertheless, they rely solely on discrete state-action space with a small search space. In this paper, we propose a continuous RL-based DDA methodology for a visual working memory (VWM) game to handle the complex search space for the difficulty of memorization. The proposed RL-based DDA tailors game difficulty based on the player's score and game difficulty in the last trial. We defined a continuous metric for the difficulty of memorization. Then, we consider the task difficulty and the vector of difficulty-score as the RL's action and state, respectively. We evaluated the proposed method through a within-subject experiment involving 52 subjects. The proposed approach was compared with two rule-based difficulty adjustment methods in terms of player's score and game experience measured by a questionnaire. The proposed RL-based approach resulted in a significantly better game experience in terms of competence, tension, and negative and positive affect. Players also achieved higher scores and win rates. Furthermore, the proposed RL-based DDA led to a significantly less decline in the score in a 20-trial session

Effectiveness of Virtual/Augmented Reality–Based Therapeutic Interventions on Individuals With Autism Spectrum Disorder: A Comprehensive Meta-Analysis

In recent years, the application of virtual reality (VR) for therapeutic purposes has escalated dramatically. Favorable properties of VR for engaging patients with autism, in particular, have motivated an enormous body of investigations targeting autism-related disabilities with this technology. This study aims to provide a comprehensive meta-analysis for evaluating the effectiveness of VR on the rehabilitation and training of individuals diagnosed with an autism spectrum disorder. Accordingly, we conducted a systematic search of related databases and, after screening for inclusion criteria, reviewed 33 studies for more detailed analysis. Results revealed that individuals undergoing VR training have remarkable improvements with a relatively large effect size with Hedges g of 0.74. Furthermore, the results of the analysis of different skills indicated diverse effectiveness. The strongest effect was observed for daily living skills (g = 1.15). This effect was moderate for other skills: g = 0.45 for cognitive skills, g = 0.46 for emotion regulation and recognition skills, and g = 0.69 for social and communication skills. Moreover, five studies that had used augmented reality also showed promising efficacy (g = 0.92) that calls for more research on this tool. In conclusion, the application of VR-based settings in clinical practice is highly encouraged, although their standardization and customization need more research

Short-term and long-term mate preference in men and women in an Iranian population.

Mate preference in short-term relationships and long-term ones may depend on many physical, psychological, and socio-cultural factors. In this study, 178 students (81 females) in sports and 153 engineering students (64 females) answered the systemizing quotient (SQ) and empathizing quotient (EQ) questionnaires and had their digit ratio measured. They rated their preferred mate on 12 black-line drawing body figures varying in body mass index (BMI) and waist to hip ratio (WHR) for short-term and long-term relationships. Men relative to women preferred lower WHR and BMI for mate selection for both short-term and long-term relationships. BMI and WHR preference in men is independent of each other, but has a negative correlation in women. For men, digit ratio was inversely associated with BMI (p = 0.039, B = - 0.154) preference in a short-term relationship, and EQ was inversely associated with WHR preference in a long-term relationship (p = 0.045, B = - 0.164). Furthermore, men and women in sports, compared to engineering students, preferred higher (p = 0.009, B = 0.201) and lower BMI (p = 0.034, B = - 0.182) for short-term relationships, respectively. Women were more consistent in their preferences for short-term and long-term relationships relative to men. Both biological factors and social/experiential factors contribute to mate preferences in men while in women, mostly social/experiential factors contribute to them

Dopaminergic Modulation of Synaptic Plasticity, Its Role in Neuropsychiatric Disorders, and Its Computational Modeling

Neuromodulators modify intrinsic characteristics of the nervous system in order to reconfigure the functional properties of neural circuits. This reconfiguration is crucial for the flexibility of the nervous system to respond on an input-modulated basis. Such a functional rearrangement is realized by modification of intrinsic properties of the neural circuits including synaptic interactions. Dopamine is an important neuromodulator involved in motivation and stimulus-reward learning process, and adjusts synaptic dynamics in multiple time scales through different pathways. The modification of synaptic plasticity by dopamine underlies the change in synaptic transmission and integration mechanisms, which affects intrinsic properties of the neural system including membrane excitability, probability of neurotransmitters release, receptors’ response to neurotransmitters, protein trafficking, and gene transcription. Dopamine also plays a central role in behavioral control, whereas its malfunction can cause cognitive disorders. Impaired dopamine signaling is implicated in several neuropsychiatric disorders such as Parkinson’s disease, drug addiction, schizophrenia, attention-deficit/hyperactivity disorder, obsessive-compulsive disorder and Tourette’s syndrome. Therefore, dopamine plays a crucial role in the nervous system, where its proper modulation of neural circuits may enhance plasticity-related procedures, but disturbances in dopamine signaling might be involved in numerous neuropsychiatric disorders. In recent years, several computational models are proposed to formulate the involvement of dopamine in synaptic plasticity or neuropsychiatric disorders and address their connection based on the experimental findings

A Scalable FPGA Architecture for Randomly Connected Networks of Hodgkin-Huxley Neurons

Human intelligence relies on the vast number of neurons and their interconnections that form a parallel computing engine. If we tend to design a brain-like machine, we will have no choice but to employ many spiking neurons, each one has a large number of synapses. Such a neuronal network is not only compute-intensive but also memory-intensive. The performance and the configurability of the modern FPGAs make them suitable hardware solutions to deal with these challenges. This paper presents a scalable architecture to simulate a randomly connected network of Hodgkin-Huxley neurons. To demonstrate that our architecture eliminates the need to use a high-end device, we employ the XC7A200T, a member of the mid-range Xilinx Artix®-7 family, as our target device. A set of techniques are proposed to reduce the memory usage and computational requirements. Here we introduce a multi-core architecture in which each core can update the states of a group of neurons stored in its corresponding memory bank. The proposed system uses a novel method to generate the connectivity vectors on the fly instead of storing them in a huge memory. This technique is based on a cyclic permutation of a single prestored connectivity vector per core. Moreover, to reduce both the resource usage and the computational latency even more, a novel approximate two-level counter is introduced to count the number of the spikes at the synapse for the sparse network. The first level is a low cost saturated counter implemented on FPGA lookup tables that reduces the number of inputs to the second level exact adder tree. It, therefore, results in much lower hardware cost for the counter circuit. These techniques along with pipelining make it possible to have a high-performance, scalable architecture, which could be configured for either a real-time simulation of up to 5120 neurons or a large-scale simulation of up to 65536 neurons in an appropriate execution time on a cost-optimized FPGA

Attention Modulation Effects on Visual Feature-selectivity of Neurons inBrain-inspired Categorization Models

Most Brain-inspired Visual Object Recognition Models(BVORMs) do not consider local and global reciprocal con-nections in visual pathway. We addressed this weakness and implemented an attention modulation mechanism based on feed-back connections in BVORMs, where feature-selectivity is shaped and modulated by categorization of objects based on theirvisual features. This modification is inspired by the top-down neuromodulatory signals that make changes in post-synapticactivities of the feature-selective neurons. We also incorporated an implicit memory unit in BVORMs to accumulate recentHebbian synaptic plasticity’s of the neurons in each task. This mechanism guides the top-down feature-based attention modula-tion to retrieve the interrelated feature-selectivity pattern for each task.HMax and CNN models were used as two BVORMs andtested on a visual categorization problem: natural versus artificial objects in CALTECH-256. Based on experimental results,our proposed modifications not only increased their biological-plausibility but also significantly improved their categorizationaccuracies compared to the original models

Effects of methylphenidate on reinforcement learning depend on working memory capacity

RATIONALE: Brain catecholamines have long been implicated in reinforcement learning, exemplified by catecholamine drug and genetic effects on probabilistic reversal learning. However, the mechanisms underlying such effects are unclear. OBJECTIVES AND METHODS: Here we investigated effects of an acute catecholamine challenge with methylphenidate (20 mg, oral) on a novel probabilistic reversal learning paradigm in a within-subject, double-blind randomised design. The paradigm was designed to disentangle effects on punishment avoidance from effects on reward perseveration. Given the known large individual variability in methylphenidate’s effects, we stratified our effects by working memory capacity and trait impulsivity, putatively modulating the effects of methylphenidate, in a large sample (n = 102) of healthy volunteers. RESULTS: Contrary to our prediction, methylphenidate did not alter performance in the reversal phase of the task. Our key finding is that methylphenidate altered learning of choice-outcome contingencies in a manner that depended on individual variability in working memory span. Specifically, methylphenidate improved performance by adaptively reducing the effective learning rate in participants with higher working memory capacity. CONCLUSIONS: This finding emphasises the important role of working memory in reinforcement learning, as reported in influential recent computational modelling and behavioural work, and highlights the dependence of this interplay on catecholaminergic function. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s00213-021-05974-w