Dynamic Memory-based Curiosity: A Bootstrap Approach for Exploration

The sparsity of extrinsic rewards poses a serious challenge for reinforcement learning (RL). Currently, many efforts have been made on curiosity which can provide a representative intrinsic reward for effective exploration. However, the challenge is still far from being solved. In this paper, we present a novel curiosity for RL, named DyMeCu, which stands for Dynamic Memory-based Curiosity. Inspired by human curiosity and information theory, DyMeCu consists of a dynamic memory and dual online learners. The curiosity arouses if memorized information can not deal with the current state, and the information gap between dual learners can be formulated as the intrinsic reward for agents, and then such state information can be consolidated into the dynamic memory. Compared with previous curiosity methods, DyMeCu can better mimic human curiosity with dynamic memory, and the memory module can be dynamically grown based on a bootstrap paradigm with dual learners. On multiple benchmarks including DeepMind Control Suite and Atari Suite, large-scale empirical experiments are conducted and the results demonstrate that DyMeCu outperforms competitive curiosity-based methods with or without extrinsic rewards. We will release the code to enhance reproducibility

Motion sensors for knee angle recognition in muscle rehabilitation solutions

The progressive loss of functional capacity due to aging is a serious problem that can compromise human locomotion capacity, requiring the help of an assistant and reducing independence. The NanoStim project aims to develop a system capable of performing treatment with electrostimulation at the patient’s home, reducing the number of consultations. The knee angle is one of the essential attributes in this context, helping understand the patient’s movement during the treatment session. This article presents a wearable system that recognizes the knee angle through IMU sensors. The hardware chosen for the wearables are low cost, including an ESP32 microcontroller and an MPU-6050 sensor. However, this hardware impairs signal accuracy in the multitasking environment expected in rehabilitation treatment. Three optimization filters with algorithmic complexity O(1) were tested to improve the signal’s noise. The complementary filter obtained the best result, presenting an average error of 0.6 degrees and an improvement of 77% in MSE. Furthermore, an interface in the mobile app was developed to respond immediately to the recognized movement. The systems were tested with volunteers in a real environment and could successfully measure the movement performed. In the future, it is planned to use the recognized angle with the electromyography sensor.This work was funded by European Regional Development Fund (ERDF) through the Operational Programme for Competitiveness and Internationalization (COMPETE 2020), under Portugal 2020 in the framework of the NanoStim (POCI-01-0247-FEDER-045908) project, and Fundação para a Ciência e a Tecnologia under Projects UIDB/05757/2020, UIDB/00319/2020, and PhD grant 2020.05704.BD

Contrastive Masked Autoencoders are Stronger Vision Learners

Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{https://github.com/ZhichengHuang/CMAE}.Comment: Accepted by TPAM

Targeting Treatment-Resistant Auditory Verbal Hallucinations in Schizophrenia with fMRI-Based Neurofeedback – Exploring Different Cases of Schizophrenia

Auditory verbal hallucinations (AVHs) are a hallmark of schizophrenia and can significantly impair patients' emotional, social, and occupational functioning. Despite progress in psychopharmacology, over 25% of schizophrenia patients suffer from treatment-resistant hallucinations. In the search for alternative treatment methods, neurofeedback (NF) emerges as a promising therapy tool. NF based on real-time functional magnetic resonance imaging (rt-fMRI) allows voluntarily change of the activity in a selected brain region - even in patients with schizophrenia. This study explored effects of NF on ongoing AVHs. The selected participants were trained in the self-regulation of activity in the anterior cingulate cortex (ACC), a key monitoring region involved in generation and intensity modulation of AVHs. Using rt-fMRI, three right-handed patients, suffering from schizophrenia and ongoing, treatment-resistant AVHs, learned control over ACC activity on three separate days. The effect of NF training on hallucinations' severity was assessed with the Auditory Vocal Hallucination Rating Scale (AVHRS) and on the affective state - with the Positive and Negative Affect Schedule (PANAS). All patients yielded significant upregulation of the ACC and reported subjective improvement in some aspects of AVHs (AVHRS) such as disturbance and suffering from the voices. In general, mood (PANAS) improved during NF training, though two patients reported worse mood after NF on the third day. ACC and reward system activity during NF learning and specific effects on mood and symptoms varied across the participants. None of them profited from the last training set in the prolonged three-session training. Moreover, individual differences emerged in brain networks activated with NF and in symptom changes, which were related to the patients' symptomatology and disease history. NF based on rt-fMRI seems a promising tool in therapy of AVHs. The patients, who suffered from continuous hallucinations for years, experienced symptom changes that may be attributed to the NF training. In order to assess the effectiveness of NF as a therapeutic method, this effect has to be studied systematically in larger groups; further, long-term effects need to be assessed. Particularly in schizophrenia, future NF studies should take into account the individual differences in reward processing, fatigue, and motivation to develop individualized training protocols

Translating Neurocognitive Models of Auditory-Verbal Hallucinations into Therapy: Using Real-time fMRI-Neurofeedback to Treat Voices

Auditory-verbal hallucinations (AVHs) are frequent and disabling symptoms, which can be refractory to conventional psychopharmacological treatment in more than 25% of the cases. Recent advances in brain imaging allow for a better understanding of the neural underpinnings of AVHs. These findings strengthened transdiagnostic neurocognitive models that characterize these frequent and disabling experiences. At the same time, technical improvements in real-time functional magnetic resonance imaging (fMRI) enabled the development of innovative and non-invasive methods with the potential to relieve psychiatric symptoms, such as fMRI-based neurofeedback (fMRI-NF). During fMRI-NF, brain activity is measured and fed back in real time to the participant in order to help subjects to progressively achieve voluntary control over their own neural activity. Precisely defining the target brain area/network(s) appears critical in fMRI-NF protocols. After reviewing the available neurocognitive models for AVHs, we elaborate on how recent findings in the field may help to develop strong a priori strategies for fMRI-NF target localization. The first approach relies on imaging-based “trait markers” (i.e., persistent traits or vulnerability markers that can also be detected in the presymptomatic and remitted phases of AVHs). The goal of such strategies is to target areas that show aberrant activations during AVHs or are known to be involved in compensatory activation (or resilience processes). Brain regions, from which the NF signal is derived, can be based on structural MRI and neurocognitive knowledge, or functional MRI information collected during specific cognitive tasks. Because hallucinations are acute and intrusive symptoms, a second strategy focuses more on “state markers.” In this case, the signal of interest relies on fMRI capture of the neural networks exhibiting increased activity during AVHs occurrences, by means of multivariate pattern recognition methods. The fine-grained activity patterns concomitant to hallucinations can then be fed back to the patients for therapeutic purpose. Considering the potential cost necessary to implement fMRI-NF, proof-of-concept studies are urgently required to define the optimal strategy for application in patients with AVHs. This technique has the potential to establish a new brain imaging-guided psychotherapy for patients that do not respond to conventional treatments and take functional neuroimaging to therapeutic applications

Spectrum Hole Prediction And White Space Ranking For Cognitive Radio Network Using An Artificial Neural Network

With spectrum becoming an ever scarcer resource, it is critical that new communication systems utilize all the available frequency bands as efficiently as possible in time, frequency and spatial domain. rHowever, spectrum allocation policies most of the licensed spectrums grossly underutilized while the unlicensed spectrums are overcrowded. Hence, all future wireless communication devices beequipped with cognitive capability to maximize quality of service (QoS); require a lot of time and energartificial intelligence and machine learning in cognitive radio deliver optimum performance. In this paper, we proposed a novel way of spectrum holes prediction using artificial neural network (ANN). The ANN was trained to adapt to the radio spectrum traffic of 20 channels and the trained network was used for prediction of future spectrum holes. The input of the neural network consist of a time domain vector of length six i.e. minute, hour, date, day, week and month. The output is a vector of length 20 each representing the probability of the channel being idle. The channels are ranked in order of decreasing probability of being idleminimizing We assumed that all the channels have the same noise and quality of service; and only one vacant channel is needed for communication. The result of the spectrum holes search using ANN was compared with that of blind linear and blind stochastic search and was found to be superior. The performance of the ANN that was trained to predict the probability of the channels being idle outperformed the ANN that will predict the exact channel states (busy or idle). In the ANN that was trained to predict the exact channels states, all channels predicted to be idle are randomly searched until the first spectrum hole was found; no information about search direction regarding which channel should be sensed first

Optimization for machine learning: Memory-efficient and tractible solutions to large-scale non-convex systems

Neural networks generally require large amounts of data to adequately model thedomain space. In situations where the data are limited, the predictions from these models, which are typically obtained from stochastic gradient descent (SGD) minimization algorithms, can be poor. In addition, the data is commonly corrupted due to poor imaging appatus. In these cases, the use of more sophisticated optimization approaches and model architectures becomes crucial to increase the impact of each training iteration. Second-order methods can capture curvature information, providing a more informed guess on the direction and step length. However, they require vast amounts of storage and can be computationally time demanding. To address the computational issue, we propose an optimization algorithm that uses second-derivative information, exploiting curvature information for avoiding saddle points. We utilize a Hessian-free approach where we do not explicitly store the second-derivative matrix, by applying a conjugate gradient method. The algorithm uses a trust-region method, which does not require the Hessian to be positive definite. We present numerical experiments which demonstrate the improvement in classification accuracy using our proposed approach over a standard SGD approach. We propose using a limited-memory symmetric rank-one quasi-Newton approach which further addresses the time and space computational complexity. The approach allows for indefinite Hessian approximations, enabling directions of negative curvature to be exploited. Furthermore, we use a modified adaptive regularized using cubics approach, which generates a sequence of cubic subproblems that have closed-form solutions with suitable regularization choices and investigate the performance of our proposed and compare our approach to state-of-the-art first-order and other quasi-Newton methods. To incorporate the benefits of an exponential moving average algorithm to a quasi-Newton approach, we propose a quasi-Adam approach. Judicious choices of quasi-Newton matrices can lead to guaranteed descent in the objective function and improved convergence. In this work, we integrate search directions obtained from using these quasi-Newton Hessian approximations with the Adam optimization algorithm. We provide convergence guarantees and demonstrate improved performance through an extensive experimentation on a variety of applications. Finally, to mitigate the issue of data corruption, we propose a variety of architectures for various applications in image processing. We propose a blind source signal separator, which involves separating image signals which have been superimposed by a common observing apparatus. We propose novel deep learning architectures for low photon count image denoising, which contains Gaussian noise in a low-photon count setting. Then we propose a novel architecture for lowphoton count and downsampled imaging, where the signal is interfered with some Gaussian noise, Poisson noise and then downsampled. Finally, we propose a novel adversarial detection method for white-box attacks using Radial basis function and Discrete Cosine Transforms