Unsupervised decoding of long-term, naturalistic human neural recordings with automated video and audio annotations

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Most ongoing efforts have focused on training decoders on specific, stereotyped tasks in laboratory settings. Implementing brain-computer interfaces (BCIs) in natural settings requires adaptive strategies and scalable algorithms that require minimal supervision. Here we propose an unsupervised approach to decoding neural states from human brain recordings acquired in a naturalistic context. We demonstrate our approach on continuous long-term electrocorticographic (ECoG) data recorded over many days from the brain surface of subjects in a hospital room, with simultaneous audio and video recordings. We first discovered clusters in high-dimensional ECoG recordings and then annotated coherent clusters using speech and movement labels extracted automatically from audio and video recordings. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Our results show that our unsupervised approach can discover distinct behaviors from ECoG data, including moving, speaking and resting. We verify the accuracy of our approach by comparing to manual annotations. Projecting the discovered cluster centers back onto the brain, this technique opens the door to automated functional brain mapping in natural settings

Dense attention network identifies EEG abnormalities during working memory performance of patients with schizophrenia

IntroductionPatients with schizophrenia typically exhibit deficits in working memory (WM) associated with abnormalities in brain activity. Alterations in the encoding, maintenance and retrieval phases of sequential WM tasks are well established. However, due to the heterogeneity of symptoms and complexity of its neurophysiological underpinnings, differential diagnosis remains a challenge. We conducted an electroencephalographic (EEG) study during a visual WM task in fifteen schizophrenia patients and fifteen healthy controls. We hypothesized that EEG abnormalities during the task could be identified, and patients successfully classified by an interpretable machine learning algorithm.MethodsWe tested a custom dense attention network (DAN) machine learning model to discriminate patients from control subjects and compared its performance with simpler and more commonly used machine learning models. Additionally, we analyzed behavioral performance, event-related EEG potentials, and time-frequency representations of the evoked responses to further characterize abnormalities in patients during WM.ResultsThe DAN model was significantly accurate in discriminating patients from healthy controls, ACC = 0.69, SD = 0.05. There were no significant differences between groups, conditions, or their interaction in behavioral performance or event-related potentials. However, patients showed significantly lower alpha suppression in the task preparation, memory encoding, maintenance, and retrieval phases F(1,28) = 5.93, p = 0.022, η2 = 0.149. Further analysis revealed that the two highest peaks in the attention value vector of the DAN model overlapped in time with the preparation and memory retrieval phases, as well as with two of the four significant time-frequency ROIs.DiscussionThese results highlight the potential utility of interpretable machine learning algorithms as an aid in diagnosis of schizophrenia and other psychiatric disorders presenting oscillatory abnormalities

Stress impairs intentional memory control through altered theta oscillations in lateral parietal cortex

Accumulating evidence suggests that forgetting is not necessarily a passive process but that we can, to some extent, actively control what we remember and what we forget. Although this intentional control of memory has potentially far-reaching implications, the factors that influence our capacity to intentionally control our memory are largely unknown. Here, we tested whether acute stress may disrupt the intentional control of memory and, if so, through which neural mechanism. We exposed healthy men and women to a stress (n=27) or control (n=26) procedure before they aimed repeatedly to retrieve some previously learned cue-target pairs and to actively suppress others. While control participants showed reduced memory for supressed compared to baseline pairs in a subsequent memory test, this suppression-induced forgetting was completely abolished after stress. Using magnetoencephalography (MEG), we show that the reduced ability to suppress memories after stress is associated with altered theta activity in the inferior temporal cortex when the control process (retrieval or suppression) is triggered and in the lateral parietal cortex when control is exerted, with the latter being directly correlated with the stress hormone cortisol. Moreover, the suppression-induced forgetting was linked to altered connectivity between the hippocampus and right dorsolateral prefrontal cortex, which in turn was negatively correlated to stress-induced cortisol increases. These findings provide novel insights into conditions under which our capacity to actively control our memory breaks down and may have considerable implications for stress-related psychopathologies, such as posttraumatic stress disorder, that are characterized by unwanted memories of distressing events.Significance Statement: It is typically assumed that forgetting is a passive process that can hardly be controlled. There is, however, evidence that we may actively control, to some extent, what we remember and what we forget. This intentional memory control has considerable implications for mental disorders in which patients suffer from unwanted (e.g., traumatic) memories. Here, we demonstrate that the capacity to intentionally control our memory breaks down after stress. Using magnetoencephalography, we show that this stress-induced memory control deficit is linked to altered activity in the lateral parietal cortex and the connectivity between the hippocampus and right prefrontal cortex. These findings provide novel insights into conditions under which memory control fails and are highly relevant in the context of stress-related psychopathologies

Classication of semantic memories using multitaper spectral estimation

The research on classication of semantic memories is still very young. Several methods have been tested ranging from magnetic resonance imaging (MRI) to electrocorticog- raphy (ECoG). This report describes an alternative way of classifying signals collected from an electroencephalogram (EEG) into categories using the Thomson multitaper method of spectral estimation, as well as a logistic regression model. The aim for this report is to expand the research eld with an approach that complements the current options of classication. Data was distributed from the department of Psychology at Lund University, and the experimental paradigm was to classify three types of semantic memories (faces, landmarks and objects) based on their neural patterns. Based on the cross-validation from the mentioned methods, a classier could successfully be trained for the "faces" and "landmarks" categories with an average success rate of 55% and 51% respectively. The classier accurately responded to the onset of the stimuli (p < 0:001 for faces, p = 0:015 for landmarks). No classier for the "objects" category could be trained using this method. These results indicate that the multitaper method of spec- tral estimation can be useful in detecting neural patterns. Several ways to rene these methods are discussed

Linking quantitative radiology to molecular mechanism for improved vascular disease therapy selection and follow-up

Objective: Therapeutic advancements in atherosclerotic cardiovascular disease have improved the prevention of ischemic stroke and myocardial infarction. However, diagnostic methods for atherosclerotic plaque phenotyping to aid individualized therapy are lacking. In this thesis, we aimed to elucidate plaque biology through the analysis of computed-tomography angiography (CTA) with sufficient sensitivity and specificity to capture the differentiated drivers of the disease. We then aimed to use such data to calibrate a systems biology model of atherosclerosis with adequate granularity to be clinically relevant. Such development may be possible with computational modeling, but given, the multifactorial biology of atherosclerosis, modeling must be based on complete biological networks that capture protein-protein interactions estimated to drive disease progression. Approach and Results: We employed machine intelligence using CTA paired with a molecular assay to determine cohort-level associations and individual patient predictions. Examples of predicted transcripts included ion transporters, cytokine receptors, and a number of microRNAs. Pathway analyses elucidated enrichment of several biological processes relevant to atherosclerosis and plaque pathophysiology. The ability of the models to predict plaque gene expression from CTAs was demonstrated using sequestered patients with transcriptomes of corresponding lesions. We further performed a case study exploring the relationship between biomechanical quantities and plaque morphology, indicating the ability to determine stress and strain from tissue characteristics. Further, we used a uniquely constituted plaque proteomic dataset to create a comprehensive systems biology disease model, which was finally used to simulate responses to different drug categories in individual patients. Individual patient response was simulated for intensive lipid-lowering, anti-inflammatory drugs, anti-diabetic, and combination therapy. Plaque tissue was collected from 18 patients with 6735 proteins at two locations per patient. 113 pathways were identified and included in the systems biology model of endothelial cells, vascular smooth muscle cells, macrophages, lymphocytes, and the integrated intima, altogether spanning 4411 proteins, demonstrating a range of 39-96% plaque instability. Simulations of drug responses varied in patients with initially unstable lesions from high (20%, on combination therapy) to marginal improvement, whereas patients with initially stable plaques showed generally less improvement, but importantly, variation across patients. Conclusion: The results of this thesis show that atherosclerotic plaque phenotyping by multi-scale image analysis of conventional CTA can elucidate the molecular signatures that reflect atherosclerosis. We further showed that calibrated system biology models may be used to simulate drug response in terms of atherosclerotic plaque instability at the individual level, providing a potential strategy for improved personalized management of patients with cardiovascular disease. These results hold promise for optimized and personalized therapy in the prevention of myocardial infarction and ischemic stroke, which warrants further investigations in larger cohorts

Rhythmic fluctuations in mnemonic signatures during associative recall

The intricate linking of information processing and neural representations to the underlying hippocampal neural rhythms during episodic memory retrieval are yet to be fully explored in human subjects. In this doctoral thesis, the temporal order of these relationships was investigated, with emphasis on how the processes evolve and take place over time. Empirical evidence and neural network models suggest that hippocampus and the hippocampal theta rhythm play a central role in episodic memory. In the first two studies, different oscillatory dynamics in the hippocampal circuit thought to provide optimal states for encoding and retrieval were investigated. The third study investigated the role of the hippocampal theta oscillation as an adaptive mechanism in regulating competition between similar memories. And lastly, the fourth study investigated sharp-wave ripples in promoting successful episodic memory retrieval. Throughout the four chapters, memory content was decoded using multivariate pattern classification, and the timing of memory reactivation was linked to two prominent oscillatory brain signatures: the hippocampal theta rhythm on the one hand, and hippocampal sharp-wave ripples on the other. In sum, this doctoral thesis provides support for the powerful computations along the hippocampal theta oscillation, and the close interplay between hippocampus and neocortical areas, foremost at time of retrieva

Neural avkodning genom multivariat mönsteranalys av elektroencefalografisk data

Multivariate pattern analysis (MVPA) is a machine learning method that has, over the past 15 years, been used in brain imaging studies to decode and to classify neural representations that emerges when participants cognitively process the exposure of various stimuli. The purpose of this study was, with the use of classifiers, to decode the electroencephalographic brain activity that occurred when subjects observed images from three different categories: famous faces, everyday objects, and famous landmarks. To determine the decoding performance the accuracy of each classifier was calculated when predicting which category the activation patterns of the 23 subjects belonged to. The results revealed that the accuracy peaked at 64.76% for the representations of the facial images, and at 48.21% for the images composed of landmarks. The classification of the object category showed no significant difference in performance when compared against chance. The average classification accuracy for the three image categories amounted to 49.25%.Multivariat mönsteranalys (MVPA) är en maskininlärningsmetod som de senaste 15 åren använts inom hjärnavbildningsstudier för att avkoda och klassificera neurala representationer som uppstår vid kognitiv bearbetning hos försöksdeltagare då de utsätts för olika slags stimuli. Syftet med denna undersökning har varit att med hjälp av klassificerare avkoda den elektroencefalografiska hjärnaktivitet som uppstod då försöksdeltagare iakttog bilder från tre olika kategorier: välkända ansikten, vardagsobjekt och kända landmärken. För att avgöra hur väl avkodningen lyckats har klassificerarnas procentuella träffsäkerhet beräknats vid predicerandet av vilken bildkategori som aktiveringsmönstren från de 23 försöksdeltagarna tillhörde. Resultatet visade att träffsäkerheten uppgick till 64.76% för representationerna av ansiktsbilderna, och till 48.21% för bilderna på landmärken. Klassificeringen av objektkategorin visade ingen signifikant prestationsskillnad i jämförelse med ett slumpmässigt resultat. Den genomsnittliga träffsäkerheten för bildkategoriernas samtliga klassificerare uppgick till 49.25%

Earth Resources: A continuing bibliography with indexes

This bibliography lists 623 reports, articles, and other documents introduced into the NASA scientific and technical information system between April 1 and June 30, 1983. Emphasis is placed on the use of remote sensing and geophysical instrumentation in spacecraft and aircraft to survey and inventory natural resources and urban areas. Subject matter is grouped according to agriculture and forestry, environmental changes and cultural resources, geodesy and cartography, geology and mineral resources, hydrology and water management, data processing and distribution systems, instrumentation and sensors, and economic analysis