Point process modeling and estimation: advances in the analysis of dynamic neural spiking data

A common interest of scientists in many fields is to understand the relationship between the dynamics of a physical system and the occurrences of discrete events within such physical system. Seismologists study the connection between mechanical vibrations of the Earth and the occurrences of earthquakes so that future earthquakes can be better predicted. Astrophysicists study the association between the oscillating energy of celestial regions and the emission of photons to learn the Universe's various objects and their interactions. Neuroscientists study the link between behavior and the millisecond-timescale spike patterns of neurons to understand higher brain functions. Such relationships can often be formulated within the framework of state-space models with point process observations. The basic idea is that the dynamics of the physical systems are driven by the dynamics of some stochastic state variables and the discrete events we observe in an interval are noisy observations with distributions determined by the state variables. This thesis proposes several new methodological developments that advance the framework of state-space models with point process observations at the intersection of statistics and neuroscience. In particular, we develop new methods 1) to characterize the rhythmic spiking activity using history-dependent structure, 2) to model population spike activity using marked point process models, 3) to allow for real-time decision making, and 4) to take into account the need for dimensionality reduction for high-dimensional state and observation processes. We applied these methods to a novel problem of tracking rhythmic dynamics in the spiking of neurons in the subthalamic nucleus of Parkinson's patients with the goal of optimizing placement of deep brain stimulation electrodes. We developed a decoding algorithm that can make decision in real-time (for example, to stimulate the neurons or not) based on various sources of information present in population spiking data. Lastly, we proposed a general three-step paradigm that allows us to relate behavioral outcomes of various tasks to simultaneously recorded neural activity across multiple brain areas, which is a step towards closed-loop therapies for psychological diseases using real-time neural stimulation. These methods are suitable for real-time implementation for content-based feedback experiments

Multimodal learning from visual and remotely sensed data

Autonomous vehicles are often deployed to perform exploration and monitoring missions in unseen environments. In such applications, there is often a compromise between the information richness and the acquisition cost of different sensor modalities. Visual data is usually very information-rich, but requires in-situ acquisition with the robot. In contrast, remotely sensed data has a larger range and footprint, and may be available prior to a mission. In order to effectively and efficiently explore and monitor the environment, it is critical to make use of all of the sensory information available to the robot. One important application is the use of an Autonomous Underwater Vehicle (AUV) to survey the ocean floor. AUVs can take high resolution in-situ photographs of the sea floor, which can be used to classify different regions into various habitat classes that summarise the observed physical and biological properties. This is known as benthic habitat mapping. However, since AUVs can only image a tiny fraction of the ocean floor, habitat mapping is usually performed with remotely sensed bathymetry (ocean depth) data, obtained from shipborne multibeam sonar. With the recent surge in unsupervised feature learning and deep learning techniques, a number of previous techniques have investigated the concept of multimodal learning: capturing the relationship between different sensor modalities in order to perform classification and other inference tasks. This thesis proposes related techniques for visual and remotely sensed data, applied to the task of autonomous exploration and monitoring with an AUV. Doing so enables more accurate classification of the benthic environment, and also assists autonomous survey planning. The first contribution of this thesis is to apply unsupervised feature learning techniques to marine data. The proposed techniques are used to extract features from image and bathymetric data separately, and the performance is compared to that with more traditionally used features for each sensor modality. The second contribution is the development of a multimodal learning architecture that captures the relationship between the two modalities. The model is robust to missing modalities, which means it can extract better features for large-scale benthic habitat mapping, where only bathymetry is available. The model is used to perform classification with various combinations of modalities, demonstrating that multimodal learning provides a large performance improvement over the baseline case. The third contribution is an extension of the standard learning architecture using a gated feature learning model, which enables the model to better capture the ‘one-to-many’ relationship between visual and bathymetric data. This opens up further inference capabilities, with the ability to predict visual features from bathymetric data, which allows image-based queries. Such queries are useful for AUV survey planning, especially when supervised labels are unavailable. The final contribution is the novel derivation of a number of information-theoretic measures to aid survey planning. The proposed measures predict the utility of unobserved areas, in terms of the amount of expected additional visual information. As such, they are able to produce utility maps over a large region that can be used by the AUV to determine the most informative locations from a set of candidate missions. The models proposed in this thesis are validated through extensive experiments on real marine data. Furthermore, the introduced techniques have applications in various other areas within robotics. As such, this thesis concludes with a discussion on the broader implications of these contributions, and the future research directions that arise as a result of this work

Abstracts of the 2014 Brains, Minds, and Machines Summer School

A compilation of abstracts from the student projects of the 2014 Brains, Minds, and Machines Summer School, held at Woods Hole Marine Biological Lab, May 29 - June 12, 2014.This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216

Gamma Band Oscillation Response to Somatosensory Feedback Stimulation Schemes Constructed on Basis of Biphasic Neural Touch Representation

abstract: Prosthetic users abandon devices due to difficulties performing tasks without proper graded or interpretable feedback. The inability to adequately detect and correct error of the device leads to failure and frustration. In advanced prostheses, peripheral nerve stimulation can be used to deliver sensations, but standard schemes used in sensorized prosthetic systems induce percepts inconsistent with natural sensations, providing limited benefit. Recent uses of time varying stimulation strategies appear to produce more practical sensations, but without a clear path to pursue improvements. This dissertation examines the use of physiologically based stimulation strategies to elicit sensations that are more readily interpretable. A psychophysical experiment designed to investigate sensitivities to the discrimination of perturbation direction within precision grip suggests that perception is biomechanically referenced: increased sensitivities along the ulnar-radial axis align with potential anisotropic deformation of the finger pad, indicating somatosensation uses internal information rather than environmental. Contact-site and direction dependent deformation of the finger pad activates complimentary fast adapting and slow adapting mechanoreceptors, exhibiting parallel activity of the two associate temporal patterns: static and dynamic. The spectrum of temporal activity seen in somatosensory cortex can be explained by a combined representation of these distinct response dynamics, a phenomenon referred in this dissertation to “biphasic representation.” In a reach-to-precision-grasp task, neurons in somatosensory cortex were found to possess biphasic firing patterns in their responses to texture, orientation, and movement. Sensitivities seem to align with variable deformation and mechanoreceptor activity: movement and smooth texture responses align with potential fast adapting activation, non-movement and coarse texture responses align with potential increased slow adapting activation, and responses to orientation are conceptually consistent with coding of tangential load. Using evidence of biphasic representations’ association with perceptual priorities, gamma band phase locking is used to compare responses to peripheral nerve stimulation patterns and mechanical stimulation. Vibrotactile and punctate mechanical stimuli are used to represent the practical and impractical percepts commonly observed in peripheral nerve stimulation feedback. Standard patterns of constant parameters closely mimic impractical vibrotactile stimulation while biphasic patterns better mimic punctate stimulation and provide a platform to investigate intragrip dynamics representing contextual activation.Dissertation/ThesisDoctoral Dissertation Biomedical Engineering 201

Neuronal correlates of tactile working memory in rat barrel cortex and prefrontal cortex

The neuronal mechanisms of parametric working memory \u2013 the short-term storage of graded stimuli to guide behavior \u2013 are not fully elucidated. We have designed a working memory task where rats compare two sequential vibrations, S1 and S2, delivered to their whiskers (Fassihi et al, 2014). Vibrations are a series of velocities sampled from a zero-mean normal distribution. Rats must judge which stimulus had greater velocity standard deviation, \u3c3 (e.g. \u3c31 > \u3c32 turn left, \u3c31 < \u3c32 turn right). A critical operation in this task is to hold S1 information in working memory for subsequent comparison. In an earlier work we uncovered this cognitive capacity in rats (Fassihi et al, 2014), an ability previously ascribed only to primates. Where in the brain is such a memory kept and what is the nature of its representation? To address these questions, we performed simultaneous multi-electrode recordings from barrel cortex \u2013 the entryway of whisker sensory information into neocortex \u2013 and prelimbic area of medial prefrontal cortex (mPFC) which is involved in higher order cognitive functioning in rodents. During the presentation of S1 and S2, a majority of neurons in barrel cortex encoded the ongoing stimulus by monotonically modulating their firing rate as a function of \u3c3; i.e. 42% increased and 11% decreased their firing rate for progressively larger \u3c3 values. During the 2 second delay interval between the two stimuli, neuronal populations in barrel cortex kept a graded representation of S1 in their firing rate; 30% at early delay and 15% at the end. In mPFC, neurons expressed divers coding characteristics yet more than one-fourth of them varied their discharge rate according to the ongoing stimulus. Interestingly, a similar proportion carried the stimulus signal up to early parts of delay period. A smaller but considerable proportion (10%) kept the memory until the end of delay interval. We implemented novel information theoretic measures to quantify the stimulus and decision signals in neuronal responses in different stages of the task. By these measures, a decision signal was present in barrel cortex neurons during the S2 period and during the post stimulus delay, when the animal needed to postpone its action. Medial PFC units also represented animal choice, but later in the trial in comparison to barrel cortex. Decision signals started to build up in this area after the termination of S2. We implemented a regularized linear discriminant algorithm (RDA) to decode stimulus and decision signals in the population activity of barrel cortex and mPFC neurons. The RDA outperformed individual clusters and the standard linear discriminant analysis (LDA). The stimulus and animal\u2019s decision could be extracted from population activity simply by linearly weighting the responses of neuronal clusters. The population signal was present even in epochs of trial where no single cluster was informative. We predicted that coherent oscillations between brain areas might optimize the flow of information within the networks engaged by this task. Therefore, we quantified the phase synchronization of local field potentials in barrel cortex and mPFC. The two signals were coherent at theta range during S1 and S2 and, interestingly, prior to S1. We interpret the pre-stimulus coherence as reflecting top-down preparatory and expectation mechanisms. We showed, for the first time to our knowledge, the neuronal correlates of parametric working memory in rodents. The existence of both positive and negative codes in barrel cortex, besides the representation of stimulus memory and decision signals suggests that multiple functions might be folded into single modules. The mPFC also appears to be part of parametric working memory and decision making network in rats

Neural Correlates of Sensorimotor Control in Human Cortex: State Estimates and Reference Frames

Interacting with our environment involves multiple sensory-motor circuits throughout the human brain. How do these circuits transform sensory inputs into discernable motor actions? Our understanding of this question is critical to behavioral neuroscience and implementation of brain-machine interfaces (BMIs). In this thesis, we present experiments that explore the contributions of human cerebral cortex (parietal, premotor, and primary somatosensory cortices) to sensory-motor transformations. First, we provide evidence in support of primary somatosensory cortex (S1) encoding cognitive motor signals. Next, we describe a series of experiments that explore contributions of posterior parietal cortex (PPC) to the internal state estimate. Neural correlates for the state estimate are found in PPC; furthermore, it is found to be encoded with respect to gaze position. Finally, we investigate reference frame encoding in regions throughout human cortex (AIP, SMG, PMv, and S1) during an imagined reaching task. We find the greatest heterogeneity among brain regions during movement planning, which collapses to a largely single reference frame representation (hand-centered) during execution of the imagined reach. However, this result is dependent upon brain region. These findings yield new perspectives and evidence on the organization of sensory-motor transformations and the location the human brain’s internal estimate of the body’s state.</p

Redundancy reduction in motor control

Research in machine learning and neuroscience has made remarkable progress by investigating statistical redundancy in representations of natural environments, but to date much of this work has focused on sensory information like images and sounds. This dissertation explores the notions of redundancy and efficiency in the motor domain, where several different forms of independence exist. The dissertation begins by discussing redundancy at a conceptual level and presents relevant background material. Next, three main branches of original research are described. The first branch consists of a novel control framework for integrating low-bandwidth sensory updates with model uncertainty and action selection for navigating complex, multi-task environments. The second branch of research applies existing machine learning techniques to movement information and explores the mismatch between these methods for extracting independent components and the forms of redundancy that exist in the motor domain. The third branch of work analyzes full-body, goal-directed reaching movements gathered in a novel laboratory experiment, using explicitly measured information about the goal of each movement to uncover patterns in the movement dynamics. Each branch of research explores redundancy reduction in movement from a different perspective, building up a sort of catalog of the types of information present in movements. Redundancy is discussed throughout as an an important aspect of movement in the natural world. The dissertation concludes by summarizing the contributions of these three branches of work, and discussing promising areas for future work spurred by these investigations. More detailed models of voluntary movements hold promise not only for better treatments, improved prosthetics, smoother animations, and more fluid robots, but also as an avenue for scientific insight into the very foundations of cognition.Computer Science

Understanding Video Transformers for Segmentation: A Survey of Application and Interpretability

Video segmentation encompasses a wide range of categories of problem formulation, e.g., object, scene, actor-action and multimodal video segmentation, for delineating task-specific scene components with pixel-level masks. Recently, approaches in this research area shifted from concentrating on ConvNet-based to transformer-based models. In addition, various interpretability approaches have appeared for transformer models and video temporal dynamics, motivated by the growing interest in basic scientific understanding, model diagnostics and societal implications of real-world deployment. Previous surveys mainly focused on ConvNet models on a subset of video segmentation tasks or transformers for classification tasks. Moreover, component-wise discussion of transformer-based video segmentation models has not yet received due focus. In addition, previous reviews of interpretability methods focused on transformers for classification, while analysis of video temporal dynamics modelling capabilities of video models received less attention. In this survey, we address the above with a thorough discussion of various categories of video segmentation, a component-wise discussion of the state-of-the-art transformer-based models, and a review of related interpretability methods. We first present an introduction to the different video segmentation task categories, their objectives, specific challenges and benchmark datasets. Next, we provide a component-wise review of recent transformer-based models and document the state of the art on different video segmentation tasks. Subsequently, we discuss post-hoc and ante-hoc interpretability methods for transformer models and interpretability methods for understanding the role of the temporal dimension in video models. Finally, we conclude our discussion with future research directions

VIDEO FOREGROUND LOCALIZATION FROM TRADITIONAL METHODS TO DEEP LEARNING

These days, detection of Visual Attention Regions (VAR), such as moving objects has become an integral part of many Computer Vision applications, viz. pattern recognition, object detection and classification, video surveillance, autonomous driving, human-machine interaction (HMI), and so forth. The moving object identification using bounding boxes has matured to the level of localizing the objects along their rigid borders and the process is called foreground localization (FGL). Over the decades, many image segmentation methodologies have been well studied, devised, and extended to suit the video FGL. Despite that, still, the problem of video foreground (FG) segmentation remains an intriguing task yet appealing due to its ill-posed nature and myriad of applications. Maintaining spatial and temporal coherence, particularly at object boundaries, persists challenging, and computationally burdensome. It even gets harder when the background possesses dynamic nature, like swaying tree branches or shimmering water body, and illumination variations, shadows cast by the moving objects, or when the video sequences have jittery frames caused by vibrating or unstable camera mounts on a surveillance post or moving robot. At the same time, in the analysis of traffic flow or human activity, the performance of an intelligent system substantially depends on its robustness of localizing the VAR, i.e., the FG. To this end, the natural question arises as what is the best way to deal with these challenges? Thus, the goal of this thesis is to investigate plausible real-time performant implementations from traditional approaches to modern-day deep learning (DL) models for FGL that can be applicable to many video content-aware applications (VCAA). It focuses mainly on improving existing methodologies through harnessing multimodal spatial and temporal cues for a delineated FGL. The first part of the dissertation is dedicated for enhancing conventional sample-based and Gaussian mixture model (GMM)-based video FGL using probability mass function (PMF), temporal median filtering, and fusing CIEDE2000 color similarity, color distortion, and illumination measures, and picking an appropriate adaptive threshold to extract the FG pixels. The subjective and objective evaluations are done to show the improvements over a number of similar conventional methods. The second part of the thesis focuses on exploiting and improving deep convolutional neural networks (DCNN) for the problem as mentioned earlier. Consequently, three models akin to encoder-decoder (EnDec) network are implemented with various innovative strategies to improve the quality of the FG segmentation. The strategies are not limited to double encoding - slow decoding feature learning, multi-view receptive field feature fusion, and incorporating spatiotemporal cues through long-shortterm memory (LSTM) units both in the subsampling and upsampling subnetworks. Experimental studies are carried out thoroughly on all conditions from baselines to challenging video sequences to prove the effectiveness of the proposed DCNNs. The analysis demonstrates that the architectural efficiency over other methods while quantitative and qualitative experiments show the competitive performance of the proposed models compared to the state-of-the-art