What Makes a Pattern? Matching Decoding Methods to Data in Multivariate Pattern Analysis

Research in neuroscience faces the challenge of integrating information across different spatial scales of brain function. A promising technique for harnessing information at a range of spatial scales is multivariate pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) data. While the prevalence of MVPA has increased dramatically in recent years, its typical implementations for classification of mental states utilize only a subset of the information encoded in local fMRI signals. We review published studies employing multivariate pattern classification since the technique’s introduction, which reveal an extensive focus on the improved detection power that linear classifiers provide over traditional analysis techniques. We demonstrate using simulations and a searchlight approach, however, that non-linear classifiers are capable of extracting distinct information about interactions within a local region. We conclude that for spatially localized analyses, such as searchlight and region of interest, multiple classification approaches should be compared in order to match fMRI analyses to the properties of local circuits

Hemodynamic signals of mixed messages during a social exchange

The present study used functional magnetic resonance imaging (fMRI) to characterize hemodynamic activation patterns recruited when participants view mixed social communicative messages during a common interpersonal exchange. Mixed messages were defined as conflicting sequences of biological motion and facial affect signals that are unexpected within a particular social context (for example, observing the reception of a gift). Across four social vignettes, valenced facial expressions were crossed with rejecting and accepting gestures in a virtual avatar responding to presentation of a gift from the participant. Results indicate that conflicting facial affect and gesture activated superior temporal sulcus, a region implicated in expectancy violations, as well as inferior frontal gyrus and putamen. Scenarios conveying rejection differentially activated the insula and putamen, regions implicated in embodied cognition and motivated learning, as well as frontoparietal cortex. Characterizing how meaning is inferred from integration of conflicting nonverbal communicative cues is essential to understand nuances and complexities of human exchange

Dorsal premammillary projection to periaqueductal gray controls escape vigor from innate and conditioned threats

Escape from threats has paramount importance for survival. However, it is unknown if a single circuit controls escape vigor from innate and conditioned threats. Cholecystokinin (cck)-expressing cells in the hypothalamic dorsal premammillary nucleus (PMd) are necessary for initiating escape from innate threats via a projection to the dorsolateral periaqueductal gray (dlPAG). We now show that in mice PMd-cck cells are activated during escape, but not other defensive behaviors. PMd-cck ensemble activity can also predict future escape. Furthermore, PMd inhibition decreases escape speed from both innate and conditioned threats. Inhibition of the PMd-cck projection to the dlPAG also decreased escape speed. Intriguingly, PMd-cck and dlPAG activity in mice showed higher mutual information during exposure to innate and conditioned threats. In parallel, human functional magnetic resonance imaging data show that a posterior hypothalamic-to-dlPAG pathway increased activity during exposure to aversive images, indicating that a similar pathway may possibly have a related role in humans. Our data identify the PMd-dlPAG circuit as a central node, controlling escape vigor elicited by both innate and conditioned threats. &nbsp;</p

Dorsal premammillary projection to periaqueductal gray controls escape vigor from innate and conditioned threats

Escape from threats has paramount importance for survival. However, it is unknown if a single circuit controls escape vigor from innate and conditioned threats. Cholecystokinin (cck)-expressing cells in the hypothalamic dorsal premammillary nucleus (PMd) are necessary for initiating escape from innate threats via a projection to the dorsolateral periaqueductal gray (dlPAG). We now show that in mice PMd-cck cells are activated during escape, but not other defensive behaviors. PMd-cck ensemble activity can also predict future escape. Furthermore, PMd inhibition decreases escape speed from both innate and conditioned threats. Inhibition of the PMd-cck projection to the dlPAG also decreased escape speed. Intriguingly, PMd-cck and dlPAG activity in mice showed higher mutual information during exposure to innate and conditioned threats. In parallel, human functional magnetic resonance imaging data show that a posterior hypothalamic-to-dlPAG pathway increased activity during exposure to aversive images, indicating that a similar pathway may possibly have a related role in humans. Our data identify the PMd-dlPAG circuit as a central node, controlling escape vigor elicited by both innate and conditioned threats

Multivariate pattern classification reveals autonomic and experiential representations of discrete emotions. Emotion. Epub ahead of print. doi

Defining the structural organization of emotions is a central unresolved question in affective science. In particular, the extent to which autonomic nervous system activity signifies distinct affective states remains controversial. Most prior research on this topic has used univariate statistical approaches in attempts to classify emotions from psychophysiological data. In the present study, electrodermal, cardiac, respiratory, and gastric activity, as well as self-report measures were taken from healthy subjects during the experience of fear, anger, sadness, surprise, contentment, and amusement in response to film and music clips. Information pertaining to affective states present in these response patterns was analyzed using multivariate pattern classification techniques. Overall accuracy for classifying distinct affective states was 58.0% for autonomic measures and 88.2% for self-report measures, both of which were significantly above chance. Further, examining the error distribution of classifiers revealed that the dimensions of valence and arousal selectively contributed to decoding emotional states from self-report, whereas a categorical configuration of affective space was evident in both self-report and autonomic measures. Taken together, these findings extend recent multivariate approaches to study emotion and indicate that pattern classification tools may improve upon univariate approaches to reveal the underlying structure of emotional experience and physiological expression. Keywords: emotion theory, autonomic nervous system, multivariate pattern classification, psychophysiology, affective state A core theoretical question in emotion research concerns understanding how emotions are organized and represented in behavior and bodily responses. Two predominant views on the structure of emotions include dimensional and categorical models. Dimensional models organize emotions based on their coordinates in an affective space commonly anchored using valence and arousal (or related constructs) as dimensional axes Categorical and dimensional theories predict different relationships between emotions and the structure of affective space. For categorical theories, each emotion is hypothesized to be governed by a distinct biological mechanism and specific physiological state. This response configuration should produce a sparse structure in which emotions are distant from each other. Dimensional accounts of emotion suggest responses should be interrelated along either valence or arousal dimensions (although the general consensus is that both these dimensions are required; for a review of models see A key criticism of the available evidence to date is that most empirical studies have used univariate statistical approaches to address this theoretical question. Univariate approaches test the relationship between a single dependent measure and experimental manipulations with one or more independent variables. In contrast, multivariate statistical approaches are able to jointly consider multiple dependent variables and experimental manipulations. Given that emotions engage a complex set of physiological components that likely make unique contributions to different affective states, univariate approaches are not optimal for identifying such interactions that are critical in characterizing emotions. Even when multiple dependent variables are measured and used to construct an analysis of variance (ANOVA; e.g., if a dependent variable is used to construct an additional factor to test an interaction with an independent variable), they are assumed to be independent. In multivariate approaches, the correlation between both independent and dependent variables is used in the analysis. By jointly considering multiple variables, multivariate approaches can reveal organization in data that is lost when response variables are treated independently or examined one at a time. A series of investigations using multivariate pattern classification techniques shows promise for testing the autonomic specificity of distinct emotions. Similar to conventional univariate approaches, these studies have elicited distinct affective states using a range of induction methods such as film The goal of the present study was to investigate the structure of emotion representations in subjective experience and physiological expression, by using pattern classification methods to decode emotional state. The term decode refers to the use of multivariate pattern classifiers to assign a class label to a set of dependent measures. Within the field of cognitive neuroscience, this approach has been widely used to infer the mental state of a participant from patterns of neural activity, termed mind reading or decoding To test the organization of emotion evidenced in self-report and peripheral autonomic expression, we compared the distribution of observed classification errors to those predicted by categorical versus dimensional models of emotion. This approach parallels the well-established use of confusion data in psychophysics studies of perceptual categorization and recognition, where participants label stimuli and the distribution of errors is used to characterize the mental representation of stimuli (e.g., Method Participants Twenty healthy volunteers (10 women, 10 men, 15 White, three Black, two Asian, M age ϭ 23.5 years, age range: 19 -36 years) gave written informed consent and participated in the study. The study was approved in accordance with the Institutional Review This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. KRAGEL AND LABAR Board at Duke University. Participants were compensated either $10 per hour or with course credit. Materials and Procedure Standardized music and film clips were presented to elicit the discrete emotions of fear, anger, sadness, surprise, contentment, amusement, and a neutral control condition. Two emotion induction techniques were used to ensure the patterning of responses was not specific to the method of elicitation. The stimuli selected were the same as those used and validated in Psychophysiological Recording and Feature Extraction Psychophysiological data were acquired using a BIOPAC MP-150 data acquisition system and subsequently processed using AcqKnowledge software (BIOPAC Systems Inc., Goleta, CA) and custom in-house scripts (MATLAB 2010a, The MathWorks Inc., Natick, MA, 2010). Physiological activity was recorded as an analog signal and digitized at a frequency of 200 Hz. Finite sampling of any continuous measure can introduce error in its reconstruction, and selecting a sufficiently high sampling rate is critical. Although the selected sampling rate may contribute as a potential source of error in the computation of measures requiring temporal precision such as heart-rate variability Digitized physiological data were processed on a trial-by-trial basis for each instance of emotion induction to extract features relevant to autonomic patterning. Electrodermal data were processed using custom MATLAB scripts to extract the tonic skin conductance level (SCL) and the skin conductance response rate (SRR). SCL was obtained by calculating the mean level of skin conductance. SRR was derived by counting the number of individual skin conductance responses during the emotion induction, indicated by continuous increases in electrodermal activity with amplitude greater than .02 S. ECG data were processed using heart rate variability software (Acharya, Joseph, These 16 autonomic measures were adjusted to reflect relative change from the washout preceding the trial by subtracting the mean value of the 60 s prior to the end of the emotion induction from the last 60 s of the preceding washout. This averaging served as a temporal filter, reducing the impact of artifacts and ensuring a stable measure of peripheral responses. As the stimuli had variable duration and on average lasted 124 s, averaging during this window ensures the emotions have had sufficient time to emerge and offers an even sampling for all stimuli. Beyond this This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. MULTIVARIATE PATTERNING OF EMOTION windowed averaging, no data were explicitly removed due to the presence of artifacts, as the preprocessing and analytical approach used should minimize their impact. Prior to pattern classification analysis, autonomic measures were converted to a standardized z-score within subjects to remove variability related to overall levels of autonomic responsivity between subjects. Statistical Analysis The effectiveness of the emotion induction was assessed by performing an ANOVA using categorical items from the selfreport scales. Post hoc contrasts were constructed to compare the targeted emotional response against all alternatives. This manipulation check was used to ensure the experience of emotion occurred as intended. The amount of emotion-related information present in autonomic responding and self-report was investigated using pattern classification analysis. In this type of analysis, a classifier is constructed that determines the categorical grouping of response variables, called features, by creating a decision rule. We constructed classifiers using either autonomic responses or self-report as input in an attempt to decode the categorical emotion label for each trial. Ground truth for classification was determined a priori based on previous work validating the stimuli We selected a support vector machine algorithm, implemented in libSVM To evaluate our first hypothesis that distinct affective states are represented in patterns of peripheral autonomic activity and selfreport, we conducted the following statistical analyses. We calculated the accuracy of the classifier for each repetition by comparing the categories predicted by the classifier to the true categories in the testing set. Nonparametric statistical tests were performed to ensure robustness against violations of normal distribution assumptions. Chance performance was estimated by permuting emotion labels and running the random subsampling procedure on the same data. Statistical significance of performance was determined by entering the multiple repeated estimates of classifier accuracy using true labels against those with the random labels with a Wilcoxon signed-ranks test. This approach has been suggested to overcome violations of the assumptions of parametric tests created when making inference on cross validated data To investigate the degree to which response patterning supports different theoretical organizations of emotion, we examined the distribution of errors produced by pattern classifiers. Using the true and predicted labels from classification, we constructed a confusion matrix to characterize the structure of performance on each repetition. The confusion matrix was then used to tally the number of errors made for the 21 possible pairwise combinations of emotions that could constitute an error (e.g., mistaking fear and anger). The distribution of errors on each repetition was compared to the difference in subjective experience along discrete categories and dimensions of arousal and valence. Categorical items directly corresponded to the seven self-report measures for each emotion: &quot;content,&quot; &quot;amused,&quot; &quot;surprised,&quot; &quot;fearful,&quot; &quot;angry,&quot; &quot;sad,&quot; and &quot;neutral.&quot; Measures of valence were computed by averaging the values from the self-report ratings of &quot;good,&quot; &quot;positive,&quot; and &quot;pleasant,&quot; while subtracting the average of the values from the &quot;bad,&quot; &quot;negative,&quot; and &quot;unpleasant&quot; ratings for each emotion category. Similarly, arousal indices were calculated by averaging ratings for the items &quot;agitated,&quot; &quot;active,&quot; and &quot;excited,&quot; while subtracting the average values of &quot;calm,&quot; &quot;passive,&quot; and &quot;relaxed.&quot; We calculated the Euclidean distance between each pair of emotion categories, yielding 21 measures of proximity of the induced emotions in an affective space characterized either by valence and This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. 684 KRAGEL AND LABAR arousal or discrete categories. Distances were calculated on test data of each repetition, ensuring independence from model construction. Correlations between these proximity values and the number of classification errors were computed on each repetition of the subsampling procedure to test whether information captured in pattern classification is driven by these affective dimensions. Results Manipulation Check Contrasts from the ANOVA of self-report variables revealed that the target emotions were successfully induced. Post hoc analysis of planned contrasts revealed that, on average, the experience of each category of stimuli was judged to be greater for the intended emotion than alternative categories (see Decoding Distinct Emotions from Response Patterns Feature extraction produced an array of 23 self-report and 16 autonomic activity measures (see Examining the relationship between the number of classification errors and location in a two-dimensional affective space revealed that valence and arousal contributed selectively to classification of self-report compared to physiological responses. Plotting the location of all 560 trials based on self-report of valence and arousal revealed a broad sampling of affective space, with considerable overlap between distinct emotions (see Testing the relationship between the similarity of experienced emotion on the basis of categorical self-report items and classification errors indicated that both experiential and physiological responding were organized in a categorical manner (see methods). Classification errors were fewer when the Euclidean distance increased in an affective space constructed from seven distinct categories for both self-report (r ϭ Ϫ0.52, z ϭ Ϫ8.68, p Ͻ .001) and peripheral physiological responses (r ϭ Ϫ0.17, z ϭ Ϫ1.19, p Ͻ .001). Further, direct comparisons revealed both self-report (z ϭ Ϫ8.51, p Ͻ .001) and peripheral autonomic measures (z ϭ Ϫ7.93, p Ͻ .001) exhibited a closer correspondence to categorical than dimensional arrangement of affective space. Together, the present results demonstrate that information in peripheral physiological signals capable of discriminating between distinct emotional states primarily reflects categorical aspects of emotional experience, whereas self-reports incorporate aspects of both categorical and dimensional structure

What makes a pattern? Matching decoding methods to data in multivariate pattern analysis

Research in neuroscience faces the challenge of integrating information across different spatial scales of brain function. A promising technique for harnessing information at a range of spatial scales is multivariate pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) data. While the prevalence of MVPA has increased dramatically in recent years, its typical implementations for classification of mental states utilize only a subset of the information encoded in local fMRI signals. We review published studies employing multivariate pattern classification since the technique’s introduction, which reveal an extensive focus on the improved detection power that linear classifiers provide over traditional analysis techniques. We demonstrate using simulations and a searchlight approach, however, that nonlinear classifiers are capable of extracting distinct information about interactions within a local region. We conclude that for spatially localized analyses, such as searchlight and region of interest, multiple classification approaches should be compared in order to match fMRI analyses to the properties of local circuits

Neural networks supporting autobiographical memory retrieval in posttraumatic stress disorder

Posttraumatic stress disorder (PTSD) affects the functional recruitment and connectivity between neural regions during autobiographical memory (AM) retrieval that overlap with default and control networks. Whether such univariate changes relate to potential differences in the contributions of the large-scale neural networks supporting cognition in PTSD is unknown. In the present functional MRI study, we employed independent-component analysis to examine the influence of the engagement of neural networks during the recall of personal memories in a PTSD group (15 participants) as compared to non-trauma-exposed healthy controls (14 participants). We found that the PTSD group recruited similar neural networks when compared to the controls during AM recall, including default-network subsystems and control networks, but group differences emerged in the spatial and temporal characteristics of these networks. First, we found spatial differences in the contributions of the anterior and posterior midline across the networks, and of the amygdala in particular, for the medial temporal subsystem of the default network. Second, we found temporal differences within the medial prefrontal subsystem of the default network, with less temporal coupling of this network during AM retrieval in PTSD relative to controls. These findings suggest that the spatial and temporal characteristics of the default and control networks potentially differ in a PTSD group versus healthy controls and contribute to altered recall of personal memory

Visual looming is a primitive for human emotion

Summary: The neural computations for looming detection are strikingly similar across species. In mammals, information about approaching threats is conveyed from the retina to the midbrain superior colliculus, where approach variables are computed to enable defensive behavior. Although neuroscientific theories posit that midbrain representations contribute to emotion through connectivity with distributed brain systems, it remains unknown whether a computational system for looming detection can predict both defensive behavior and phenomenal experience in humans. Here, we show that a shallow convolutional neural network based on the Drosophila visual system predicts defensive blinking to looming objects in infants and superior colliculus responses to optical expansion in adults. Further, the neural network’s responses to naturalistic video clips predict self-reported emotion largely by way of subjective arousal. These findings illustrate how a simple neural network architecture optimized for a species-general task relevant for survival explains motor and experiential components of human emotion

Dynamic neural networks supporting memory retrieval

How do separate neural networks interact to support complex cognitive processes such as remembrance of the personal past? Autobiographical memory (AM) retrieval recruits a consistent pattern of activation that potentially comprises multiple neural networks. However, it is unclear how such large-scale neural networks interact and are modulated by properties of the memory retrieval process. In the present functional MRI (fMRI) study, we combined independent component analysis (ICA) and dynamic causal modeling (DCM) to understand the neural networks supporting AM retrieval. ICA revealed four task-related components consistent with the previous literature: 1) medial prefrontal cortex (PFC) network, associated with self-referential processes, 2) medial temporal lobe (MTL) network, associated with memory, 3) frontoparietal network, associated with strategic search, and 4) cingulooperculum network, associated with goal maintenance. DCM analysis revealed that the medial PFC network drove activation within the system, consistent with the importance of this network to AM retrieval. Additionally, memory accessibility and recollection uniquely altered connectivity between these neural networks. Recollection modulated the influence of the medial PFC on the MTL network during elaboration, suggesting that greater connectivity among subsystems of the default network supports greater re-experience. In contrast, memory accessibility modulated the influence of frontoparietal and MTL networks on the medial PFC network, suggesting that ease of retrieval involves greater fluency among the multiple networks contributing to AM. These results show the integration between neural networks supporting AM retrieval and the modulation of network connectivity by behavior