Measuring metacognitive performance: type 1 performance dependence and test-retest reliability

Research on metacognition-thinking about thinking-has grown rapidly and fostered our understanding of human cognition in healthy individuals and clinical populations. Of central importance is the concept of metacognitive performance, which characterizes the capacity of an individual to estimate and report the accuracy of primary (type 1) cognitive processes or actions ensuing from these processes. Arguably one of the biggest challenges for measures of metacognitive performance is their dependency on objective type 1 performance, although more recent methods aim to address this issue. The present work scrutinizes the most popular metacognitive performance measures in terms of two critical characteristics: independence of type 1 performance and test-retest reliability. Analyses of data from the Confidence Database (total N = 6912) indicate that no current metacognitive performance measure is independent of type 1 performance. The shape of this dependency is largely reproduced by extending current models of metacognition with a source of metacognitive noise. Moreover, the reliability of metacognitive performance measures is highly sensitive to the combination of type 1 performance and trial number. Importantly, trial numbers frequently employed in metacognition research are too low to achieve an acceptable level of test-retest reliability. Among common task characteristics, simultaneous choice and confidence reports most strongly improved reliability. Finally, general recommendations about design choices and analytical remedies for studies investigating metacognitive performance are provided

Reverse engineering of metacognition

The human ability to introspect on thoughts, perceptions or actions - metacognitive ability - has become a focal topic of both cognitive basic and clinical research. At the same time it has become increasingly clear that currently available quantitative tools are limited in their ability to make unconfounded inferences about metacognition. As a step forward, the present work intro-duces a comprehensive modeling framework of metacognition that allows for inferences about metacognitive noise and metacognitive biases during the readout of decision values or at the confi-dence reporting stage. The model assumes that confidence results from a continuous but noisy and potentially biased transformation of decision values, described by a confidence link function. A canonical set of metacognitive noise distributions is introduced which differ, amongst others, in their predictions about metacognitive sign flips of decision values. Successful recovery of model param-eters is demonstrated, and the model is validated on an empirical data set. In particular, it is shown that metacognitive noise and bias parameters correlate with conventional behavioral measures. Crucially, in contrast to these conventional measures, metacognitive noise parameters inferred from the model are shown to be independent of performance. This work is accompanied by a toolbox (ReMeta) that allows researchers to estimate key parameters of metacognition in confidence datasets

Using machine learning to resolve the neural basis of alcohol dependence

Alcohol dependence is a psychiatric disorder with a lifetime prevalence of over 10% and a leading cause of morbidity and premature death. A better understanding of the neural mechanisms underlying alcohol dependence to improve prevention, diagnosis and treatment is thus of great societal interest. Recent advancements in the analysis of neuroimaging data based on machine learning have opened new paths to a better quantitative understanding of the disorder. The present habilitation reviews both works with a focus on improving machine learning methodology and empirical works in which machine learning was applied to investigate the neural basis of alcohol dependence. The methodological works advanced several aspects of machine learning in neuroimaging. In particular, they introduced i) a novel classifier (weighted robust distance – WeiRD), which operates parameter-free, computationally efficient and enables a transparent inspection of feature importances, ii) a method to preprocess neuroimaging data based on multivariate noise normalization, which yielded a substantial improvement in classification performance compared to previous the state-of-the-art, and iii) a novel method to reintroduce meaningful graded information into discretized classification accuracies by utilizing classifier decision values. Drawing on a large neuroimaging dataset of alcohol-dependent patients and controls from the LeAD-study (www.lead-studie.de; clinical trial number: NCT01679145), machine learning methods were applied in empirical works to investigate structural and functional alterations in alcohol dependence. Structural damage associated with alcohol dependence were investigated from two conceptually different angles. A first study was aimed at providing the first quantitative evidence for a long-standing hypothesis about the damaging effects of alcohol – the premature aging hypothesis. To this end, a machine learning model was trained on the relationship between grey-matter pattern information and chronological age in a healthy control group and then applied to the sample of alcohol-dependent patients. The predicted ‘brain age’ of patients was found to be was several years higher than their chronological age, thus not only providing quantitative evidence for brain aging in alcohol dependence, but also showing that these aging effects are indeed substantial in relation to the human lifespan. The second study used machine learning to quantify the predictive accuracy of grey-matter pattern information for the diagnosis and a severity measure (lifetime consumption) of alcohol dependence. On average, machine learning models correctly predicted the diagnosis in three of four subjects and accurately estimated the amount of lifetime alcohol consumption. Closer inspection of the prediction model indicated an important role of dorsal anterior cingulate cortex. Comparison with an experienced radiologist, who, like the classifier, was provided with the structural brain scans of the subjects, demonstrated superior performance of computer-based classification and in addition a more effective consideration of demographic information (age and gender). Finally, a third study used functional magnetic resonance imaging to investigate a specific hypothesis about reduced goal-directed learning in alcohol dependence as well as its relation to relapse after detoxification. Computational modelling in combination with machine learning revealed that the interaction of model-based learning and high alcohol expectancies was predictive of diagnosis (patients versus controls) and treatment outcome (abstainers versus relapsers). This finding was paralleled by a signature of model-based learning in medial prefrontal cortex, which was reduced in patients relative to controls and in relapsers relative to abstainers. In sum, the works presented in this habilitation advance machine learning methods for neuroimaging and show that these methods yield novel insights into the neural basis of alcohol dependence. An emerging theme across the three empirical studies on alcohol dependence is the disturbance of executive frontal brain structure and function, supporting a top-down rather than bottom-up view for the aetiology of alcohol dependence

Mesolimbic confidence signals guide perceptual learning in the absence of external feedback

It is well established that learning can occur without external feedback, yet normative reinforcement learning theories have difficulties explaining such instances of learning. Here, we propose that human observers are capable of generating their own feedback signals by monitoring internal decision variables. We investigated this hypothesis in a visual perceptual learning task using fMRI and confidence reports as a measure for this monitoring process. Employing a novel computational model in which learning is guided by confidence-based reinforcement signals, we found that mesolimbic brain areas encoded both anticipation and prediction error of confidence—in remarkable similarity to previous findings for external reward-based feedback. We demonstrate that the model accounts for choice and confidence reports and show that the mesolimbic confidence prediction error modulation derived through the model predicts individual learning success. These results provide a mechanistic neurobiological explanation for learning without external feedback by augmenting reinforcement models with confidence-based feedback

A multimodal neuroimaging classifier for alcohol dependence

With progress in magnetic resonance imaging technology and a broader dissemination of state-of-the-art imaging facilities, the acquisition of multiple neuroimaging modalities is becoming increasingly feasible. One particular hope associated with multimodal neuroimaging is the development of reliable data-driven diagnostic classifiers for psychiatric disorders, yet previous studies have often failed to find a benefit of combining multiple modalities. As a psychiatric disorder with established neurobiological effects at several levels of description, alcohol dependence is particularly well-suited for multimodal classification. To this aim, we developed a multimodal classification scheme and applied it to a rich neuroimaging battery (structural, functional task-based and functional resting-state data) collected in a matched sample of alcohol-dependent patients (N = 119) and controls (N = 97). We found that our classification scheme yielded 79.3% diagnostic accuracy, which outperformed the strongest individual modality - grey-matter density - by 2.7%. We found that this moderate benefit of multimodal classification depended on a number of critical design choices: a procedure to select optimal modality-specific classifiers, a fine-grained ensemble prediction based on cross-modal weight matrices and continuous classifier decision values. We conclude that the combination of multiple neuroimaging modalities is able to moderately improve the accuracy of machine-learning-based diagnostic classification in alcohol dependence

A multimodal neuroimaging classifier for alcohol dependence

With progress in magnetic resonance imaging technology and a broader dissemination of state-of-the-art imaging facilities, the acquisition of multiple neuroimaging modalities is becoming increasingly feasible. One particular hope associated with multimodal neuroimaging is the development of reliable data-driven diagnostic classifiers for psychiatric disorders, yet previous studies have often failed to find a benefit of combining multiple modalities. As a psychiatric disorder with established neurobiological effects at several levels of description, alcohol dependence is particularly well-suited for multimodal classification. To this aim, we developed a multimodal classification scheme and applied it to a rich neuroimaging battery (structural, functional task-based and functional resting-state data) collected in a matched sample of alcohol-dependent patients (N = 119) and controls (N = 97). We found that our classification scheme yielded 79.3% diagnostic accuracy, which outperformed the strongest individual modality - grey-matter density - by 2.7%. We found that this moderate benefit of multimodal classification depended on a number of critical design choices: a procedure to select optimal modality-specific classifiers, a fine-grained ensemble prediction based on cross-modal weight matrices and continuous classifier decision values. We conclude that the combination of multiple neuroimaging modalities is able to moderately improve the accuracy of machine-learning-based diagnostic classification in alcohol dependence

Spatial attention enhances object coding in local and distributed representations of the lateral occipital complex

The modulation of neural activity in visual cortex is thought to be a key mechanism of visual attention. The investigation of attentional modulation in high-level visual areas, however, is hampered by the lack of clear tuning or contrast response functions. In the present functional magnetic resonance imaging study we therefore systematically assessed how small voxel-wise biases in object preference across hundreds of voxels in the lateral occipital complex were affected when attention was directed to objects. We found that the strength of attentional modulation depended on a voxel's object preference in the absence of attention, a pattern indicative of an amplificatory mechanism. Our results show that such attentional modulation effectively increased the mutual information between voxel responses and object identity. Further, these local modulatory effects led to improved information-based object readout at the level of multi-voxel activation patterns and to an increased reproducibility of these patterns across repeated presentations. We conclude that attentional modulation enhances object coding in local and distributed object representations of the lateral occipital complex

A multimodal neuroimaging classifier for alcohol dependence

With progress in magnetic resonance imaging technology and a broader dissemination of state-of-the-art imaging facilities, the acquisition of multiple neuroimaging modalities is becoming increasingly feasible. One particular hope associated with multimodal neuroimaging is the development of reliable data-driven diagnostic classifiers for psychiatric disorders, yet previous studies have often failed to find a benefit of combining multiple modalities. As a psychiatric disorder with established neurobiological effects at several levels of description, alcohol dependence is particularly well-suited for multimodal classification. To this aim, we developed a multimodal classification scheme and applied it to a rich neuroimaging battery (structural, functional task-based and functional resting-state data) collected in a matched sample of alcohol-dependent patients (N = 119) and controls (N = 97). We found that our classification scheme yielded 79.3% diagnostic accuracy, which outperformed the strongest individual modality - grey-matter density - by 2.7%. We found that this moderate benefit of multimodal classification depended on a number of critical design choices: a procedure to select optimal modality-specific classifiers, a fine-grained ensemble prediction based on cross-modal weight matrices and continuous classifier decision values. We conclude that the combination of multiple neuroimaging modalities is able to moderately improve the accuracy of machine-learning-based diagnostic classification in alcohol dependence

Non-holistic coding of objects in lateral occipital complex with and without attention

A fundamental issue in visual cognition is whether high-level visual areas code objects in a part-based or a view-based (holistic) format. By examining the viewpoint invariance of object recognition, previous behavioral and neuroimaging studies have yielded ambiguous results, supporting both types of representational formats. A critical factor distinguishing the two formats could be the availability of attentional resources, as a number of studies have found greater viewpoint invariance for attended compared to unattended objects. It has therefore been suggested that attention is necessary to enable part-based representations, whereas holistic representations are automatically activated irrespective of attention. In this functional magnetic resonance imaging study we used a multivariate approach to probe the format of object representations in human lateral occipital complex (LOC) and its dependence on attention. We presented human participants with intact and half-split versions of objects that were either attended or unattended. Cross-classifying between intact and split objects, we found that the objectrelated information coded in activation patterns of intact objects is fully preserved in the patterns of split objects and vice versa. Importantly, the generalization between intact and split objects did not depend on attention. Our findings demonstrate that LOC codes objects in a non-holistic format, both in the presence and absence of attention