Deliverable D1.1 State of the art and requirements analysis for hypervideo

This deliverable presents a state-of-art and requirements analysis report for hypervideo authored as part of the WP1 of the LinkedTV project. Initially, we present some use-case (viewers) scenarios in the LinkedTV project and through the analysis of the distinctive needs and demands of each scenario we point out the technical requirements from a user-side perspective. Subsequently we study methods for the automatic and semi-automatic decomposition of the audiovisual content in order to effectively support the annotation process. Considering that the multimedia content comprises of different types of information, i.e., visual, textual and audio, we report various methods for the analysis of these three different streams. Finally we present various annotation tools which could integrate the developed analysis results so as to effectively support users (video producers) in the semi-automatic linking of hypervideo content, and based on them we report on the initial progress in building the LinkedTV annotation tool. For each one of the different classes of techniques being discussed in the deliverable we present the evaluation results from the application of one such method of the literature to a dataset well-suited to the needs of the LinkedTV project, and we indicate the future technical requirements that should be addressed in order to achieve higher levels of performance (e.g., in terms of accuracy and time-efficiency), as necessary

Applying Wikipedia to Interactive Information Retrieval

There are many opportunities to improve the interactivity of information retrieval systems beyond the ubiquitous search box. One idea is to use knowledge bases—e.g. controlled vocabularies, classification schemes, thesauri and ontologies—to organize, describe and navigate the information space. These resources are popular in libraries and specialist collections, but have proven too expensive and narrow to be applied to everyday webscale search. Wikipedia has the potential to bring structured knowledge into more widespread use. This online, collaboratively generated encyclopaedia is one of the largest and most consulted reference works in existence. It is broader, deeper and more agile than the knowledge bases put forward to assist retrieval in the past. Rendering this resource machine-readable is a challenging task that has captured the interest of many researchers. Many see it as a key step required to break the knowledge acquisition bottleneck that crippled previous efforts. This thesis claims that the roadblock can be sidestepped: Wikipedia can be applied effectively to open-domain information retrieval with minimal natural language processing or information extraction. The key is to focus on gathering and applying human-readable rather than machine-readable knowledge. To demonstrate this claim, the thesis tackles three separate problems: extracting knowledge from Wikipedia; connecting it to textual documents; and applying it to the retrieval process. First, we demonstrate that a large thesaurus-like structure can be obtained directly from Wikipedia, and that accurate measures of semantic relatedness can be efficiently mined from it. Second, we show that Wikipedia provides the necessary features and training data for existing data mining techniques to accurately detect and disambiguate topics when they are mentioned in plain text. Third, we provide two systems and user studies that demonstrate the utility of the Wikipedia-derived knowledge base for interactive information retrieval

Brain Computations and Connectivity [2nd edition]

This is an open access title available under the terms of a CC BY-NC-ND 4.0 International licence. It is free to read on the Oxford Academic platform and offered as a free PDF download from OUP and selected open access locations. Brain Computations and Connectivity is about how the brain works. In order to understand this, it is essential to know what is computed by different brain systems; and how the computations are performed. The aim of this book is to elucidate what is computed in different brain systems; and to describe current biologically plausible computational approaches and models of how each of these brain systems computes. Understanding the brain in this way has enormous potential for understanding ourselves better in health and in disease. Potential applications of this understanding are to the treatment of the brain in disease; and to artificial intelligence which will benefit from knowledge of how the brain performs many of its extraordinarily impressive functions. This book is pioneering in taking this approach to brain function: to consider what is computed by many of our brain systems; and how it is computed, and updates by much new evidence including the connectivity of the human brain the earlier book: Rolls (2021) Brain Computations: What and How, Oxford University Press. Brain Computations and Connectivity will be of interest to all scientists interested in brain function and how the brain works, whether they are from neuroscience, or from medical sciences including neurology and psychiatry, or from the area of computational science including machine learning and artificial intelligence, or from areas such as theoretical physics

Computational Unfolding of the Human Hippocampus

The hippocampal subfields are defined by their unique cytoarchitectures, which many recent studies have tried to map to human in-vivo MRI because of their promise to further our understanding of hippocampal function, or its dysfunction in disease. However, recent anatomical literature has highlighted broad inter-individual variability in hippocampal morphology and subfield locations, much of which can be attributed to different folding configurations within hippocampal (or archicortical) tissue. Inspired in part by analogous surface-based neocortical analysis methods, the current thesis aimed to develop a standardized coordinate framework, or surface-based method, that respects the topology of all hippocampal folding configurations. I developed such a coordinate framework in Chapter 2, which was initialized by detailed manual segmentations of hippocampal grey matter and high myelin laminae which are visible in 7-Tesla MRI and which separate different hippocampal folds. This framework was leveraged to i) computationally unfold the hippocampus which provided implicit topological inter-individual alignment, ii) delineate subfields with high reliability and validity, and iii) extract novel structural features of hippocampal grey matter. In Chapter 3, I applied this coordinate framework to the open source BigBrain 3D histology dataset. With this framework, I computationally extracted morphological and laminar features and showed that they are sufficient to derive hippocampal subfields in a data-driven manner. This underscores the sensitivity of these computational measures and the validity of the applied subfield definitions. Finally, the unfolding coordinate framework developed in Chapter 2 and extended in Chapter 3 requires manual detection of different tissue classes that separate folds in hippocampal grey matter. This is costly in the time and the expertise required. Thus, in Chapter 4, I applied state-of-the-art deep learning methods in the open source Human Connectome Project MRI dataset to automate this process. This allowed for scalable application of the methods described in Chapters 2, 3, and 4 to similar new datasets, with support for extensions to suit data of different modalities or resolutions. Overall, the projects presented here provide multifaceted evidence for the strengths of a surface-based approach to hippocampal analysis as developed in this thesis, and these methods are readily deployable in new neuroimaging work

Spatio-temporal feature representations of reactivated memories

How does the human brain recover memories of past events? The neural processes of memory retrieval are still not fully uncovered. This doctoral thesis is concerned with the spatio-temporal feature representations of reactivated episodic memories. Classical theories and empirical evidence suggest that the revival of memory representations in the brain is initiated in the hippocampus, before activity patterns in cortical regions reactivate to represent previously experienced events. The current doctoral project tests the assumption that the neural processing cascade during retrieval is reversed with respect to perception. This general framework predicts that semantic concepts and modality-independent information is reconstructed before modality-specific sensory details. This backward information flow is also assumed to affect the neural representations when memories are recalled repeatedly, enhancing the integration of new information into existing conceptual networks. The first two studies investigate the neural information flow during retrieval with respect to the reactivated mnemonic representations. First, simultaneous EEG-fMRI is used to track the presumed reversed reconstruction from abstract modality-independent to sensory-specific visual and auditory memory representations. The second EEG-fMRI project then zooms in on the recall of visual memories, testing whether the visual retrieval process propagates backwards along the ventral visual stream transferring from abstract conceptual to detailed perceptual representations. The reverse reconstruction framework predicts that conceptual information, due to its prioritisation, should benefit more from repeated recall than perceptual information. Hence, the last, behavioural study investigated whether retrieval strengthens conceptual representations over perceptual ones and thus promotes the semanticisation of episodic memories. Altogether, the findings offer novel insights into retrieval-related processing cascades, in terms of their temporal and spatial dynamics and the nature of the reactivated representations. The results also provide an understanding of memory transformations during the consolidation processes that are amplified through repeated retrieval

Processing of nonverbal vocalisations in dementia

Nonverbal emotional vocalisations are fundamental communicative signals used to convey a diverse repertoire of social and emotional information. They transcend the boundaries of language and cultural specificity that hamper many neuropsychological tests, making them ideal candidates for understanding impaired socio-emotional signal processing in dementia. Symptoms related to changes in social behaviour and emotional responsiveness are poorly understood yet have significant impact on patients with dementia and those who care for them. In this thesis, I investigated processing of nonverbal emotional vocalisations in patients with Alzheimer’s disease and frontotemporal dementia (FTD), a disease spectrum encompassing three canonical syndromes characterised by marked socio-emotional and communication difficulties - behavioural variant FTD (bvFTD), semantic variant primary progressive aphasia (svPPA) and nonfluent/agrammatic variant primary progressive aphasia (nfvPPA). I demonstrated distinct profiles of impairment in identifying three salient vocalisations (laughter, crying and screaming) and the emotions they convey. All three FTD syndromes showed impairments, with the most marked deficits of emotion categorisation seen in the bvFTD group. Voxel-based morphometry was used to define critical brain substrates for processing vocalisations, identifying correlates of vocal sound processing with auditory perceptual regions (superior temporal sulcus and posterior insula) and emotion identification with limbic and medial frontal regions. The second half of this thesis focused on the more fine-grained distinction of laughter subtypes. I studied cognitive (labelling), affective (valence) and autonomic (pupillometric) processing of laughter subtypes representing dimensions of valence (mirthful versus hostile) and arousal (spontaneous versus posed). Again, FTD groups showed greatest impairment with profiles suggestive of primary perceptual deficits in nfvPPA, cognitive overgeneralisation in svPPA and disordered reward and hedonic valuation in bvFTD. Neuroanatomical correlates of explicit laughter identification included inferior frontal and cingulo-insular cortices whilst implicit processing (indexed as autonomic arousal) was particularly impaired in those conditions associated with insular compromise (nfvPPA and bvFTD). These findings demonstrate the potential of nonverbal emotional vocalisations as a probe of neural mechanisms underpinning socio-emotional dysfunction in neurodegenerative diseases