Novel methods to evaluate blindsight and develop rehabilitation strategies for patients with cortical blindness

20 à 57 % des victimes d'un accident vasculaire cérébral (AVC) sont diagnostiqués aves des déficits visuels qui réduisent considérablement leur qualité de vie. Parmi les cas extrêmes de déficits visuels, nous retrouvons les cécités corticales (CC) qui se manifestent lorsque la région visuelle primaire (V1) est atteinte. Jusqu'à présent, il n'existe aucune approche permettant d'induire la restauration visuelle des fonctions et, dans la plupart des cas, la plasticité est insuffisante pour permettre une récupération spontanée. Par conséquent, alors que la perte de la vue est considérée comme permanente, des fonctions inconscientes mais importantes, connues sous le nom de vision aveugle (blindsight), pourraient être utiles pour les stratégies de réhabilitation visuelle, ce qui suscite un vif intérêt dans le domaine des neurosciences cognitives. La vision aveugle est un phénomène rare qui dépeint une dissociation entre la performance et la conscience, principalement étudiée dans des études de cas. Dans le premier chapitre de cette thèse, nous avons abordé plusieurs questions concernant notre compréhension de la vision aveugle. Comme nous le soutenons, une telle compréhension pourrait avoir une influence significative sur la réhabilitation clinique des patients souffrant de CC. Par conséquent, nous proposons une stratégie unique pour la réhabilitation visuelle qui utilise les principes du jeu vidéo pour cibler et potentialiser les mécanismes neuronaux dans le cadre de l'espace de travail neuronal global, qui est expliqué théoriquement dans l'étude 1 et décrit méthodologiquement dans l'étude 5. En d'autres termes, nous proposons que les études de cas, en conjonction avec des critères méthodologiques améliorés, puissent identifier les substrats neuronaux qui soutiennent la vision aveugle et inconsciente. Ainsi, le travail de cette thèse a fourni trois expériences empiriques (études 2, 3 et 4) en utilisant de nouveaux standards dans l'analyse électrophysiologique qui décrivent les cas de patients SJ présentant une cécité pour les scènes complexes naturelles affectives et ML présentant une cécité pour les stimuli de mouvement. Dans les études 2 et 3, nous avons donc sondé les substrats neuronaux sous-corticaux et corticaux soutenant la cécité affective de SJ en utilisant la MEG et nous avons comparé ces corrélats à sa perception consciente. L’étude 4 nous a permis de caractériser les substrats de la détection automatique des changements en l'absence de conscience visuelle, mesurée par la négativité de discordance (en anglais visual mismatch negativity : vMMN) chez ML et dans un groupe neurotypique. Nous concluons en proposant la vMMN comme biomarqueur neuronal du traitement inconscient dans la vision normale et altérée indépendante des évaluations comportementales. Grâce à ces procédures, nous avons pu aborder certains débats ouverts dans la littérature sur la vision aveugle et sonder l'existence de voies neurales secondaires soutenant le comportement inconscient. En conclusion, cette thèse propose de combiner les perspectives empiriques et cliniques en utilisant des avancées méthodologiques et de nouvelles méthodes pour comprendre et cibler les substrats neurophysiologiques sous-jacents à la vision aveugle. Il est important de noter que le cadre offert par cette thèse de doctorat pourrait aider les études futures à construire des outils thérapeutiques ciblés efficaces et des stratégies de réhabilitation multimodale.20 to 57% of victims of a cerebrovascular accident (CVA) develop visual deficits that considerably reduce their quality of life. Among the extreme cases of visual deficits, we find cortical blindness (CC) which manifests when the primary visual region (V1) is affected. Until now, there is no approach that induces restoration of visual function and in most cases, plasticity is insufficient to allow spontaneous recovery. Therefore, while sight loss is considered permanent, unconscious yet important functions, known as blindsight, could be of use for visual rehabilitation strategies raising strong interest in cognitive neurosciences. Blindsight is a rare phenomenon that portrays a dissociation between performance and consciousness mainly investigated in case reports. In the first chapter of this thesis, we’ve addressed multiple issues about our comprehension of blindsight and conscious perception. As we argue, such understanding might have a significant influence on clinical rehabilitation patients suffering from CB. Therefore, we propose a unique strategy for visual rehabilitation that uses video game principles to target and potentiate neural mechanisms within the global neuronal workspace framework, which is theoretically explained in study 1 and methodologically described in study 5. In other words, we propose that case reports, in conjunction with improved methodological criteria, might identify the neural substrates that support blindsight and unconscious processing. Thus, the work in this Ph.D. work provided three empirical experiments (studies 2, 3, and 4) that used new standards in electrophysiological analyses as they describe the cases of patients SJ presenting blindsight for affective natural complex scenes and ML presenting blindsight for motion stimuli. In studies 2 and 3, we probed the subcortical and cortical neural substrates supporting SJ’s affective blindsight using MEG as we compared these unconscious correlates to his conscious perception. Study 4 characterizes the substrates of automatic detection of changes in the absence of visual awareness as measured by the visual mismatch negativity (vMMN) in ML and a neurotypical group. We conclude by proposing the vMMN as a neural biomarker of unconscious processing in normal and altered vision independent of behavioral assessments. As a result of these procedures, we were able to address certain open debates in the blindsight literature and probe the existence of secondary neural pathways supporting unconscious behavior. In conclusion, this thesis proposes to combine empirical and clinical perspectives by using methodological advances and novel methods to understand and target the neurophysiological substrates underlying blindsight. Importantly, the framework offered by this doctoral dissertation might help future studies build efficient targeted therapeutic tools and multimodal rehabilitation training

FTFDNet: Learning to Detect Talking Face Video Manipulation with Tri-Modality Interaction

DeepFake based digital facial forgery is threatening public media security, especially when lip manipulation has been used in talking face generation, and the difficulty of fake video detection is further improved. By only changing lip shape to match the given speech, the facial features of identity are hard to be discriminated in such fake talking face videos. Together with the lack of attention on audio stream as the prior knowledge, the detection failure of fake talking face videos also becomes inevitable. It's found that the optical flow of the fake talking face video is disordered especially in the lip region while the optical flow of the real video changes regularly, which means the motion feature from optical flow is useful to capture manipulation cues. In this study, a fake talking face detection network (FTFDNet) is proposed by incorporating visual, audio and motion features using an efficient cross-modal fusion (CMF) module. Furthermore, a novel audio-visual attention mechanism (AVAM) is proposed to discover more informative features, which can be seamlessly integrated into any audio-visual CNN architecture by modularization. With the additional AVAM, the proposed FTFDNet is able to achieve a better detection performance than other state-of-the-art DeepFake video detection methods not only on the established fake talking face detection dataset (FTFDD) but also on the DeepFake video detection datasets (DFDC and DF-TIMIT).Comment: arXiv admin note: substantial text overlap with arXiv:2203.0517

Multiple Media Correlation: Theory and Applications

This thesis introduces multiple media correlation, a new technology for the automatic alignment of multiple media objects such as text, audio, and video. This research began with the question: what can be learned when multiple multimedia components are analyzed simultaneously? Most ongoing research in computational multimedia has focused on queries, indexing, and retrieval within a single media type. Video is compressed and searched independently of audio, text is indexed without regard to temporal relationships it may have to other media data. Multiple media correlation provides a framework for locating and exploiting correlations between multiple, potentially heterogeneous, media streams. The goal is computed synchronization, the determination of temporal and spatial alignments that optimize a correlation function and indicate commonality and synchronization between media objects. The model also provides a basis for comparison of media in unrelated domains. There are many real-world applications for this technology, including speaker localization, musical score alignment, and degraded media realignment. Two applications, text-to-speech alignment and parallel text alignment, are described in detail with experimental validation. Text-to-speech alignment computes the alignment between a textual transcript and speech-based audio. The presented solutions are effective for a wide variety of content and are useful not only for retrieval of content, but in support of automatic captioning of movies and video. Parallel text alignment provides a tool for the comparison of alternative translations of the same document that is particularly useful to the classics scholar interested in comparing translation techniques or styles. The results presented in this thesis include (a) new media models more useful in analysis applications, (b) a theoretical model for multiple media correlation, (c) two practical application solutions that have wide-spread applicability, and (d) Xtrieve, a multimedia database retrieval system that demonstrates this new technology and demonstrates application of multiple media correlation to information retrieval. This thesis demonstrates that computed alignment of media objects is practical and can provide immediate solutions to many information retrieval and content presentation problems. It also introduces a new area for research in media data analysis

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

Task-dependent modulation of the visual sensory thalamus assists visual-speech recognition

The cerebral cortex modulates early sensory processing via feed-back connections to sensory pathway nuclei. The functions of this top-down modulation for human behavior are poorly understood. Here, we show that top-down modulation of the visual sensory thalamus (the lateral geniculate body, LGN) is involved in visual-speech recognition. In two independent functional magnetic resonance imaging (fMRI) studies, LGN response increased when participants processed fast-varying features of articulatory movements required for visual-speech recognition, as compared to temporally more stable features required for face identification with the same stimulus material. The LGN response during the visual-speech task correlated positively with the visual-speech recognition scores across participants. In addition, the task-dependent modulation was present for speech movements and did not occur for control conditions involving non-speech biological movements. In face-to-face communication, visual speech recognition is used to enhance or even enable understanding what is said. Speech recognition is commonly explained in frameworks focusing on cerebral cortex areas. Our findings suggest that task-dependent modulation at subcortical sensory stages has an important role for communication: Together with similar findings in the auditory modality the findings imply that task-dependent modulation of the sensory thalami is a general mechanism to optimize speech recognition