Retrospective on the First Passive Brain-Computer Interface Competition on Cross-Session Workload Estimation

International audienceAs is the case in several research domains, data sharing is still scarce in the field of Brain-Computer Interfaces (BCI), and particularly in that of passive BCIs— i.e ., systems that enable implicit interaction or task adaptation based on a user's mental state(s) estimated from brain measures. Moreover, research in this field is currently hindered by a major challenge, which is tackling brain signal variability such as cross-session variability. Hence, with a view to develop good research practices in this field and to enable the whole community to join forces in working on cross-session estimation, we created the first passive brain-computer interface competition on cross-session workload estimation. This competition was part of the 3rd International Neuroergonomics conference. The data were electroencephalographic recordings acquired from 15 volunteers (6 females; average 25 y.o.) who performed 3 sessions—separated by 7 days—of the Multi-Attribute Task Battery-II (MATB-II) with 3 levels of difficulty per session (pseudo-randomized order). The data -training and testing sets—were made publicly available on Zenodo along with Matlab and Python toy code ( https://doi.org/10.5281/zenodo.5055046 ). To this day, the database was downloaded more than 900 times (unique downloads of all version on the 10th of December 2021: 911). Eleven teams from 3 continents (31 participants) submitted their work. The best achieving processing pipelines included a Riemannian geometry-based method. Although better than the adjusted chance level (38% with an α at 0.05 for a 3-class classification problem), the results still remained under 60% of accuracy. These results clearly underline the real challenge that is cross-session estimation. Moreover, they confirmed once more the robustness and effectiveness of Riemannian methods for BCI. On the contrary, chance level results were obtained by one third of the methods—4 teams- based on Deep Learning. These methods have not demonstrated superior results in this contest compared to traditional methods, which may be due to severe overfitting. Yet this competition is the first step toward a joint effort to tackle BCI variability and to promote good research practices including reproducibility

Towards model-based flexible and adaptive image forensics

Les images numériques sont devenues un moyen de communication standard et universel. Elles prennent place dans notre vie de tous les jours, ce qui entraîne directement des inquiétudes quant à leur intégrité. Nos travaux de recherche étudient différentes méthodes pour examiner l’authenticité d’une image numérique. Nous nous plaçons dans un contexte réaliste où les images sont en grandes quantités et avec une large diversité de manipulations et falsifications ainsi que de sources. Cela nous a poussé à développer des méthodes flexibles et adaptative face à cette diversité.Nous nous sommes en premier lieu intéressés à la détection de manipulations à l’aide de la modélisation statistiques des images. Les manipulations sont des opérations élémentaires telles qu’un flou, l’ajout de bruit ou une compression. Dans ce cadre, nous nous sommes plus particulièrement focalisés sur les effets d’un pré-traitement. A cause de limitations de stockage et autres, une image peut être re-dimensionnée ou re-compressée juste après sa capture. L’ajout d’une manipulation se fait donc ensuite sur une image déjà pré-traitée. Nous montrons qu’un pré-redimensionnement pour les images de test induit une chute de performance pour des détecteurs entraînés avec des images en pleine taille. Partant de ce constat, nous introduisons deux nouvelles méthodes pour mitiger cette chute de performance pour des détecteurs basés sur l’utilisation de mixtures de gaussiennes. Ces détecteurs modélisent les statistiques locales, sur des tuiles (patches), d’images naturelles. Cela nous permet de proposer une adaptation de modèle guidée par les changements dans les statistiques locales de l’image. Notre première méthode est une adaptation entièrement non-supervisée, alors que la seconde requière l’accès à quelques labels, faiblement supervisé, pour les images pré-resizées.Ensuite, nous nous sommes tournés vers la détection de falsifications et plus spécifiquement l’identification de copier-coller. Le copier-coller est l’une des falsification les plus populaires. Une zone source est copiée vers une zone cible de la même image. La grande majorité des détecteurs existants identifient indifféremment les deux zones (source et cible). Dans un scénario opérationnel, seulement la zone cible est intéressante car uniquement elle représente une zone de falsification. Ainsi, nous proposons une méthode pour discerner les deux zones. Notre méthode utilise également la modélisation locale des statistiques de l’image à l’aide de mixtures de gaussiennes. La procédure est spécifique à chaque image et ainsi évite la nécessité d’avoir recours à de larges bases d’entraînement et permet une plus grande flexibilité.Des résultats expérimentaux pour toutes les méthodes précédemment décrites sont présentés sur des benchmarks classiques de la littérature et comparés aux méthodes de l’état de l’art. Nous montrons que le détecteur classique de détection de manipulations basé sur les mixtures de gaussiennes, associé à nos nouvelles méthodes d’adaptation de modèle peut surpasser les résultats de récentes méthodes deep-learning. Notre méthode de discernement entre source/cible pour copier-coller égale ou même surpasse les performances des dernières méthodes d’apprentissage profond. Nous expliquons ces bons résultats des méthodes classiques face aux méthodes d’apprentissage profond par la flexibilité et l’adaptabilité supplémentaire dont elles font preuve.Pour finir, cette thèse s’est déroulée dans le contexte très spécial d’un concours organisé conjointement par l’Agence National de la Recherche et la Direction Général de l’Armement. Nous décrivons dans un appendice, les différents tours de ce concours et les méthodes que nous avons développé. Nous dressons également un bilan des enseignements de cette expérience qui avait pour but de passer de benchmarks publics à une détection de falsifications d’images très réalistes.Images are nowadays a standard and mature medium of communication.They appear in our day to day life and therefore they are subject to concernsabout security. In this work, we study different methods to assess theintegrity of images. Because of a context of high volume and versatilityof tampering techniques and image sources, our work is driven by the necessity to developflexible methods to adapt the diversity of images.We first focus on manipulations detection through statistical modeling ofthe images. Manipulations are elementary operations such as blurring,noise addition, or compression. In this context, we are more preciselyinterested in the effects of pre-processing. Because of storagelimitation or other reasons, images can be resized or compressed justafter their capture. Addition of a manipulation would then be applied on analready pre-processed image. We show that a pre-resizing of test datainduces a drop of performance for detectors trained on full-sized images.Based on these observations, we introduce two methods to counterbalancethis performance loss for a pipeline of classification based onGaussian Mixture Models. This pipeline models the local statistics, onpatches, of natural images. It allows us to propose adaptation of themodels driven by the changes in local statistics. Our first method ofadaptation is fully unsupervised while the second one, only requiring a fewlabels, is weakly supervised. Thus, our methods are flexible to adaptversatility of source of images.Then we move to falsification detection and more precisely to copy-moveidentification. Copy-move is one of the most common image tampering technique. Asource area is copied into a target area within the same image. The vastmajority of existing detectors identify indifferently the two zones(source and target). In an operational scenario, only the target arearepresents a tampering area and is thus an area of interest. Accordingly, wepropose a method to disentangle the two zones. Our method takesadvantage of local modeling of statistics in natural images withGaussian Mixture Model. The procedure is specific for each image toavoid the necessity of using a large training dataset and to increase flexibility.Results for all the techniques described above are illustrated on publicbenchmarks and compared to state of the art methods. We show that theclassical pipeline for manipulations detection with Gaussian MixtureModel and adaptation procedure can surpass results of fine-tuned andrecent deep-learning methods. Our method for source/target disentanglingin copy-move also matches or even surpasses performances of the latestdeep-learning methods. We explain the good results of these classicalmethods against deep-learning by their additional flexibility andadaptation abilities.Finally, this thesis has occurred in the special context of a contestjointly organized by the French National Research Agency and theGeneral Directorate of Armament. We describe in the Appendix thedifferent stages of the contest and the methods we have developed, as well asthe lessons we have learned from this experience to move the image forensics domain into the wild

Vers une approche basée modèle-image flexible et adaptative en criminalistique des images

Images are nowadays a standard and mature medium of communication.They appear in our day to day life and therefore they are subject to concernsabout security. In this work, we study different methods to assess theintegrity of images. Because of a context of high volume and versatilityof tampering techniques and image sources, our work is driven by the necessity to developflexible methods to adapt the diversity of images.We first focus on manipulations detection through statistical modeling ofthe images. Manipulations are elementary operations such as blurring,noise addition, or compression. In this context, we are more preciselyinterested in the effects of pre-processing. Because of storagelimitation or other reasons, images can be resized or compressed justafter their capture. Addition of a manipulation would then be applied on analready pre-processed image. We show that a pre-resizing of test datainduces a drop of performance for detectors trained on full-sized images.Based on these observations, we introduce two methods to counterbalancethis performance loss for a pipeline of classification based onGaussian Mixture Models. This pipeline models the local statistics, onpatches, of natural images. It allows us to propose adaptation of themodels driven by the changes in local statistics. Our first method ofadaptation is fully unsupervised while the second one, only requiring a fewlabels, is weakly supervised. Thus, our methods are flexible to adaptversatility of source of images.Then we move to falsification detection and more precisely to copy-moveidentification. Copy-move is one of the most common image tampering technique. Asource area is copied into a target area within the same image. The vastmajority of existing detectors identify indifferently the two zones(source and target). In an operational scenario, only the target arearepresents a tampering area and is thus an area of interest. Accordingly, wepropose a method to disentangle the two zones. Our method takesadvantage of local modeling of statistics in natural images withGaussian Mixture Model. The procedure is specific for each image toavoid the necessity of using a large training dataset and to increase flexibility.Results for all the techniques described above are illustrated on publicbenchmarks and compared to state of the art methods. We show that theclassical pipeline for manipulations detection with Gaussian MixtureModel and adaptation procedure can surpass results of fine-tuned andrecent deep-learning methods. Our method for source/target disentanglingin copy-move also matches or even surpasses performances of the latestdeep-learning methods. We explain the good results of these classicalmethods against deep-learning by their additional flexibility andadaptation abilities.Finally, this thesis has occurred in the special context of a contestjointly organized by the French National Research Agency and theGeneral Directorate of Armament. We describe in the Appendix thedifferent stages of the contest and the methods we have developed, as well asthe lessons we have learned from this experience to move the image forensics domain into the wild.Les images numériques sont devenues un moyen de communication standard et universel. Elles prennent place dans notre vie de tous les jours, ce qui entraîne directement des inquiétudes quant à leur intégrité. Nos travaux de recherche étudient différentes méthodes pour examiner l’authenticité d’une image numérique. Nous nous plaçons dans un contexte réaliste où les images sont en grandes quantités et avec une large diversité de manipulations et falsifications ainsi que de sources. Cela nous a poussé à développer des méthodes flexibles et adaptative face à cette diversité.Nous nous sommes en premier lieu intéressés à la détection de manipulations à l’aide de la modélisation statistiques des images. Les manipulations sont des opérations élémentaires telles qu’un flou, l’ajout de bruit ou une compression. Dans ce cadre, nous nous sommes plus particulièrement focalisés sur les effets d’un pré-traitement. A cause de limitations de stockage et autres, une image peut être re-dimensionnée ou re-compressée juste après sa capture. L’ajout d’une manipulation se fait donc ensuite sur une image déjà pré-traitée. Nous montrons qu’un pré-redimensionnement pour les images de test induit une chute de performance pour des détecteurs entraînés avec des images en pleine taille. Partant de ce constat, nous introduisons deux nouvelles méthodes pour mitiger cette chute de performance pour des détecteurs basés sur l’utilisation de mixtures de gaussiennes. Ces détecteurs modélisent les statistiques locales, sur des tuiles (patches), d’images naturelles. Cela nous permet de proposer une adaptation de modèle guidée par les changements dans les statistiques locales de l’image. Notre première méthode est une adaptation entièrement non-supervisée, alors que la seconde requière l’accès à quelques labels, faiblement supervisé, pour les images pré-resizées.Ensuite, nous nous sommes tournés vers la détection de falsifications et plus spécifiquement l’identification de copier-coller. Le copier-coller est l’une des falsification les plus populaires. Une zone source est copiée vers une zone cible de la même image. La grande majorité des détecteurs existants identifient indifféremment les deux zones (source et cible). Dans un scénario opérationnel, seulement la zone cible est intéressante car uniquement elle représente une zone de falsification. Ainsi, nous proposons une méthode pour discerner les deux zones. Notre méthode utilise également la modélisation locale des statistiques de l’image à l’aide de mixtures de gaussiennes. La procédure est spécifique à chaque image et ainsi évite la nécessité d’avoir recours à de larges bases d’entraînement et permet une plus grande flexibilité.Des résultats expérimentaux pour toutes les méthodes précédemment décrites sont présentés sur des benchmarks classiques de la littérature et comparés aux méthodes de l’état de l’art. Nous montrons que le détecteur classique de détection de manipulations basé sur les mixtures de gaussiennes, associé à nos nouvelles méthodes d’adaptation de modèle peut surpasser les résultats de récentes méthodes deep-learning. Notre méthode de discernement entre source/cible pour copier-coller égale ou même surpasse les performances des dernières méthodes d’apprentissage profond. Nous expliquons ces bons résultats des méthodes classiques face aux méthodes d’apprentissage profond par la flexibilité et l’adaptabilité supplémentaire dont elles font preuve.Pour finir, cette thèse s’est déroulée dans le contexte très spécial d’un concours organisé conjointement par l’Agence National de la Recherche et la Direction Général de l’Armement. Nous décrivons dans un appendice, les différents tours de ce concours et les méthodes que nous avons développé. Nous dressons également un bilan des enseignements de cette expérience qui avait pour but de passer de benchmarks publics à une détection de falsifications d’images très réalistes

Disentangling Copy-Moved Source and Target Areas

International audienceCopy-move is a very popular image falsification where a semantically coherent part of the image, the source area, is copied and pasted at another position within the same image as the so-called target area. The majority of existing copy-move detectors search for matching areas and thus identify the source and target zones indifferently, while only the target really represents a tampered area. To the best of our knowledge, at the moment of preparing this paper there has been only one published method called BusterNet that is capable of performing source and target disambiguation by using a specifically designed deep neural network. Different from the deep-learning-based BusterNet method, we propose in this paper a source and target disentangling approach based on local statistical model of image patches. Our proposed method acts as a secondstage detector after a first stage of copy-move detection of duplicated areas. We had the following intuition: even if no manipulation (e.g., scaling and rotation) is added on target area, its boundaries should expose a statistical deviation from the pristine area and the source area; further, if the target area is manipulated, the deviation should appear not only on the boundaries but on the full zone. Our method relies on machine learning tool with Gaussian Mixture Model to describe likelihood of image patches. Likelihoods are then compared between the pristine region and the candidate source/target areas as identified by the first-stage detector. Experiments and comparisons demonstrate the effectiveness of the proposed method

GRAFT : Adaptation non-supervisée au re-dimensionnement pour la détection de manipulation d'image

National audienc

GRAFT: Unsupervised Adaptation to Resizing for Detection of Image Manipulation

International audienceA large number of methods for forensics of image manipulation relies on detecting fingerprints in residuals or noises. Therefore, these detection methods are bound to be sensitive to noise generated by the image acquisition process, as well as any pre-processing. We show that a difference in pre-processing pipelines between training and testing sets induces performance losses for various classifiers. We focus on a particular pre-processing: resizing. It corresponds to a typical scenario where images may be resized (e.g., downscaled to reduce storage) prior to being manipulated. This performance loss due to pre-resizing could be troublesome but has been rarely investigated in the image forensics field. We propose a new and effective adaptation method for one state-of-the-art image manipulation detection pipeline, and we call our proposed method Gaussian mixture model Resizing Adaptation by Fine-Tuning (GRAFT). Adaptation is performed in an unsupervised fashion, i.e., without using any ground-truth label in the pre-resized testing domain, for the detection of image manipulation on very small patches. Experimental results show that the proposed GRAFT method can effectively improve the detection accuracy in this challenging scenario of unsupervised adaptation to resizing pre-processing

Benchmarking cEEGrid and Solid Gel-Based Electrodes to Classify Inattentional Deafness in a Flight Simulator

Transfer from experiments in the laboratory to real-life tasks is challenging due notably to the inability to reproduce the complexity of multitasking dynamic everyday life situations in a standardized lab condition and to the bulkiness and invasiveness of recording systems preventing participants from moving freely and disturbing the environment. In this study, we used a motion flight simulator to induce inattentional deafness to auditory alarms, a cognitive difficulty arising in complex environments. In addition, we assessed the possibility of two low-density EEG systems a solid gel-based electrode Enobio (Neuroelectrics, Barcelona, Spain) and a gel-based cEEGrid (TMSi, Oldenzaal, Netherlands) to record and classify brain activity associated with inattentional deafness (misses vs. hits to odd sounds) with a small pool of expert participants. In addition to inducing inattentional deafness (missing auditory alarms) at much higher rates than with usual lab tasks (34.7% compared to the usual 5%), we observed typical inattentional deafness-related activity in the time domain but also in the frequency and time-frequency domains with both systems. Finally, a classifier based on Riemannian Geometry principles allowed us to obtain more than 70% of single-trial classification accuracy for both mobile EEG, and up to 71.5% for the cEEGrid (TMSi, Oldenzaal, Netherlands). These results open promising avenues toward detecting cognitive failures in real-life situations, such as real flight

Burst c-VEP Based BCI: Optimizing stimulus design for enhanced classification with minimal calibration data and improved user experience

The utilization of aperiodic flickering visual stimuli under the form of code-modulated Visual Evoked Potentials (c-VEP) represents a pivotal advancement in the field of reactive Brain–Computer Interface (rBCI). A major advantage of the c-VEP approach is that the training of the model is independent of the number and complexity of targets, which helps reduce calibration time. Nevertheless, the existing designs of c-VEP stimuli can be further improved in terms of visual user experience but also to achieve a higher signal-to-noise ratio, while shortening the selection time and calibration process. In this study, we introduce an innovative variant of code-VEP, referred to as “Burst c-VEP”. This original approach involves the presentation of short bursts of aperiodic visual flashes at a deliberately slow rate, typically ranging from two to four flashes per second. The rationale behind this design is to leverage the sensitivity of the primary visual cortex to transient changes in low-level stimuli features to reliably elicit distinctive series of visual evoked potentials. In comparison to other types of faster-paced code sequences, burst c-VEP exhibit favorable properties to achieve high bitwise decoding performance using convolutional neural networks (CNN), which yields potential to attain faster selection time with the need for less calibration data. Furthermore, our investigation focuses on reducing the perceptual saliency of c-VEP through the attenuation of visual stimuli contrast and intensity to significantly improve users’ visual comfort. The proposed solutions were tested through an offline 4-classes c-VEP protocol involving 12 participants. Following a factorial design, participants were instructed to focus on c-VEP targets whose pattern (burst and maximum-length sequences) and amplitude (100% or 40% amplitude depth modulations) were manipulated across experimental conditions. Firstly, the full amplitude burst c-VEP sequences exhibited higher accuracy, ranging from 90.5% (with 17.6s of calibration data) to 95.6% (with 52.8s of calibration data), compared to its m-sequence counterpart (71.4% to 85.0%). The mean selection time for both types of codes (1.5 s) compared favorably to reports from previous studies. Secondly, our findings revealed that lowering the intensity of the stimuli only slightly decreased the accuracy of the burst code sequences to 94.2% while leading to substantial improvements in terms of user experience. Taken together, these results demonstrate the high potential of the proposed burst codes to advance reactive BCI both in terms of performance and usability. The collected dataset, along with the proposed CNN architecture implementation, are shared through open-access repositories

Improving user experience of SSVEP-BCI through reduction of stimuli amplitude depth

Steady-States Visually Evoked Potentials (SSVEP) is currently one of the most widely used paradigms in Brain-Computer Interfaces (BCI). Although the SSVEP-BCI are characterized by their high and robust classification performance, the repetitive presentation of flickering stimuli is uncomfortable from a user experience perspective point of view. Indeed, the low-level visual features of SSVEP stimuli make them straining to the eyes over time and could be disruptive to the execution of tasks requiring sustained attention. They could even induce epileptic seizures. This study explores the reduction of the stimulation amplitude depth (a magnitude diminution of 90%) for the design of SSVEP stimuli as a solution to improve user comfort. The classification accuracy obtained by different pipelines was systematically compared between low amplitude and standard full amplitude SSVEP stimuli. The results reveal high classification accuracy for both high (99.8%) and low magnitude (80.2%) stimuli using the Task-Related Component Analysis (TRCA) classification method. The present findings demonstrate the validity of reducing SSVEP stimuli amplitude to increase users’ comfort paving the way for transparent BCI operation

Improving user experience of SSVEP BCI through low amplitude depth and high frequency stimuli design

Steady-States Visually Evoked Potentials (SSVEP) refer to the sustained rhythmic activity observed in surface electroencephalography (EEG) in response to the presentation of repetitive visual stimuli (RVS). Due to their robustness and rapid onset, SSVEP have been widely used in Brain Computer Interfaces (BCI). However, typical SSVEP stimuli are straining to the eyes and present risks of triggering epileptic seizures. Reducing visual stimuli contrast or extending their frequency range both appear as relevant solutions to address these issues. It however remains sparsely documented how BCI performance is impacted by these features and to which extent user experience can be improved. We conducted two studies to systematically characterize the effects of frequency and amplitude depth reduction on SSVEP response. The results revealed that although high frequency stimuli improve visual comfort, their classification performance were not competitive enough to design a reliable/responsive BCI. Importantly, we found that the amplitude depth reduction of low frequency RVS is an effective solution to improve user experience while maintaining high classification performance. These findings were further validated by an online T9 SSVEP-BCI in which stimuli with 40% amplitude depth reduction achieved comparable results (>90% accuracy) to full amplitude stimuli while significantly improving user experience