The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.EC/FP7/611570/EU/Symbiotic Mind Computer Interaction for Information Seeking/MindSeeEC/FP7/625991/EU/Hyperscanning 2.0 Analyses of Multimodal Neuroimaging Data: Concept, Methods and Applications/HYPERSCANNING 2.0DFG, 103586207, GRK 1589: Verarbeitung sensorischer Informationen in neuronalen Systeme

A COMPARISON OF CONDUCTING EFFECTIVENESS, INSTRUMENTALISTS PERFORMANCES, AND THE UNDERLYING BIOMECHANICS OF A CONDUCTOR WHILE PERFORMING EXAGGERATED AND UNDERSTATED VARIATIONS OF STACCATO AND LEGATO GESTURES

A COMPARISON OF CONDUCTING EFFECTIVENESS, INSTRUMENTALISTS PERFORMANCES, AND THE UNDERLYING BIOMECHANICS OF A CONDUCTOR WHILE PERFORMING EXAGGERATED AND UNDERSTATED VARIATIONS OF STACCATO AND LEGATO GESTURE

Monitoring and Control of Hydrocyclones by Use of Convolutional Neural Networks and Deep Reinforcement Learning

The use of convolutional neural networks for monitoring hydrocyclones from underflow images was investigated. Proof-of-concept and applied industrial considerations for hydrocyclone state detection and underflow particle size inference sensors were demonstrated. The behaviour and practical considerations of model-free reinforcement learning, incorporating the additional information provided by the sensors developed, was also discussed in a mineral processing context

How Can Physiological Computing Benefit Human-Robot Interaction?

As systems grow more automatized, the human operator is all too often overlooked. Although human-robot interaction (HRI) can be quite demanding in terms of cognitive resources, the mental states (MS) of the operators are not yet taken into account by existing systems. As humans are no providential agents, this lack can lead to hazardous situations. The growing number of neurophysiology and machine learning tools now allows for efficient operators' MS monitoring. Sending feedback on MS in a closed-loop solution is therefore at hands. Involving a consistent automated planning technique to handle such a process could be a significant asset. This perspective article was meant to provide the reader with a synthesis of the significant literature with a view to implementing systems that adapt to the operator's MS to improve human-robot operations' safety and performance. First of all, the need for this approach is detailed as regards remote operation, an example of HRI. Then, several MS identified as crucial for this type of HRI are defined, along with relevant electrophysiological markers. A focus is made on prime degraded MS linked to time-on-task and task demands, as well as collateral MS linked to system outputs (i.e. feedback and alarms). Lastly, the principle of symbiotic HRI is detailed and one solution is proposed to include the operator state vector into the system using a mixed-initiative decisional framework to drive such an interaction

Multi-modal post-editing of machine translation

As MT quality continues to improve, more and more translators switch from traditional translation from scratch to PE of MT output, which has been shown to save time and reduce errors. Instead of mainly generating text, translators are now asked to correct errors within otherwise helpful translation proposals, where repetitive MT errors make the process tiresome, while hard-to-spot errors make PE a cognitively demanding activity. Our contribution is three-fold: first, we explore whether interaction modalities other than mouse and keyboard could well support PE by creating and testing the MMPE translation environment. MMPE allows translators to cross out or hand-write text, drag and drop words for reordering, use spoken commands or hand gestures to manipulate text, or to combine any of these input modalities. Second, our interviews revealed that translators see value in automatically receiving additional translation support when a high CL is detected during PE. We therefore developed a sensor framework using a wide range of physiological and behavioral data to estimate perceived CL and tested it in three studies, showing that multi-modal, eye, heart, and skin measures can be used to make translation environments cognition-aware. Third, we present two multi-encoder Transformer architectures for APE and discuss how these can adapt MT output to a domain and thereby avoid correcting repetitive MT errors.Angesichts der stetig steigenden Qualität maschineller Übersetzungssysteme (MÜ) post-editieren (PE) immer mehr Übersetzer die MÜ-Ausgabe, was im Vergleich zur herkömmlichen Übersetzung Zeit spart und Fehler reduziert. Anstatt primär Text zu generieren, müssen Übersetzer nun Fehler in ansonsten hilfreichen Übersetzungsvorschlägen korrigieren. Dennoch bleibt die Arbeit durch wiederkehrende MÜ-Fehler mühsam und schwer zu erkennende Fehler fordern die Übersetzer kognitiv. Wir tragen auf drei Ebenen zur Verbesserung des PE bei: Erstens untersuchen wir, ob andere Interaktionsmodalitäten als Maus und Tastatur das PE unterstützen können, indem wir die Übersetzungsumgebung MMPE entwickeln und testen. MMPE ermöglicht es, Text handschriftlich, per Sprache oder über Handgesten zu verändern, Wörter per Drag & Drop neu anzuordnen oder all diese Eingabemodalitäten zu kombinieren. Zweitens stellen wir ein Sensor-Framework vor, das eine Vielzahl physiologischer und verhaltensbezogener Messwerte verwendet, um die kognitive Last (KL) abzuschätzen. In drei Studien konnten wir zeigen, dass multimodale Messung von Augen-, Herz- und Hautmerkmalen verwendet werden kann, um Übersetzungsumgebungen an die KL der Übersetzer anzupassen. Drittens stellen wir zwei Multi-Encoder-Transformer-Architekturen für das automatische Post-Editieren (APE) vor und erörtern, wie diese die MÜ-Ausgabe an eine Domäne anpassen und dadurch die Korrektur von sich wiederholenden MÜ-Fehlern vermeiden können.Deutsche Forschungsgemeinschaft (DFG), Projekt MMP

Performance evaluation for tracker-level fusion in video tracking

PhDTracker-level fusion for video tracking combines outputs (state estimations) from multiple trackers, to address the shortcomings of individual trackers. Furthermore, performance evaluation of trackers at run time (online) can determine low performing trackers that can be removed from the fusion. This thesis presents a tracker-level fusion framework that performs online tracking performance evaluation for fusion. We first introduce a method to determine time instants of tracker failure that is divided into two steps. First, we evaluate tracking performance by comparing the distributions of the tracker state and a region around the state. We use Distribution Fields to generate the distributions of both regions and compute a tracking performance score by comparing the distributions using the L1 distance. Then, we model this score as a time series and employ the Auto Regressive Moving Average method to forecast future values of the performance score. A difference between the original and forecast returns the forecast error signal that we use to detect tracking failure. We test the method with different datasets and then demonstrate its flexibility using tracking results and sequences from the Visual Object Tracking (VOT) challenge. The second part presents a tracker-level fusion method that combines the outputs of multiple trackers. The method is divided into three steps. First, we group trackers into clusters based on the spatio-temporal pair-wise relationships of their outputs. Then, we evaluate tracking performance based on reverse-time analysis with an adaptive reference frame and define the cluster with trackers that appear to be successfully following the target as the on-target cluster. Finally, we fuse the outputs of the trackers in the on-target cluster to obtain the final target state. The fusion approach uses standard tracker outputs and can therefore combine various types of trackers. We test the method with several combinations of state-of-the-art trackers, and also compare it with individual trackers and other fusion approaches.EACEA, under the EMJD ICE Project

Creating music in the classroom with tablet computers: An activity system analysis of two secondary school communities.

Tablet computers are becoming inextricably linked with innovation and change in schools. Increasingly therefore, music teachers must consider how tablet computers might influence creative musical development in their own classroom. This qualitative research into two secondary school communities aims to develop understandings about what really happens when students and a music teacher-researcher compose music in partnership with a tablet computer. A sociocultural definition of creativity, theories of Activity, and the musicking argument inform a new systemic framework which guides fieldwork. This framework becomes the unit of analysis from which the research questions and a multi-case, multimodal methodology emerge. The methodology developed here honours the situated nature of those meanings which emerge in each of the two school communities. Consequently, research findings are presented as two separate case reports. Five mixed-ability pairs are purposively sampled from each community to represent the broad range of musical experience present in that setting. A Video-enhanced, participant-observation method ensures that systemic, multimodal musicking behaviours are captured as they emerge overtime. Naturalistic group interviewing at the end of the project reveals how students’ broader musical cultures, interests and experiences influence their tablet-mediated classroom behaviour. Findings develop new understandings about how tablet-mediated creative musical action champions inclusive musicking (musical experience notwithstanding) and better connects the music classroom and its institutional requirements with students’ informal music-making practices. The systems of classroom Activity which emerge also compensate for those moments when the tablet attempts to overtly determine creative behaviour or conversely, does not do enough to ensure a creative outcome. In fact, all system dimensions (e.g. student partner/teacher/student/tablet) influence tablet- mediated action by feeding the system with musical and technological knowledge, which was also pedagogically conditioned. This musical, technological and pedagogical conditioning is mashed-up, influencing action just-in-time, according to cultural, local and personal need. A new method of visual charting is developed to ‘peer inside’ these classroom-situated systems. Colour-coded charts evidence how classroom musicians make use of and synthesize different system dimensions to find, focus and fix their creative musical ideas over time. There are also implications for research, policy and practice going forward. In terms of researching digitally-mediated creativity, a new social-cultural Activity framework is presented which encourages researchers to revise their definition of creativity itself. Such a definition would emphasise the role of cultural, local and personal constraint in creative musical development. With reference to classroom practice, this research discovers that when students partner with tablet computers, their own musical interests, experiences and desires are forwarded. Even though these desires become fused with institutional requirements, students take ownership of their learning and are found rightfully proud of their creative products. This naturalistic, community-driven form of tablet- mediated creative musical development encourages policy makers and teachers to reposition the music classroom: to reconnect it with the local community it serves