Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications

Lücking A, Bergmann K, Hahn F, Kopp S, Rieser H. Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications. Journal on Multimodal User Interfaces. 2013;7(1-2):5-18.Communicating face-to-face, interlocutors frequently produce multimodal meaning packages consisting of speech and accompanying gestures. We discuss a systematically annotated speech and gesture corpus consisting of 25 route-and-landmark-description dialogues, the Bielefeld Speech and Gesture Alignment corpus (SaGA), collected in experimental face-to-face settings. We first describe the primary and secondary data of the corpus and its reliability assessment. Then we go into some of the projects carried out using SaGA demonstrating the wide range of its usability: on the empirical side, there is work on gesture typology, individual and contextual parameters influencing gesture production and gestures’ functions for dialogue structure. Speech-gesture interfaces have been established extending unification-based grammars. In addition, the development of a computational model of speech-gesture alignment and its implementation constitutes a research line we focus on

Encephalopathy and polyneuropathy caused by organic solvents - review of recent literature and patient examination at the Institute of Occuptaional Medicine, University of Erlangen

1.1. Hintergrund und Ziele Im März 2005 wurde das seit Dezember 1997 gültige „Merkblatt für die ärztliche Untersuchung zur BK Nr. 1317: Polyneuropathie oder Enzephalopathie durch organische Lösungsmittel oder deren Gemische“ (31) durch eine Neufassung (32) ersetzt. Ziel dieser Dissertation ist es, Patientendaten des Erlanger Institutes und Poliklinik für Arbeits-, Sozial- und Umweltmedizin IPASUM zur Thematik lösungsmitteltoxischer Erkrankungen darzustellen, aktuelle Literatur auszuwerten und die Unterschiede zwischen altem und neuem Merkblatt kritisch zu bewerten. Abschließend erfolgt eine gemeinsame Diskussion der drei vorgenannten Themen. 1.2. Methoden Die verwendete Literatur wurde anhand der medizinischen Datenbank PubMed ermittelt bzw. stand teilweise in der Bibliothek des IPASUM zur Verfügung. Gegenstand der Suche waren Artikel zum Thema toxischer Gehirn- und Nervenschäden, die zwischen 1990 und 2006 veröffentlicht wurden und in englischer oder deutscher Sprache zur Verfügung standen. Die Daten des IPASUM stammen von Patienten, die sich im Zeitraum von 2000 bis 2005 am Institut vorstellten. Das Heraussuchen relevanter Akten erfolgte anhand der Aufnahmebücher des Institutes. Die Datenerfassung erfolgte mittels einer Microsoft EXCEL™-Tabelle (Microsoft Office Professional, 1997), die zur statistischen Auswertung in SPSS 12 übertragen wurde. 1.3. Ergebnisse und Beobachtungen Von den 113 ausgewerteten Fällen des IPASUM wurde bei 31 Fällen eine Berufskrankheit zur Anerkennung empfohlen, die in 22 Fällen von den Berufsgenossenschaften anerkannt wurden. Es ergab sich unter anderem eine Abhängigkeit des Erkrankungsrisikos von der Expositionsdauer, dem Lebensalter und der Expositionshöhe. Die Literaturrecherche führte zu ähnlichen Erkenntnissen wie die Auswertung der eigenen Daten. Verwertbare Schlussfolgerungen zur Frage der Latenz zwischen Exposition und Erkrankungsbeginn ließen sich weder aus der Literatur noch aus den Daten des IPASUM ziehen. Es ergaben sich Hinweise, dass eine vollständige Reversibilität der Erkrankungen in vielen Fällen nicht zu erreichen ist, jedoch v. a. in schweren Stadien und hier v. a. dann, wenn keine Expositionskarenz eingehalten wurde. 1.4. Praktische Schlussfolgerungen Die Neufassung des Merkblattes für die ärztliche Untersuchung zur die BK Nr. 1317 entschärft in ihren Formulierungen die Abgrenzungsmöglichkeiten zwischen toxischer Enzephalopathie und Polyneuropathie, ohne dass in der Literatur die entsprechenden Hinweise zweifelsfrei aufgedeckt wurden. Die Anerkennung einer beruflich bedingten toxischen Enzephalopathie oder Polyneuropathie wurde dadurch für den betroffenen Arbeitnehmer einfacher, die medizinische Bewertung jedoch für den beurteilenden Arzt unsicherer.1.1. Background: In March 2005 the bulletin (31) for the examination of patients suspected suffering from an occupational disease caused by organic solvents (BK 1317 according to list of occupational diseases in Germany) was replaced by a new version (32). It is the intention of this dissertation to analyse patients examined at the institute for occupational medicine, University of Erlangen, to review recent literature and to assess the changes between the old and the new bulletin. Finally these three items will be discussed together. 1.2. Methods: The literature reviewed was raised via PubMed or was available in the institute’s library. Objects of the search were articles concerning toxic encephalopathy or neuropathy that were published between 1990 und 2006 in English or German language. The analysed data were collected from patients examined between 2000 and 2005. Finding the relevant files was realised by searching the “check-in-books” of the institute. The Information was stored in a Microsoft EXCEL™-chart (Microsoft Office Professional, 1997) and then transferred to SPSS 12 for statistical evaluation. 1.3. Results: Out of the 113 evaluated cases 31 were proposed to be approved as an occupational disease. 22 of these were approved by the particular Accident Prevention & Insurance Association. Risk of neurological disease depended on length of exposition, age and extent of exposure. The review of recent literature on neurological diseases due to organic solvent exposure led to similar results as the analysis of our institute’s data. Neither literature nor our data allowed useful conclusions about latency between exposure and beginning of symptoms. We found, that complete reversibility of symptoms often could not be reached, especially in higher grades of disease and if exposure did not seize after a neurological deficit was diagnosed. 1.4. Conclusions: The amended version of the bulletin on occupational disease BK 1317 mitigates the discrimination of toxic encephalopathy and polyneuropathy although literature did not surly corroborate this. This made it easier to approve the occupational aetiology of a neurologic disease, on the other hand the bulletin does not give reliable hints what discriminates a toxic from a non-toxic neuropathy

Speech-Gesture Alignment

Kopp S, Rieser H, Wachsmuth I, Bergmann K, Lücking A. Speech-Gesture Alignment. In: 3rd Conference of the International Society for Gesture Studies. 2007

Deictic object reference in task-oriented dialogue

Abstract. This chapter presents a collaborative approach towards a detailed understanding of the usage of pointing gestures accompanying referring expressions. This effort is undertaken in the context of human-machine interaction integrating empirical studies, theory of grammar and logics, and simulation techniques. In particular, we take steps to classify the role of pointing in deictic expressions and to model the focussed area of pointing gestures, the so-called pointing cone. This pointing cone serves as a central concept in a formal account of multi-modal integration at the linguistic speech-gesture interface as well as in computational models of processing multi-modal deictic expressions. 1

The Bielefeld Speech and Gesture Alignment Corpus (SaGA)

Lücking A, Bergmann K, Hahn F, Kopp S, Rieser H. The Bielefeld Speech and Gesture Alignment Corpus (SaGA). In: Kipp M, Martin J-P, Paggio P, Heylen D, eds. LREC 2010 Workshop: Multimodal Corpora–Advances in Capturing, Coding and Analyzing Multimodality. 2010: 92-98.People communicate multimodally. Most prominently, they co-produce speech and gesture. How do they do that? Studying the interplay of both modalities has to be informed by empirically observed communication behavior. We present a corpus built of speech and gesture data gained in a controlled study. We describe 1) the setting underlying the data; 2) annotation of the data; 3) reliability evalution methods and results; and 4) applications of the corpus in the research domain of speech and gesture alignment

Optimized Flat-Detector CT in Stroke Imaging: Ready for First-Line Use?

Background: Using flat-detector CT (FD-CT) for stroke imaging has the advantage that both diagnostic imaging and endovascular therapy can be performed directly within the Angio Suite without any patient transfer and time delay. Thus, stroke management could be speeded up significantly, and patient outcome might be improved. But as precondition for using FD-CT as primary imaging modality, a reliable exclusion of intracranial hemorrhage (ICH) has to be possible. This study aimed to investigate whether optimized native FD-CT, using a newly implemented reconstruction algorithm, may reliably detect ICH in stroke patients. Additionally, the potential to identify ischemic changes was evaluated. Methods: Cranial FD-CT scans were obtained in 102 patients presenting with acute ischemic stroke (n = 32), ICH (n = 45) or transient ischemic attack (n = 25). All scans were reconstructed with a newly implemented half-scan cone-beam algorithm. Two experienced neuroradiologists, unaware of clinical findings, evaluated independently the FD-CTs screening for hemorrhage or ischemic signs. The findings were correlated to CT, and rater and inter-rater agreement was assessed. Results: FD-CT demonstrated high sensitivity (95-100%) and specificity (100%) in detecting intracerebral and intraventricular hemorrhage (IVH). Overall, interobserver agreement (κ = 0.92) was almost perfect and rater agreement to CT highly significant (r = 0.81). One infratentorial ICH and 10 or 11 of 22 subarachnoid hemorrhages (SAHs) were missed of whom 7 were perimesencephalic. The sensitivity for detecting acute ischemic signs was poor in blinded readings (0 or 25%, respectively). Conclusions: Optimized FD-CT, using a newly implemented reconstruction algorithm, turned out as a reliable tool for detecting supratentorial ICH and IVH. However, detection of infratentorial ICH and perimesencephalic SAH is limited. The potential of FD-CT in detecting ischemic changes is poor in blinded readings. Thus, plain FD-CT seems insufficient as a standalone modality in acute stroke, but within a multimodal imaging approach primarily using the FD technology, native FD-CT seems capable to exclude reliably supratentorial hemorrhage. Currently, FD-CT imaging seems not yet ready for wide adoption, replacing regular CT, and should be reserved for selected patients. Furthermore, prospective evaluations are necessary to validate this approach in the clinical setting

Measuring and Reconstructing Pointing in Visual Contexts

Kranstedt A, Lücking A, Pfeiffer T, Rieser H, Staudacher M. Measuring and Reconstructing Pointing in Visual Contexts. In: Schlangen D, Fernández R, eds. Proceedings of the brandial 2006 - The 10th Workshop on the Semantics and Pragmatics of Dialogue. Potsdam: Universitätsverlag Potsdam; 2006: 82-89.We describe an experiment to gather original data on geometrical aspects of pointing. In particular, we are focusing upon the concept of the pointing cone, a geometrical model of a pointing’s extension. In our setting we employed methodological and technical procedures of a new type to integrate data from annotations as well as from tracker recordings. We combined exact information on position and orientation with rater’s classifications. Our first results seem to challenge classical linguistic and philosophical theories of demonstration in that they advise to separate pointings from reference

Deictic object reference in task-oriented dialogue

Kranstedt A, Lücking A, Pfeiffer T, Rieser H, Wachsmuth I. Deictic object reference in task-oriented dialogue. In: Rickheit G, Wachsmuth I, eds. Situated Communication. Berlin: Mouton de Gruyter; 2006: 155-208.This chapter presents an original approach towards a detailed understanding of the usage of pointing gestures accompanying referring expressions. This effort is undertaken in the context of human-machine interaction integrating empirical studies, theory of grammar and logics, and simulation techniques. In particular, we take steps to classify the role of pointing in deictic expressions and to model the focussed area of pointing gestures, the so-called pointing cone. This pointing cone serves as a central concept in a formal account of multi-modal integration at the linguistic speech-gesture interface as well as in a computational model of processing multi-modal deictic expressions