A review of data collection practices using electromagnetic articulography

This paper reviews data collection practices in electromagnetic articulography (EMA) studies, with a focus on sensor placement. It consists of three parts: in the first part, we introduce electromagnetic articulography as a method. In the second part, we focus on existing data collection practices. Our overview is based on a literature review of 905 publications from a large variety of journals and conferences, identified through a systematic keyword search in Google Scholar. The review shows that experimental designs vary greatly, which in turn may limit researchers' ability to compare results across studies. In the third part of this paper we describe an EMA data collection procedure which includes an articulatory-driven strategy for determining where to position sensors on the tongue without causing discomfort to the participant. We also evaluate three approaches for preparing (NDI Wave) EMA sensors reported in the literature with respect to the duration the sensors remain attached to the tongue: 1) attaching out-of-the-box sensors, 2) attaching sensors coated in latex, and 3) attaching sensors coated in latex with an additional latex flap. Results indicate no clear general effect of sensor preparation type on adhesion duration. A subsequent exploratory analysis reveals that sensors with the additional flap tend to adhere for shorter times than the other two types, but that this pattern is inverted for the most posterior tongue sensor

The Phonetics of Speech Production and Medical Research

The production of speech requires the interplay of a number of cognitive and motoric activities, which make it an interesting object of study from both a linguistic and a medical point of view. In this paper, we discuss, first, the features and domain of application of the most used technologies in linguistic research on speech production, focusing on those that have been applied to medicine. Second, we offer an insight into the main results that have been obtained so far in studying dysarthria in Italian Parkinson's Disease, as an example of the interdisciplinary, experimental research at the border between linguistics and medicine

Accuracy assessment of two electromagnetic articulographs:NDI Wave and NDI Vox

Purpose This study compares two electromagnetic articulographs manufactured by Northern Digital, Inc.: the NDI Wave System (from 2008) and the NDI Vox-EMA System (from 2020). Method Four experiments were completed: (a) comparison of statically positioned sensors, (b) tracking dynamic movements of sensors manipulated using a motor-driven LEGO apparatus, (c) tracking small and large movements of sensors mounted in a rigid bar manipulated by hand, and (d) tracking movements of sensors rotated on a circular disc. We assessed spatial variability for statically positioned sensors, variability in the transduced Euclidean distances between sensor pairs, and missing data rates. For sensors tracking circular movements, we compared the fit between fitted ideal circles and actual trajectories. Results The average sensor pair tracking error (i.e., the standard deviation of the Euclidean distances) was 1.37 mm for the WAVE and 0.12 mm for the VOX during automated trials at the fastest speed, and 0.35 mm for the WAVE and 0.14 mm for the VOX during the tracking of large manual movements. The average standard deviation of the fitted circle radii charted by manual circular disc movements was 0.72 mm for the WAVE sensors and 0.14 mm for the VOX sensors. There was no significant difference between the WAVE and the VOX in the number of missing frames. Conclusions In general, the VOX system significantly outperformed the WAVE on measures of both static precision and dynamic accuracy (automated and manual). For both systems, positional precision and spatial variability were influenced by the sensors' position relative to the field generator unit (worse when further away)

Learning the Mapping Function from Voltage Amplitudes to Sensor Positions in 3D-EMA Using Deep Neural Networks

The first generation of three-dimensional Electromagnetic Articulography devices (Carstens AG500) suffered from occasional critical tracking failures. Although now superseded by new devices, the AG500 is still in use in many speech labs and many valuable data sets exist. In this study we investigate whether deep neural networks (DNNs) can learn the mapping function from raw voltage amplitudes to sensor positions based on a comprehensive movement data set. This is compared to arriving sample by sample at individual position values via direct optimisation as used in previous methods. We found that with appropriate hyperparameter settings a DNN was able to approximate the mapping function with good accuracy, leading to a smaller error than the previous methods, but that the DNN-based approach was not able to solve the tracking problem completely

Comparison of speech quality with and without sensors in electromagnetic articulograph AG 501 recording

Abstract In the recordings using electromagnetic articulograph AG 501, sensors are glued to subject's articulators such as jaw, lips and tongue and both speech and articulatory movements are simultaneously recorded. In this work, we study the effect of the presence of the sensors on the quality of speech spoken by the subject. This is done by recording when a subject speaks a set of 19 VCV stimuli while sensors are attached to subject's articulators. For comparison we also record the same set of stimuli spoken by the same subject but with no sensors attached to subject's articulators. Both subjective and objective comparisons are made on the recorded stimuli in these two settings. Subjective evaluation is carried out using 16 evaluators. Listening experiments with recordings from five subjects show that the recordings with sensors attached are significantly different from those without sensors attached in terms of human recognition score as well as on a perceptual difference measure. This is also supported in the objective comparison which computes dissimilarity measure using the spectral shape information

Is markerless acquisition of speech production accurate ?

International audienceIn this study, the precision of markerless acquisition techniques have been assessed when used to acquire articulatory data for speech production studies. Two different markerless systems have been evaluated and compared to a marker-based one. The main finding is that both markerless systems provide reasonable result during normal speech and the quality is uneven during fast articulated speech. The quality of the data is dependent on the temporal resolution of the markerless system

Multi‐speaker experimental designs: Methodological considerations

Research on language use has become increasingly interested in the multimodal and interactional aspects of language – theoretical models of dialogue, such as the Communication Accommodation Theory and the Interactive Alignment Model are examples of this. In addition, researchers have started to give more consideration to the relationship between physiological processes and language use. This article aims to contribute to the advancement in studies of physiological and/or multimodal language use in naturalistic settings. It does so by providing methodological recommendations for such multi-speaker experimental designs. It covers the topics of (a) speaker preparation and logistics, (b) experimental tasks and (c) data synchronisation and post-processing. The types of data that will be considered in further detail include audio and video, electroencephalography, respiratory data and electromagnetic articulography. This overview with recommendations is based on the answers to a questionnaire that was sent amongst the members of the Horizon 2020 research network ‘Conversational Brains’, several researchers in the field and interviews with three additional experts.H2020 Marie Skłodowska‐Curie Actions http://dx.doi.org/10.13039/100010665Peer Reviewe

Registration and statistical analysis of the tongue shape during speech production

This thesis analyzes the human tongue shape during speech production. First, a semi-supervised approach is derived for estimating the tongue shape from volumetric magnetic resonance imaging data of the human vocal tract. Results of this extraction are used to derive parametric tongue models. Next, a framework is presented for registering sparse motion capture data of the tongue by means of such a model. This method allows to generate full three-dimensional animations of the tongue. Finally, a multimodal and statistical text-to-speech system is developed that is able to synthesize audio and synchronized tongue motion from text.Diese Dissertation beschäftigt sich mit der Analyse der menschlichen Zungenform während der Sprachproduktion. Zunächst wird ein semi-überwachtes Verfahren vorgestellt, mit dessen Hilfe sich Zungenformen von volumetrischen Magnetresonanztomographie- Aufnahmen des menschlichen Vokaltrakts schätzen lassen. Die Ergebnisse dieses Extraktionsverfahrens werden genutzt, um ein parametrisches Zungenmodell zu konstruieren. Danach wird eine Methode hergeleitet, die ein solches Modell nutzt, um spärliche Bewegungsaufnahmen der Zunge zu registrieren. Dieser Ansatz erlaubt es, dreidimensionale Animationen der Zunge zu erstellen. Zuletzt wird ein multimodales und statistisches Text-to-Speech-System entwickelt, das in der Lage ist, Audio und die dazu synchrone Zungenbewegung zu synthetisieren.German Research Foundatio

Analysis of Interference between Electromagnetic Articulography and Electroglottograph Systems

Electromagnetic Articulography (EMA) has become an integral tool for researchers and clinicians who seek to characterize speech kinematics. The position and orientation of the articulators – which include the jaw, lips, and tongue – are recorded by attaching sensors to the articulators and tracking the movement of the sensors through an electromagnetic field. This has been used by researchers and clinicians to better understand dysarthria and synthesize speech, among other applications. Another speech tool, electroglottography (EGG), is used to analyze the movement of the vocal folds during speech production. This is achieved by measuring the time variation of the contact of the vocal folds and analyzing it with regards to the speech produced. Clinically, EGG is used to identify voice abnormalities, including those without visual or acoustic abnormalities. These systems are not used concurrently because of the electromagnetic field used with the EMA system; NDI and Carstens affirm that metal should be kept out of the field during EMA use. Concurrent use of these systems would lead to simultaneous measurements of the laryngeal and upper airway-articulatory abilities, which could increase understanding of motor speech issues. Parameters derived from the EGG signal could also be incorporated with articulatory parameters to improve synthesized voice quality and synthesize a more realistic voice. The objective of this research is to investigate whether the interference present between the EMA and EGG systems is significant and, if so, to characterize it so the systems can be used simultaneously. Analysis of the interference was obtained through several data collections. The first assessed the degree of interference when the EMA sensors were stationary. The second data set was collected using a model that maintained sensor orientation while the sensors were at a nonzero speed. The final data set was obtained with a model that in which the sensors were in a fixed position while changing the orientation of the sensors, with minimal translational velocity. Sources of interference that were present included the EGG system and orthodontia. The resulting data led to the conclusion that the presence of the EGG or the orthodontic appliances does not cause significant interference