We analyse Ultrasound Tongue Imaging (UTI) data from five speakers, whose native languages (L1) are English (3 peakers), German (1 speaker), and Finnish (1 speaker). The data consist of single words spoken in the subjects' respective native tongues as responses to a picture naming task. The focus of this study is on automating the analysis of ultrasound recordings of tongue movements that take place after the subject is presented with a stimulus. We analyse these movements with a pixel difference method (McMillan and Corley 2010; Drake, Schaeffler, and Corley 2013a; Drake, Schaeffler, and Corley 2013b), which yields an estimate on the rate of change on a frame by frame basis. We describe typical time dependent pixel difference contours and report grand average contours for each of the speakers.caslpub3534pu

Palo, Pertti

Schaeffler, Sonja

Scobbie, James M.

Queen Margaret University eTheses

Pre-speech tongue movements recorded with ultrasound

Queen Margaret University eResearch

Pre-Speech Tongue Movements Recorded with UltrasoundPertti Palo1, Sonja Schaeffler1, James M. Scobbie11CASL Research Centre, Queen Margaret Uiniversity, Edinburgh, United Kingdomppaloqmu.ac.uk, sschaeflerqmu.ac.uk, jscobbieqmu.ac.ukAbstractWe analyse Ultrasound Tongue Imaging (UTI) data from fivespeakers, whose native languages (L1) are English (3 speak-ers), German (1 speaker), and Finnish (1 speaker). The dataconsist of single words spoken in the subjects’ respective nativetongues as responses to a picture naming task. The focus of thisstudy is on automating the analysis of ultrasound recordings oftongue movements that take place after the subject is presentedwith a stimulus. We analyse these movements with a pixel dif-ference method (McMillan and Corley 2010; Drake, Schaeffler,and Corley 2013a; Drake, Schaeffler, and Corley 2013b), whichyields an estimate on the rate of change on a frame by framebasis. We describe typical time dependent pixel difference con-tours and report grand average contours for each of the speak-ers.Keywords: Pre-speech articulation, ultrasound tongue imag-ing, pixel difference, automatic data analysis1. IntroductionStudy of speech preparation has come under increasing inter-est in recent years. Studies based on the acoustic modalityanalysing e.g. questions of phonological preparation (Rastleet al. 2000), and conversation turn taking (Heldner and Edlund2010) have been complemented with studies based on articula-tory data such as the recent analysis of organisation of speechpreparation processes by Tilsen and Goldstein 2012.The most readily accessible modality of speech production– and therefore of speech preparation – is sound. However,recording speech sounds and sounds produced during speechpreparation is an indirect way of observing the physical pro-cesses that produce them. Furthermore, the speech preparationmovements are mostly silent and thus best observed by directlyrecording the articulation itself.By varying the complexity of the tasks performed by theparticipants of a reaction time experiment light can be shed onthe organisation of cognitive speech preparation processes (seee.g. Rastle et al. 2000; Tilsen and Goldstein 2012). For thispurpose researchers are usually concerned with the onset timeof acoustic speech, onset of phonation, or the onset of phoneti-cally meaningful articulatory movements.Ultrasound Tongue Imaging (UTI) has a long history inspeech studies in general see Minifie, Kelsey, and Zagzebski1971, for one of the first studies. Its applications range fromspeech therapy see e.g. Bernhardt et al. 2005 to silent speechinterfaces Hueber et al. 2010. In comparison with the most vi-able alternative, i.e. Magnetic Resonance Imaging (MRI), UTIis relatively cheaper and simpler to use and has a far better tem-poral resolution. On the down side UTI data is fairly noisy,often contains artefacts and the imaged area is limited to thetongue and its surface.McMillan and Corley 2010; Drake, Schaeffler, and Corley2013a; Drake, Schaeffler, and Corley 2013b have automated theprocessing of UTI data by considering the difference or amountof change between consequtive ultrasound frames. This differ-ence has been defined as the Eucledian distance of two ultra-sound frames or images as they are taken to be N dimensionalvectors, with each pixel presenting a dimension.2. Materials and methodsThe experiment used the Snodgrass-Vanderwart picture namingtask (Snodgrass and Vanderwart 1980) and it had five subjects:Four females (P1, P2, P3, and G1) and one male (S1). All of thespeakers did the experiment in their native tongue – three in En-glish (participants P1, P2, and P3), one in German (participantG1) and one in Finnish (participant S1).The experiment was run with synchronised ultrasound, lipimaging and sound recording controlled with Articulate Assis-tant Advanced (AAA) software (Articulate Assistant AdvancedUser Guide: Version 2.14 2012) which will also be used forthe analysis. The participants were fitted with a purpose-builtheadset to ensure stabilisation of the ultrasound probe (Ultra-sound Stabilisation Headset Users Manual: Revision 1.4 2008).Attached to the helmet was a small Audio Technica AT803bmicrophone for high-quality acoustic recordings. Ultrasoundrecordings were obtained at a frame rate of 201 frames per sec-ond.The recording was initiated 1.5 seconds before the subjectswere shown the stimulus on a computer screen, thus capturingany movements related to speech preparation as well as makingit possible to spot cases where the subject was moving alreadybefore the onset of the stimulus. These cases as well as oneswhere the subject trouble naming the picture were excludedfrom further analysis resulting still in well over 200 analysedtokens per speaker.2.1. Pixel differencesWe analyse the samples by automatically calculating theamount of change in the UTI data as a function of time. Weuse pixel difference (McMillan and Corley 2010; Drake, Scha-effler, and Corley 2013a; Drake, Schaeffler, and Corley 2013b)as the change metric and evaluate it over each recording com-paring raw ultrasound frames in sequence.UTI commonly uses ultrasound probes which produce afan shaepd image of the tongue. Ordinary or interpolated ul-trasound data refers to the form commonly displayed by ultra-sound imaging systems as seen in Fig. 1 a). The fan image ofthe ordinary ultrasound data is produced by linear interpolationbetween the actual raw datapoints produced by the ultrasoundsystem as it images the tissues. The raw datapoints are dis-tributed along radial scanlines with the number of scanlines andthe number of data points imaged along each scanline depend-412 px38 pxa) b)Figure 1: a) 1st ultrasound frame from participant P2 naming apicture of a lemon. b) Raw (uninterpolated) version of the sameultrasound frame as in a). The participant is facing right. Redarrow points to the upper surface of the tip of the tongue.ing on the setup of the ultrasound system.Fig. 1 b) shows a raw ultrasound frame. In our experimentseach frame has 38 scanlines (x dimension of the image) and412 datapoints of pixels along each scanline (y dimension ofthe image).To calculate the pixel difference between two UTI frameswe interpret each raw frame as aN = nx×ny dimensional vec-tor. The pixel difference d1 between consecutive UTI frames isthen defined as the Eucledian distance between the two framesimk and imk+1 with indeces i and j iterating over the pixels inx and y direction:d1(k) =√√√√ nx∑i=1ny∑j=1(imk(i, j)− imk+1(i, j))2 (1)for k = {1, 2, . . . nframes − 1}. The difference can becalculated as readily for images further removed and are definedasdL(k) =√√√√ nx∑i=1ny∑j=1(imk(i, j)− imk+L(i, j))2 (2)for k = {1, 2, . . . nframes − L}. In this paper we will usethe differences d1 and d3. The time stamp tdL(k) correspondingto a single difference value dL(k) is defined as the average ofthe time stamps of the corresponding UTI frames:tdL(k) =12(timk + timk+L), (3)where the time stamp of image k is the time its acquisition ends.3. Results[Explain the pictures here:]Fig. 2 shows a typical clear production. The participanthas held still before starting linguistic articulation. From com-parison with the corresponding waveform at the bottom of thefigure, it can be seen that the articulation starts a good whilebefore acoustic onset.05001000150020002500Euclidean pixel distanceP2 Lemon  Step = 1Step = 3Stimulus onset0 0.5 1 1.5 2 2.5 3 3.5 4−0.100.1Time (s)WaveFigure 2: A clear sample: P2 naming a picture of a lemon.05001000150020002500Euclidean pixel distanceP2 Chisel  Step = 1Step = 3Stimulus onset0 0.5 1 1.5 2 2.5 3 3.5 4−0.200.2Time (s)WaveFigure 3: A hesitation: P2 naming a picture of a chisel.05001000150020002500Euclidean pixel distanceP1 Paintbrush  Step = 1Step = 3Stimulus onset0 0.5 1 1.5 2 2.5 3 3.5 4−0.100.1Time (s)WaveFigure 4: Chaos: P1 naming a picture of a paintbrush.Fig. 3 shows an example of a hesitation. The participantmoves their tongue as if they were going to speak, but returnsto rest and finally speaks later.Fig. 3 shows a chaotic example. The participant is movingalready at the time the recording starts and continues to movethrough out the whole recording.Then explain the grand averages and thus figures 5, 6, 7, 8,and 9.0 0.5 1 1.5 2 2.5 305001000150020002500P1, n = 148  Step = 1Step = 3Stimulus onsetFigure 5: Grand averages of all participant P1’s UTI record-ings longer than 2 seconds.0 0.5 1 1.5 2 2.5 305001000150020002500P2, n = 264  Step = 1Step = 3Stimulus onsetFigure 6: Grand averages of all participant P2’s UTI record-ings longer than 2 seconds.0 0.5 1 1.5 2 2.5 305001000150020002500P3, n = 219  Step = 1Step = 3Stimulus onsetFigure 7: Grand averages of all participant P3’s UTI record-ings longer than 2 seconds.4. Discussion and conclusionPoints:• Different types of productions are present in the data.Some are easier to analyse than others.• Individual differences are evident in the averages. Forexample, S1 is slower to respond than the other partic-ipants and P2 has more variation (i.e. probably moremovement) in the time leading up to linguistic articula-tion. Particularly here the use of d1 in conjunction with0 0.5 1 1.5 2 2.5 305001000150020002500G1, n = 260  Step = 1Step = 3Stimulus onsetFigure 8: Grand averages of all participant G1’s UTI record-ings longer than 2 seconds.0 0.5 1 1.5 2 2.5 305001000150020002500S1, n = 216  Step = 1Step = 3Stimulus onsetFigure 9: Grand averages of all participant S1’s UTI recordingslonger than 2 seconds.d3 or some other higher order pixel difference is essen-tial.• The use of d3 probably needs more explanation. Couldbe done by pointing to how it enhances the visibility ofhesitation in the chisel sample.• The timeline hypothesised in the original abstract, doesnot correspond with the majority of the data.• A word or two about noise floor would be good too.5. AcknowledgementsThe authors wish to thank Steve Cowen for assistance with theultrasound recordings and Professor Alan Wrench for adviceand help on extracting the raw ultrasound data from AAA andsubsequent post-processing of the data.6. ReferencesArticulate Assistant Advanced User Guide: Version 2.14 (2012). Edin-burgh, UK: Articulate Instruments Ltd.Ultrasound Stabilisation Headset Users Manual: Revision 1.4 (2008).Edinburgh, UK: Articulate Instruments Ltd.Bernhardt, B., B. Gick, P. Bacsfalvi, and M. Adler-Bock (2005). “Ultra-sound in speech therapy with adolescents and adults”. In: ClinicalLinguistics & Phonetics 19.6-7, pp. 605–617.Drake, E., S. Schaeffler, and M. Corley (2013a). “ARTICULATORYEVIDENCE FOR THE INVOLVEMENT OF THE SPEECHPRODUCTION SYSTEM IN THE GENERATION OF PREDIC-TIONS DURING COMPREHENSION”. In: Architectures andMechanisms for Language Processing (AMLaP). Marseille.— (2013b). “Does prediction in comprehension involve articulation?Evidence from speech imaging”. In: 11th Symposium of Psycholin-guistics (SCOPE). Tenerife.Heldner, M. and J. Edlund (2010). “Pauses, gaps and overlaps in conver-sations”. In: Journal of Phonetics 38.4, pp. 555–568. DOI: http://dx.doi.org/10.1016/j.wocn.2010.08.002.URL: http://www.sciencedirect.com/science/article/pii/S0095447010000628.Hueber, Thomas, Elie-Laurent Benaroya, Gérard Chollet, Bruce Denby,Gérard Dreyfus, and Maureen Stone (2010). “Development of asilent speech interface driven by ultrasound and optical images ofthe tongue and lips”. In: Speech Communication 52.4, pp. 288–300. DOI: http://dx.doi.org/10.1016/j.specom.2009.11.004. URL: http://www.sciencedirect.com/science/article/pii/S0167639309001733.McMillan, C. T. and M. Corley (2010). “Cascading influences on theproduction of speech: Evidence from articulation”. In: Cognition117.3, pp. 243–260.Minifie, F., C. Kelsey, and J. Zagzebski (1971). “Ultrasonic Scans ofthe Dorsal Surface of the Tongue”. In: Journal of the AcousticalSociety of America 49.6, pp. 1857–1860.Rastle, K., J. Harrington, M. Coltheart, and S. Palethorpe (2000).“Reading Aloud Begins When the Computation of Phonology IsComplete”. In: Journal of Experimental Psychology: Human Per-ception and Performance 26.3, pp. 1178–1191.Snodgrass, J. G. and M. Vanderwart (1980). “A Standardized Set of 260Pictures: Norms for Name Agreement, Image Agreement, Famil-iarity, and Visual Complexity”. In: Journal of Experimental Psy-chology: Human Learning and Memory 6.2, pp. 174–215.Tilsen, S. and L. Goldstein (2012). “Articulatory gestures are individu-ally selected in production”. In: Journal of Phonetics 40, pp. 764–779.7. Stuff from the template. Will removethese on MondayLATEX users: the document should compile successfully after• an initial call to pdflatex *.tex• followed by a call to biber *.bcf• followed by two more calls to pdflatex *.tex8. Page layout and styleAll papers must be submitted in compliance with the provided template.Check details of your final PDF submission against the template exam-ple file.8.1. Basic layout features• Two columns are used except for the title section.• Left margin is 20 mm.• Column width is 80 mm.• Spacing between columns is 10 mm.• Top margin 25 mm.• Text height (without headers and footers) is maximum 235 mm.• Headers and footers must be left empty.8.1.1. HeadingsSection headings are in boldface with the first word capitalized and therest of the heading in lower case. Sub-headings appear like major head-ings. Sub-sub-headings appear like sub-headings, except they are initalics and not boldface. No more than 3 levels of headings should beused.8.2. Text fontTimes or Times Roman font is used for the main text. Font size in themain text must be 9 points, and in the References section 8 points. Otherfont types may be used if needed for special purposes.IMPORTANT: All fonts must be embedded when compiling the finalPDF! Failure to embed fonts will result in missing text in the final pub-lication.LATEX users: font types, font face and type size should be pre-defined in either the document body or the issp2014.sty style file. Au-thors must not use Type 3 (bitmap) fonts.8.3. Figures and TablesFigures should preferably be line drawings. If they contain gray levelsor colors, they should be checked to print well on a high-quality non-color laser printer. Please use only solid fill colors.Table 1: This is an example of a table.Ratio Decibels1/1 02/1 ≈ 63.16 1010/1 201/10 -20100/1 401000/1 608.4. EquationsEquations should be placed on separate lines and numbered. Examplesof equations are given below. Particularly,x(t) = s(fω(t)) (4)where fω(t) is a special warping functionfω(t) =12pij∮Cν−1kdν(1− βν−1)(ν−1 − β) (5)A residue theorem states that∮CF (z)dz = 2pij∑kRes[F (z), pk] (6)Applying (6) to (4), it is straightforward to see that1 + 1 = pi (7)Finally we have proven the secret theorem of all speech sciences.No more math is needed to show how useful the result is!8.5. Submitted filesAuthors are requested to submit PDF files of their manuscripts. Toolsare available online to create the PDF such as:• http://www.pdfforge.org/products/pdfcreator• pdflatexThe PDF file should comply with the following requirements: (a) theremust be no PASSWORD protection on the PDF file at all; (b) all fontsmust be embedded; and (c) the file must be text searchable (do CTRL-Fand try to find a common word such as ’the’). The proceeding editorswill contact authors of non-complying files to obtain a replacement. Inorder not to endanger the preparation of the proceedings, papers forwhich a replacement is not provided timely will be withdrawn.8.6. Page numberingPage numbers will be added later to the document electronically. Don’tmake any footers or headers!8.7. AbstractThe total length of the abstract is limited to 1000 characters. The ab-stract included in your paper and the one you enter during web-basedsubmission must be identical. Avoid non-ASCII characters or symbolsas they may not display correctly in the abstract book.

We analyse Ultrasound Tongue Imaging (UTI) data from five speakers, whose native languages (L1) are English (3 peakers), German (1 speaker), and Finnish (1 speaker). The data consist of single words spoken in the subjects’ respective native tongues as responses to a picture naming task. The focus of this study is on automating the analysis of ultrasound recordings of tongue movements that take place after the subject is presented with a stimulus. We analyse these movements with a pixel difference method (McMillan and Corley 2010; Drake, Schaeffler, and Corley 2013a; Drake, Schaeffler, and Corley 2013b), which yields an estimate on the rate of change on a frame by frame basis. We describe typical time dependent pixel difference contours and report grand average contours for each of the speakers

Scobbie, James M

Name not available

https://eresearch.qmu.ac.uk/bitstream/20.500.12289/3534/1/eResearch%25203534.pdf

Pre-speech tongue movements recorded with ultrasound

Abstract

Similar works

Full text

Available Versions

Queen Margaret University eTheses

Queen Margaret University eResearch

Name not available