This study aims to connect data from ocular movements and reading aloud speech to syntactic and discursive properties of texts, in order to understand integrative cognitive processes during reading for understanding and to identify prosodic and eye movements’ indicators of reading fluency. Assuming that
in reading aloud there is a close interaction between syntax structure and speech prosody, we collected eye movements and reading speech data from 17 native EP speakers. Eye movements and reading speech produced simultaneously were analyzed and our results show that eyes and voice are both responsive to text complexity and to syntactic and discursive critical loci, as key points of information integration.info:eu-repo/semantics/publishedVersio

Costa, Armanda

Falé, Isabel

Luegi, Paula

Repositório Aberto da Universidade Aberta

Av. Professor Gama Pinto, 2, 1649-003 Lisboa, Portugal | Tel.: +351 217 904 700 | Fax: +351 217 965 622 | URL: www.clul.ul.pt Reading aloud: eye movements and prosody Isabel Falé1,2 Armanda Costa1 Paula Luegi1  ifale@campus.ul.pt; armandacosta@letras.ulisboa.pt; paulaluegi@letras.ulisboa.pt 1Universidade de Lisboa – CLUL | 2Universidade Aberta Speech Prosody 2016, Boston, May 31 – June 03 The reader and the reading processes have been studied for years in a psycholinguistic perspective and a lot is known about what is involved (Perfetti, 1999; Vellutino, 2006). However, there is still much to learn considering interrelated processes and behaviors, which have been separately studied. Silent reading and reading aloud inform us in different ways about linguistic processing, comprehension and fluency. Tools and measures to deal with underlying cognitive processes are specific for each modality. Eye movements provide measures that can indicate some linguistic properties of the text (Rayner et al. 2005): - First fixation in a word varies according to orthographic, phonological and morphological properties, reflecting specific processes to visual word recognition. - First pass, which includes FF and other fixations before moving the eyes to right or left regions, could tap the processes involved in lexical access of less frequent or longer words, required for their integration in a larger meaning or structural unit. - Total time of word fixation, including all fixations in a word, must reflect word integration in a semantic-discursive mental representation, and can be taken as a window for wrap-up effects. Reading speech, or speech produced in reading aloud, is strongly constrained by the communicative context, reader experience and text properties, such as vocabulary, syntactic structures, punctuation and layout. In reading speech, prosody is an important window to study phrasing processes and to relate them with syntactic parsing. To identify prosodic boundaries as important loci to study linguistic processing we consider two acoustic parameters (Gussenhoven & Rietveld 1992): - Stressed vowel length, as a marker indicating the proximity of a high level prosodic boundary: the longer the time vowel duration is, the higher the boundary.  - Fundamental frequency of the stressed vowel, as an indicator of the syntactic position of the word: the more the word is embedded in the phrase or sentence the higher frequency on the stressed vowel. 1. Eye movements and reading speech Aims • Identify processes of linguistic information integration undergoing in oral reading for understanding. • Verify the effect of linguistic and discursive properties of texts on language processing, considering their complexity level. • Find related measures of prosody in reading aloud and in eye movements in order to contribute for reading fluency assessment and teaching. Participants  17 European Portuguese native female speakers, students, proficient readers. Procedure  Eye movements were recorded with a SMI IVIEW X™ HI-SPEED system, at a 1250Hz speed, and sound was recorded with a Logitech® Webcam Pro 9000. Stimuli were divided, for presentation, into two blocks of text, font in size 22, Courier New, with two paragraphs spacing between rows, in a 17-inch screen. After reading each text participants answered a multiple-choice questionnaire, thus ensuring a reading comprehension task. Stimuli Two informative texts, similar in word number (200), syntactic and information structure and layout: CO: An easy text about a Lisbon neighborhood ITA: A hard text about thermo-acoustic proofing Thematic dimension as well as some word properties crucially contribute to differentiate the level of complexity of texts.  Word properties in each text were controlled considering: number of syllables, type of syllables and word frequency, with tools for automatic linguistic processing – FreP and CRPC (Graph 1, 2 and 3). Graph 2: Syllable types 3. Experiment 4. Results Results for Syntactic and Discursive positions Eye movements • Position effect in FP is clear when comparing SN and DB (p<0.001), SB and DB (p<0.001), with longer reading times for DB in both cases. A Position effect is consistently registered in TTF: ​SN higher than SB (p=0.03) and lower than DB (p<0.001); SB lower than DB (p<0.001) ​ (Graph 4). Reading speech • There is no Position effect in SVL, however we find an interaction between Text and Position in SVL (p<0.001).   • Considering F0 declination in European Portuguese, an expected Position effect in F0 is verified, being SN higher than SB (p<0.001) and than DB (p<0.001), and SB higher than DB (p<0.001) (Graph 6). The different complexity of the two texts and its impact on linguistic processing was confirmed. The hypothesis of readers’ responsiveness to linguistic and discursive complexity is sustained by evidences coming from speech and eyes. Discursive Boundary is the locus where processes related with structural building ending and prosodic phrasing occur: the eyes take longer to complete the structure and possibly resume prior information, while speech signals the most embedded nominal head of an NP with higher F0, which declines as the information is being integrated, with the lowest values on the periphery of the sentence. Lengthening of the stressed vowel and first fixation seem to be responsive to lexical properties. Fundamental frequency and First Pass seem to reveal structural building, integration of information and possibly wrap-up effects. References Graph 3: Three syllable words frequency Results for text Eye movements • Results show a strong effect of Text, with significant differences between the 2 texts in all dependent variables (FF p=0.02; FP p=0.001; TTF p=0.001), with lower times in CO than in ITA (Graph 4). Reading speech • We find a Text effect in SVL (Graph 5) with lower values in CO than in ITA (p<0.001).  5. Conclusions Gussenhoven, C. & T. Rietveld, “Intonation contours, prosodic structure, and preboundary lengthening,” Journal of Phonetics, vol. 20, pp. 283-303, 1992.  Martins, F., M. Vigário & S. Frota. 2016. FreP - Frequency in Portuguese. Version V2016. Software in CD-ROM. Multifunctional Computational Lexicon of Contemporary Portuguese, available at http://www.clul.ul.pt/en/resources/88-project-multifunctional-computational-lexicon-of-contemporary-portuguese  Perfetti, Ch. “Comprehending written language: the blueprint of the reader,” In C. Brown and P. Hagoort (Eds.), The neurocognition of language. Oxford University Press, pp. 167-208, 1999. Rayner, K.,  Juhasz, B. & A. Pollatsek, “Eye movements during reading,” In M. Snowling and C. Hume (Eds.) The Science of reading:a handbook. Oxford: Blackwell Publishing Ltd, 79-97, 2005. Vellutino, F. “Literacy: Reading (Early Stages),” Encyclopedia of Cognitive Science, 2006.  Text Familiar topic   CO: About a Lisbon neighborhood  Unfamiliar topic ITA: About thermo-acoustic proofing   Position Syntactic boundary (SB) Target word in the right periphery of an NP   Um aspecto interessante deste bairro deve-se          ‘An interesting aspect of this neighborhood is due’ Discursive boundary (DB) Target word in the right periphery of a sentence       nas suas ruas de passeios largos.          ‘in its wide sidewalks streets.’ Head Phrase (SN) Target word: a nominal head of an NP   A vida deste bairro mundano          ‘The life of this mundane neighborhood’ Independent variables Eye movements analysis  First fixation  (FF)  First pass  (FP) Total time of word fixation  (TTF) Speech acoustic analysis :  Stressed vowel length(SVL) Fundamental frequency of the stressed vowel (F0) Dependent variables Graph 4: Mean values (ms) for visual reading variables (FF, FP and TTF) by Text (CO, ITA) and by Position (SN, SB, DB).  Graph 5: Mean values for “reading speech” variable SVL for both texts, by Position (SN, SB, DB). 0 100 200 300 400 500 600 700 SN SB DB SN SB DB CO ITA ms FF FP TTF 0 20 40 60 80 100 120 SN SB DB SN SB DB CO ITA ms 0 50 100 150 200 250 SN SB DB SN SB DB CO ITA Hz Graph 6: Mean values for “reading speech” variable F0 for both texts, by Position (SN, SB, DB). 0 10 20 30 40 50 60 70 80 90 1 2 3 4 5 6 CO ITA Graph 1: Syllables per  word 0 50 100 150 200 250 CV V CVC CVGN CVN VC CVG VN CCV VG CO ITA 0 2 4 6 8 10 12 14 CO ITA 2. Hypothesis H1 Text complexity has an impact on  visual reading time (fixations) and on reading speech time (speech and silent pauses). Longer values for reading time variables (eyes and voice) are expected in a more complex text.  H2 Syntactic and discursive boundaries, as loci for structure building and information integration, respectively, trigger changes in prosody and in eye movements. Increased fixation time and stronger prosodic indicators at discourse than at syntactic boundaries are expected within and between texts. H3 Variables from eye movements and speech may be correlated to signal structure building and information integration. PEst-OE/LIN/UI0214/2015 

Reading aloud: eye movements and prosody

This study aims to connect data from ocular movements and reading aloud speech to syntactic and discursive properties of texts, in order to understand integrative cognitive processes during reading for understanding. Assuming that in reading aloud there is a close interaction between syntax structure and speech prosody, we collected eye-tracking and reading speech data from 17 females  native EP speakers. Eye movements and reading speech produced simultaneously were analyzed and our results are quite clear showing that eyes and voice are both responsive to text complexity and to syntactic and discursive critical loci, as key points of information integration.info:eu-repo/semantics/publishedVersio

Universidade de Lisboa: Repositório.UL

The Repositório Aberto

Av. Professor Gama Pinto, 2, 1649-003 Lisboa, Portugal | Tel.: +351 217 904 700 | Fax: 
+351 217 965 622 | URL: www.clul.ul.pt 
Reading aloud: eye movements and prosody 
Isabel Falé1,2 Armanda Costa1 Paula Luegi1  
ifale@campus.ul.pt; armandacosta@letras.ulisboa.pt; paulaluegi@letras.ulisboa.pt 
1Universidade de Lisboa – CLUL | 2Universidade Aberta 
Speech Prosody 2016, Boston, May 31 – June 03 
The reader and the reading processes have been studied for years in a psycholinguistic perspective and a lot is 
known about what is involved (Perfetti, 1999; Vellutino, 2006). However, there is still much to learn considering 
interrelated processes and behaviors, which have been separately studied. Silent reading and reading aloud 
inform us in different ways about linguistic processing, comprehension and fluency. Tools and measures to deal 
with underlying cognitive processes are specific for each modality. 
Eye movements provide measures that can indicate some linguistic properties of the text (Rayner et al. 2005): 
- First fixation in a word varies according to orthographic, phonological and morphological properties, reflecting 
specific processes to visual word recognition. 
- First pass, which includes FF and other fixations before moving the eyes to right or left regions, could tap the 
processes involved in lexical access of less frequent or longer words, required for their integration in a larger 
meaning or structural unit. 
- Total time of word fixation, including all fixations in a word, must reflect word integration in a semantic-
discursive mental representation, and can be taken as a window for wrap-up effects. 
Reading speech, or speech produced in reading aloud, is strongly constrained by the communicative context, 
reader experience and text properties, such as vocabulary, syntactic structures, punctuation and layout. In 
reading speech, prosody is an important window to study phrasing processes and to relate them with syntactic 
parsing. To identify prosodic boundaries as important loci to study linguistic processing we consider two acoustic 
parameters (Gussenhoven & Rietveld 1992): 
- Stressed vowel length, as a marker indicating the proximity of a high level prosodic boundary: the longer the 
time vowel duration is, the higher the boundary.  
- Fundamental frequency of the stressed vowel, as an indicator of the syntactic position of the word: the more 
the word is embedded in the phrase or sentence the higher frequency on the stressed vowel. 
1. Eye movements and reading speech 
Aims 
• Identify processes of linguistic information integration undergoing in oral reading for understanding. 
• Verify the effect of linguistic and discursive properties of texts on language processing, considering their 
complexity level. 
• Find related measures of prosody in reading aloud and in eye movements in order to contribute for reading 
fluency assessment and teaching. 
Participants  
17 European Portuguese native female speakers, students, proficient readers. 
Procedure  
Eye movements were recorded with a SMI IVIEW X™ HI-SPEED system, at a 1250Hz speed, and sound was 
recorded with a Logitech® Webcam Pro 9000. 
Stimuli were divided, for presentation, into two blocks of text, font in size 22, Courier New, with two paragraphs 
spacing between rows, in a 17-inch screen. 
After reading each text participants answered a multiple-choice questionnaire, thus ensuring a reading 
comprehension task. 
Stimuli 
Two informative texts, similar in word number (200), syntactic and information structure and layout: 
CO: An easy text about a Lisbon neighborhood 
ITA: A hard text about thermo-acoustic proofing 
Thematic dimension as well as some word properties crucially contribute to differentiate the level of complexity 
of texts.  
Word properties in each text were controlled considering: number of syllables, type of syllables and word 
frequency, with tools for automatic linguistic processing – FreP and CRPC (Graph 1, 2 and 3). 
Graph 2: Syllable types 
3. Experiment 
4. Results 
Results for Syntactic and Discursive positions 
Eye movements 
• Position effect in FP is clear when comparing SN and DB 
(p<0.001), SB and DB (p<0.001), with longer reading times 
for DB in both cases. A Position effect is consistently 
registered in TTF: ​SN higher than SB (p=0.03) and lower 
than DB (p<0.001); SB lower than DB (p<0.001) ​ (Graph 4). 
Reading speech 
• There is no Position effect in SVL, however we find an 
interaction between Text and Position in SVL (p<0.001).   
• Considering F0 declination in European Portuguese, an 
expected Position effect in F0 is verified, being SN higher 
than SB (p<0.001) and than DB (p<0.001), and SB higher 
than DB (p<0.001) (Graph 6). 
The different complexity of the two texts and its impact on linguistic processing was confirmed. The hypothesis 
of readers’ responsiveness to linguistic and discursive complexity is sustained by evidences coming from 
speech and eyes. 
Discursive Boundary is the locus where processes related with structural building ending and prosodic 
phrasing occur: the eyes take longer to complete the structure and possibly resume prior information, while 
speech signals the most embedded nominal head of an NP with higher F0, which declines as the information 
is being integrated, with the lowest values on the periphery of the sentence. 
Lengthening of the stressed vowel and first fixation seem to be responsive to lexical properties. 
Fundamental frequency and First Pass seem to reveal structural building, integration of information and 
possibly wrap-up effects. 
References 
Graph 3: Three syllable words frequency 
Results for text 
Eye movements 
• Results show a strong effect of Text, with significant 
differences between the 2 texts in all dependent variables 
(FF p=0.02; FP p=0.001; TTF p=0.001), with lower times in 
CO than in ITA (Graph 4). 
Reading speech 
• We find a Text effect in SVL (Graph 5) with lower values in 
CO than in ITA (p<0.001).  
5. Conclusions 
Gussenhoven, C. & T. Rietveld, “Intonation contours, prosodic structure, and preboundary lengthening,” Journal of Phonetics, vol. 20, pp. 283-303, 1992.  
Martins, F., M. Vigário & S. Frota. 2016. FreP - Frequency in Portuguese. Version V2016. Software in CD-ROM. 
Multifunctional Computational Lexicon of Contemporary Portuguese, available at http://www.clul.ul.pt/en/resources/88-project-multifunctional-computational-lexicon-
of-contemporary-portuguese  
Perfetti, Ch. “Comprehending written language: the blueprint of the reader,” In C. Brown and P. Hagoort (Eds.), The neurocognition of language. Oxford University 
Press, pp. 167-208, 1999. 
Rayner, K.,  Juhasz, B. & A. Pollatsek, “Eye movements during reading,” In M. Snowling and C. Hume (Eds.) The Science of reading:a handbook. Oxford: Blackwell 
Publishing Ltd, 79-97, 2005. 
Vellutino, F. “Literacy: Reading (Early Stages),” Encyclopedia of Cognitive Science, 2006.  
Text 
Familiar topic   CO: About a Lisbon neighborhood  
Unfamiliar topic ITA: About thermo-acoustic proofing   
Position 
Syntactic boundary (SB) 
Target word in the right periphery of an NP   Um aspecto interessante deste bairro deve-se 
         ‘An interesting aspect of this neighborhood is due’ 
Discursive boundary (DB) 
Target word in the right periphery of a sentence       nas suas ruas de passeios largos. 
         ‘in its wide sidewalks streets.’ 
Head Phrase (SN) 
Target word: a nominal head of an NP   A vida deste bairro mundano 
         ‘The life of this mundane neighborhood’ 
Independent variables 
Eye movements analysis 
 First fixation  (FF) 
 First pass  (FP) 
Total time of word fixation  (TTF) 
Speech acoustic analysis :  
Stressed vowel length(SVL) 
Fundamental frequency of the stressed vowel (F0) 
Dependent variables 
Graph 4: Mean values (ms) for visual reading 
variables (FF, FP and TTF) by Text (CO, ITA) and by 
Position (SN, SB, DB).  
Graph 5: Mean values for “reading speech” 
variable SVL for both texts, by Position (SN, SB, DB). 
0 
100 
200 
300 
400 
500 
600 
700 
SN SB DB SN SB DB 
CO ITA 
m
s
 
FF 
FP 
TTF 
0 
20 
40 
60 
80 
100 
120 
SN SB DB SN SB DB 
CO ITA 
m
s
 
0 
50 
100 
150 
200 
250 
SN SB DB SN SB DB 
CO ITA 
H
z
 
Graph 6: Mean values for “reading speech” 
variable F0 for both texts, by Position (SN, SB, DB). 
0 
10 
20 
30 
40 
50 
60 
70 
80 
90 
1 2 3 4 5 6 
CO ITA 
Graph 1: Syllables per  word 
0 
50 
100 
150 
200 
250 
C
V
 
V
 
C
V
C
 
C
V
G
N
 
C
V
N
 
V
C
 
C
V
G
 
V
N
 
C
C
V
 
V
G
 
CO ITA 
0 
2 
4 
6 
8 
10 
12 
14 
CO ITA 
2. Hypothesis 
H1 Text complexity has an impact 
on  visual reading time (fixations) 
and on reading speech time 
(speech and silent pauses). 
Longer values for reading time 
variables (eyes and voice) are 
expected in a more complex text. 
 
H2 Syntactic and discursive 
boundaries, as loci for structure 
building and information integration, 
respectively, trigger changes in 
prosody and in eye movements. 
Increased fixation time and stronger 
prosodic indicators at discourse 
than at syntactic boundaries are 
expected within and between texts. 
H3 Variables from eye movements 
and speech may be correlated to 
signal structure building and 
information integration. 
PEst-OE/LIN/UI0214/2015 


Reading aloud: eye movements and prosody

Abstract

Similar works

Full text

Available Versions

Repositório Aberto da Universidade Aberta

Universidade de Lisboa: Repositório.UL

The Repositório Aberto

Repositório Aberto da Universidade Aberta

Repositório da Universidade de Lisboa

Crossref

The Repositório Aberto