Speech recognition, machine translation, and corpus analysis for identifying farmer demands and targeting digital extension

The increasing capabilities of Artificial Intelligence-augmented data analytics present significant opportunities for agricultural extension organizations operating in the Global South. In this project, we supported Farm Radio International (FRI) in investigating the possibility of automating the process of translating and analyzing farmers' voice message data. This report reviews several approaches to overcoming technical constraints and then presents a cutting-edge approach that utilizes innovations in unsupervised learning to deliver highly accurate speech recognition and machine translation in a diverse set of languages

Social and Psychological Factors in Bilingual Speech Production

Studies in the fields of bilingualism and second language acquisition have shown that both cognitive and affective psychological factors can influence individuals’ bilingual speech production. More recently, both experimental and variationist studies of bilingual communities have examined the role of social factors on bilinguals’ speech, particularly in cases of long-term language contact and minority-language bilingualism. The Special Issue brings together work on the psychological and/or social factors that influence bilingual speech production as well as work that uses different methodological frameworks. We examine the role of such factors on bilingual speech production in diverse contexts, in order to provide a more holistic account of the ways in which extra-linguistic influences may affect bilinguals’ speech in one or both of their languages

Rapid Generation of Pronunciation Dictionaries for new Domains and Languages

This dissertation presents innovative strategies and methods for the rapid generation of pronunciation dictionaries for new domains and languages. Depending on various conditions, solutions are proposed and developed. Starting from the straightforward scenario in which the target language is present in written form on the Internet and the mapping between speech and written language is close up to the difficult scenario in which no written form for the target language exists

Communicative practices in a bi-/multilingual, rural, fourth grade classroom in Kenya

Today, multilingualism in primary education is a reality that must be fully embraced in language and literacy research. Multilingualism is the norm in schools due to linguistic heterogeneity in classrooms. Despite the growth of bilingual education all over the world in the twenty-first century (Chimbutane, 2011; Garcia, 2009), there remains little understanding around how two or more languages interact and affect learning. This study was designed to understand and document how emerging bilingual or multilingual speakers deploy their communicative practices, specifically in a fourth grade rural classroom in Kenya, and how the deployment of those resources affects knowledge construction and access to literacy. To do so, I draw on sociocultural (Bakhtin, 1981; Vygotsky, 1978) and cognitive (Collier, 1995; Cummins, 1979; 1981) theoretical perspectives. These theoretical perspectives permit recognition of the importance of native languages in the development of literacy in a second language (L2), as well as the importance of sociocultural contexts as influences on literacy learning (Bakhtin, 1981; Vygotsky, 1978, 2012). A qualitative case study approach was employed to understand the communicative practices of emerging multilingual children in a fourth grade classroom. The study was carried out in a rural primary school in eastern Kenya. The participants included the school principal, the English and science teachers, five focal students, and five parents of those students. The data collection procedures included classroom observations, interviews, shadowing, collection of artifacts, and home visits. The findings indicate that while safe talk strategies predominate in English language arts classrooms, students also would engage unofficial literacies during those lessons, an indication of a disconnection caused either by a language barrier or other factors. In the science classroom, the teacher used (officially disallowed) translanguaging approaches, which raised student participation and disrupted the Initiation, Response Feedback (IRF) discourse pattern that otherwise prevailed in the English language arts classrooms. Additionally, students used their multilingual resources in both writing and speaking practices, even when they were required to use one language. These literacy practices suggest that students enact their lived practices in school settings, thereby disclosing a need to consider and put to good purpose those resources that they bring to school. Another major finding is language as a problem and time on task ideologies that were entrenched in the language practices and linguistic decisions made by the education stakeholders (parents, students, and teachers alike). These ideologies were embodied in daily literacy practices and were articulated, and imposed, through institutional policies. We find that these ideologies eventuate in the exclusion of the rural children from literacy access due to a language barrier. They also lead to changes in pedagogical strategy such that teachers resort to teaching to the test, helping students simply to memorize formulaic phrases necessary to pass a test. In this way, student creativity and voices are silenced, and education is distanced from the child. This deployment of linguistic resources then reproduces social inequalities, most of all in the conditions that lead to continued mass illiteracy in rural settings. I call for a heteroglossic multilingual pedagogy, for bilingual and emerging multilingual children in rural Kenya. Such an education acknowledges the sociohistorical and ideological bases of current language-in-education policies—not only, for example, an exclusive choice of English for literate social functions and the reservation of indigenous languages for oral interpersonal relations and storytelling—but also the effects that this has had on formation of linguistic ideologies and attitudes towards knowledge in certain languages. Heteroglossic multilingual education acknowledges that different languages index varying viewpoints, challenges the stratification of language that tends undesirably towards oppressive universality rather than liberating heterogeneity, and holds out the feasibility of making informed decisions to support and enable the multiple voices of children, through channels like stylization and hidden dialogicality (Bakhtin, 1981). Through heteroglossic multilingual education, education can be connected or reconnected to children, so that children can be guided to acquire and use a foreign language without negating their existing linguistic resources and identities

Low-resource speech translation

We explore the task of speech-to-text translation (ST), where speech in one language (source) is converted to text in a different one (target). Traditional ST systems go through an intermediate step where the source language speech is first converted to source language text using an automatic speech recognition (ASR) system, which is then converted to target language text using a machine translation (MT) system. However, this pipeline based approach is impractical for unwritten languages spoken by millions of people around the world, leaving them without access to free and automated translation services such as Google Translate. The lack of such translation services can have important real-world consequences. For example, in the aftermath of a disaster scenario, easily available translation services can help better co-ordinate relief efforts. How can we expand the coverage of automated ST systems to include scenarios which lack source language text? In this thesis we investigate one possible solution: we build ST systems to directly translate source language speech into target language text, thereby forgoing the dependency on source language text. To build such a system, we use only speech data paired with text translations as training data. We also specifically focus on low-resource settings, where we expect at most tens of hours of training data to be available for unwritten or endangered languages. Our work can be broadly divided into three parts. First we explore how we can leverage prior work to build ST systems. We find that neural sequence-to-sequence models are an effective and convenient method for ST, but produce poor quality translations when trained in low-resource settings. In the second part of this thesis, we explore methods to improve the translation performance of our neural ST systems which do not require labeling additional speech data in the low-resource language, a potentially tedious and expensive process. Instead we exploit labeled speech data for high-resource languages which is widely available and relatively easier to obtain. We show that pretraining a neural model with ASR data from a high-resource language, different from both the source and target ST languages, improves ST performance. In the final part of our thesis, we study whether ST systems can be used to build applications which have traditionally relied on the availability of ASR systems, such as information retrieval, clustering audio documents, or question/answering. We build proof-of-concept systems for two downstream applications: topic prediction for speech and cross-lingual keyword spotting. Our results indicate that low-resource ST systems can still outperform simple baselines for these tasks, leaving the door open for further exploratory work. This thesis provides, for the first time, an in-depth study of neural models for the task of direct ST across a range of training data settings on a realistic multi-speaker speech corpus. Our contributions include a set of open-source tools to encourage further research

A neurolinguistic approach to pronominal resumption in Akan focus constructions

The current project explored the phonological and syntactic aspects of Akan pronominal resumption. The grammatical tone features of the resumptive pronoun and the clause determiner were assessed in Akan speakers with agrammatism. We found that the resumptive pronoun worsens wh-question comprehension in agrammatic speakers. However, the production of pronominal resumption was relatively spared. The ERP study investigated Akan native speakers’ sensitivity to the distribution of the resumptive pronoun by creating word-order and animacy violations. Our study represents a novel addition to the sentence processing field, as it looks into the interface between syntax, semantics, and phonology in Akan pronominal resumption

The Investigation of Code-Switching in a Computerised Corpus of Child Bilingual Language

This dissertation describes the investigation of codeswitching in a computerised corpus of child bilingual language, the LOBILL Corpus, which consists of twenty-five hours of recordings of naturalistic interactions between two bilingual Brazilian/English siblings (JAM, 3;6 and MEG, 5;10) and their family members. Collected over three years, the data was transcribed and coded using the CHAT (Codes for the Human Analysis of Transcripts) transcription system developed by MacWhinney and colleagues (MacWhinney, 1991). In addition to standard CHAT coding, language codes were inserted throughout the corpus and a specially developed postcode was added to all bilingual utterances. Addressee information for each utterance was also included. The longitudinal and heterogenous nature of the corpus and its specific coding allowed for the comprehensive investigation of the children's code-switching practices from both grammatical and pragmatic perspectives. Three levels of analyses were performed using the CLAN (Computerized Language ANalysis) software (ibid). First, quantitative analyses were carried out using the commands FREQ (which outputs frequency word lists), VOCD (which outputs vocabulary diversity scores) and WDLEN (which outputs mean word and utterance lengths). An analysis of the results pointed to the existence of relationships between the various values found and the participatory roles of English and Portuguese in code-switched utterances. The second level of analysis involved the examination and interpretation of word lists and code lists produced by the use of FREQ. Using Myers-Scotton's 4-Morpheme Model (4-M Model) (Jake & Myers-Scotton, 2009) to interpret the word lists, comparisons of morpheme types revealed the existence of an asymmetry in terms of the contributions of both languages to bilingual utterances. These results were seen to lend support to the Matrix Language/Embedded Language asymmetry proposed in the Matrix Frame Language Model (MFL Model) (ibid). The quantitative analysis of four types of codes (used to code instances of retracings and reformulations, errors, tag questions and metalinguistic usage) provided evidence for the existence of potential relationships between these features of spoken discourse and code-switching

The role of phonology in visual word recognition: evidence from Chinese

Posters - Letter/Word Processing V: abstract no. 5024The hypothesis of bidirectional coupling of orthography and phonology predicts that phonology plays a role in visual word recognition, as observed in the effects of feedforward and feedback spelling to sound consistency on lexical decision. However, because orthography and phonology are closely related in alphabetic languages (homophones in alphabetic languages are usually orthographically similar), it is difficult to exclude an influence of orthography on phonological effects in visual word recognition. Chinese languages contain many written homophones that are orthographically dissimilar, allowing a test of the claim that phonological effects can be independent of orthographic similarity. We report a study of visual word recognition in Chinese based on a mega-analysis of lexical decision performance with 500 characters. The results from multiple regression analyses, after controlling for orthographic frequency, stroke number, and radical frequency, showed main effects of feedforward and feedback consistency, as well as interactions between these variables and phonological frequency and number of homophones. Implications of these results for resonance models of visual word recognition are discussed.postprin

Interactive effects of orthography and semantics in Chinese picture naming

Posters - Language Production/Writing: abstract no. 4035Picture-naming performance in English and Dutch is enhanced by presentation of a word that is similar in form to the picture name. However, it is unclear whether facilitation has an orthographic or a phonological locus. We investigated the loci of the facilitation effect in Cantonese Chinese speakers by manipulating—at three SOAs (2100, 0, and 1100 msec)—semantic, orthographic, and phonological similarity. We identified an effect of orthographic facilitation that was independent of and larger than phonological facilitation across all SOAs. Semantic interference was also found at SOAs of 2100 and 0 msec. Critically, an interaction of semantics and orthography was observed at an SOA of 1100 msec. This interaction suggests that independent effects of orthographic facilitation on picture naming are located either at the level of semantic processing or at the lemma level and are not due to the activation of picture name segments at the level of phonological retrieval.postprin