Proceedings of the 16th Nordic Conference 
of Computational Linguistics NODALIDA-2007.
Editors: Joakim Nivre, Heiki-Jaan Kaalep, Kadri Muischnek and Mare Koit.
University of Tartu, Tartu, 2007.
ISBN 978-9985-4-0513-0 (online)
ISBN 978-9985-4-0514-7 (CD-ROM)
pp. 73-80

Granfeldt, Jonas

Nugues, Pierre

English

This paper describes a system to define and evaluate development stages in second language French. The identification of such stages can be formulated as determining the frequency of some lexical and grammatical features in the learners’ production and how they vary over time. The problems in this procedure are threefold: identify the relevant features, decide on cutoff points for the stages, and evaluate the degree of success of the model. The system addresses these three problems. It consists of a morphosyntactic analyzer called Direkt Profil and a machine-learning module connected to it. We first describe the usefulness and rationale behind its development. We then present the corpus we used to develop the analyzer. Finally, we present new and substantially improved results on training machine-learning classifiers compared to previous experiments (Granfeldt et al., 2006). We also introduce a method to select attributes in order to identify the most relevant grammatical features

Lund University Publications

LUND UNIVERSITYPO Box 117221 00 Lund+46 46-222 00 00Evaluating stages of development in second language French: A machine-learningapproachGranfeldt, Jonas; Nugues, PierrePublished in:NODALIDA 2007 PROCEEDINGS2007Link to publicationCitation for published version (APA):Granfeldt, J., & Nugues, P. (2007). Evaluating stages of development in second language French: A machine-learning approach. In J. Nivre, H-J. Kalep, K. Muischnek, & M. Koit (Eds.), NODALIDA 2007 PROCEEDINGSUniversity of Tartu.Total number of authors:2General rightsUnless other specific re-use rights are stated the following general rights apply:Copyright and moral rights for the publications made accessible in the public portal are retained by the authorsand/or other copyright owners and it is a condition of accessing publications that users recognise and abide by thelegal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private studyor research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portalRead more about Creative commons licenses: https://creativecommons.org/licenses/Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will removeaccess to the work immediately and investigate your claim.Evaluating Stages of Development in Second Language French: AMachine-Learning ApproachJonas GranfeldtCentre for languages and literatureLund universityBox 201, S-221 00 Lund, Swedenjonas.granfeldt@rom.lu.sePierre NuguesDepartment of Computer scienceLund Institute of TechnologyBox 118, S-221 00 Lund, SwedenPierre.Nugues@cs.lth.seAbstractThis paper describes a system to define andevaluate development stages in second lan-guage French. The identification of suchstages can be formulated as determining thefrequency of some lexical and grammaticalfeatures in the learners’ production and howthey vary over time. The problems in thisprocedure are threefold: identify the rele-vant features, decide on cutoff points for thestages, and evaluate the degree of success ofthe model.The system addresses these three problems.It consists of a morphosyntactic analyzercalled Direkt Profil and a machine-learningmodule connected to it. We first describe theusefulness and rationale behind its develop-ment. We then present the corpus we usedto develop the analyzer. Finally, we presentnew and substantially improved results ontraining machine-learning classifiers com-pared to previous experiments (Granfeldt etal., 2006). We also introduce a method toselect attributes in order to identify the mostrelevant grammatical features.1 IntroductionSince the beginning of systematic research in sec-ond language acquisition (SLA) in the 1970s, oneline of investigation was to identify and analyzestages of development that learners pass throughwhen acquiring a second or a foreign language.See Sharwood-Smith and Truscott (2005) for a re-cent discussion. Stage identification can be appliedto data from all linguistic levels, but it is perhapsmost interesting for the development of morphol-ogy and syntax. Within SLA, the learner’s inter-nal grammar is considered as its own system, aninterlanguage grammar, that develops and restruc-tures over time (Selinker, 1972). The objective ofthis research is to determine and model the growthof the learner’s grammar, where the identification ofrelevant grammatical features, the definition of de-velopment stages, and their evaluation are complextasks requiring a systematic methodology (Ellis andBarkhuizen, 2005), pp. 97–98.In this paper, we describe and evaluate a systemthat has fully automated this process. As possibleapplications for it, we can think of diagnostic toolsfor assessing language development and we hopethat both learners and teachers will find it usefulin this respect. However, we focus here on howour system, and more generally the methodology wepropose, can assist researchers when working withgrammatical stages. In order to understand its rele-vance, we begin with a simplified description of howstage identification is commonly carried out in thefield of SLA.2 Background2.1 Current methodology for identifying stagesof developmentThe first step to identify stages of development isto determine and extract grammatical features in theproduction (oral or written) of a representative pop-ulation of learners. The selection of features canbe theoretically or empirically motivated, the crucialpoint being that the selected features have a contentvalidity, i.e. that they are features whose realizationscan translate a qualitative change in the learner’sgrammar. A second step is to understand and modelthe development of these features over time. Somelinguistic features show a straightforward linear de-velopment, i.e. the scores for adequate use of thefeature increases steadily with time at some observ-able rate. Other features show a nonlinear, some-times U-shaped, development where the scores ini-tially are high and then decrease in a second phase,only to regain a high level of correctness in a thirdphase.Once the developmental trajectories are known, athird step is to decide on cutoff points in the datawhere the learner has reached a new stage of devel-opment. Most researchers work on several grammat-ical features at the same time, a procedure some-times referred to as grammatical profiling. Thismeans that the establishment of a stage of develop-ment has to take into consideration the analysis of alarge number of categories.2.2 Some problems with the currentmethodologyA necessary component in the method describedabove is an in-depth morphosyntactic analysis ofthe language samples produced by the learners. Inour case, these are written texts but they might alsobe transcriptions of oral productions. Most ana-lysts working with first and second language acqui-sition have now access to relatively large amountsof machine-readable data (large in SLA terms). Itis also common for widespread languages, like En-glish and French, to use tools such as morpholog-ical parsers and part-of-speech taggers (MacWhin-ney, 2000). These tools can considerably reduce theotherwise very time-consuming analysis step.But even so, a lot of manual analysis is left to bedone. First there is currently no reliable automatedtool to parse learner’s data although there have beensome attempts for English (Sagae et al., 2005). ForFrench, some of the linguistic structures and fea-tures used in grammatical profiling can be capturedusing available part-of-speech taggers and morpho-logical parsers. But other more complex structuressuch as the agreement between constituents can-not. Another problem in grammatical profiling isthat current tools usually work on one single fea-ture at the time in a pipeline architecture, while oneneeds to analyze a large number of phenomena atthe same time. A third problem concerns the arti-ficiality in identifying stages (Ellis and Barkhuizen,2005), p.98.The result of the morphosyntactic analysis is typ-ically a frequency analysis of certain features. Fora particular linguistic phenomenon, say 3rd personagreement in the present tense, a typical procedureis to identify the different realizations of the phe-nomena and count them. The compiled data for allthe features are then often inspected intuitively inorder to identify suitable stages of development. Inthe SLA domain, there are currently multiple waysof dealing with this step and there has not been anyprincipled evaluation of them. A possible reason forthis is that there is currently no framework that hasconnected any sophisticated statistical treatment tothe first two steps: the morphosyntactic analysis andthe frequency count. If a fully automated process-ing pipeline were available, all steps in this trickyprocess could be evaluated more thoroughly.We report here the current status of our systemthat aims at overcoming the methodological prob-lems discussed above. The rest of the paper is orga-nized as follows. We begin by summarizing brieflythe previous work on the morphosyntactic develop-ment of second language French. Then we describethe corpus we are using to develop the analyzer toextract the grammatical features and constructions.The analyzer, called Direkt Profil, is also presentedbriefly. In the last sections, we discuss our machine-learning approach to identify the stages of develop-ment and select attributes and we present our currentresults.3 Morphosyntactic development of secondlanguage FrenchStudies on the morphosyntactic development of sec-ond language French have to a large extent beenempirically driven. One of their specific aims wasthe identification of a large number of develop-mentally related grammatical features and construc-tions along with hypotheses about their sequenceof acquisition. The study by Bartning and Schlyter(2004) is an example of it for spoken French, wherethe authors identified some 25 different morphosyn-tactic features and proposed a definition of theirdevelopment over time in adult Swedish learners.Taken together, these features delineate six stagesof development in the shape of grammatical profiles– ranging from beginners to very advanced learn-ers. Examples of features are shown in Table 1.As the language learner moves towards an increas-ing automation of the target language, the producedstructures become more frequent, more complex,and more appropriate. Developmental sequences de-scribe this process in linguistic terms.4 The CEFLE CorpusTo develop our analyzer (see Sect. 5) and to testthe machine-learning approach to stages of devel-opment, we used the Lund CEFLE Corpus (CorpusÉcrit de Français Langue Étrangère) (Ågren, 2005).CEFLE consists of texts in French as a foreign writ-ten by 85 Swedish students with different levels ofproficiency. It contains approximately 400 texts and100,000 words. It also features a control group of22 French native speakers. CEFLE was compiledthroughout the academic year 2003/2004. Duringthis period, each student wrote four or five texts inFrench at two months intervals. The aim of thisstudy was to analyze the morphosyntactic develop-ment in written production.For the present study, we used a random selectionof 317 texts from the CEFLE corpus, see Table 2.A member of the team annotated one text from eachlearner using the criteria in Bartning and Schlyter(2004) and classified it according to the develop-mental stage the text was reflecting. For our currentexperiments (see below), we subsequently assignedthe same classification to the three or four other textsof the same learner in the CEFLE corpus. The as-sumption behind the decision to propagate the stageof development from one annotated text to all thetexts of the same learner is that a learner generallydoes not move up to the next stage during the shortperiod under which the collection of the texts tookplace.5 Direkt ProfilDirekt Profil (Granfeldt et al., 2005; Granfeldt etal., 2006) is a morphosyntactic analyzer designedfor French as a second language. The initial aimwas to implement the grammatical features and con-structions in Table 1. In the current version of thesystem, a few features are still lacking but there isalso a great number of additional ones that were notpresent from the beginning. The system has beenpresented in some detail in previous papers and weonly give a brief description of the main parts.Verb groups and noun groups represent the essen-tial grammatical support of the profile classification.The majority of syntactic annotation standards forFrench take such groups into account in one wayor another. However, in their present shape, thesestandards are insufficient to mark up constructionsof Table 1, many of which are specific to foreignlanguage writers. On the basis of the linguistic con-structions in Bartning and Schlyter (2004), we de-veloped our own annotation scheme. The currentversion of Direkt Profil, v. 2.1, detects three typesof syntactic groups, nonrecursive noun groups, verbgroups, prepositional groups, and conjunctions, thatit annotates using the XML format.Direkt Profil applies a cascade of three sets ofrules to produce the four layers of annotations. Thefirst unit segments the text in words. An interme-diate unit identifies the prefabricated expressions.The third unit annotates simultaneously the parts ofspeech and the groups. Finally, the engine creates agroup of results and connects them to a profile. Theanalyzer uses manually written rules and a lexicon ofinflected terms. The recognition of the group bound-aries is done by a set of closed-class words and theheuristics inside the rules. It should be noted thatthe engine neither annotates all the words, nor allsegments. It considers only those, which are rele-vant for the determination of the stage. The engineapplies the rules from left to right then from right toleft to solve certain problems of agreement.The current version of Direkt Pro-fil is available online from this addresshttp://www.rom.lu.se:8080/profil. The perfor-mance of Direkt Profil version 1.5.2 was evaluatedin Granfeldt et al. (2005). The results showed anoverall F-measure of 0.83 (precision and recall).

Evaluating stages of development in second language French: A machine-learning approach

DSpace at Tartu University Library

Evaluating Stages of Development in Second Language French: A Machine-Learning Approach

https://lup.lub.lu.se/search/files/5852929/625979.pdf

Evaluating Stages of Development in Second Language French: A Machine-Learning Approach

Abstract

Similar works

Full text

Available Versions

Lund University Publications

DSpace at Tartu University Library