This article reports on a pilot study to evaluate usability, effectiveness, and impact of Academic Word Suggestion Machine (AWSuM), a web-based, innovative tool, powered by the combination of rhetorical moves and lexical bundles with an auto-complete feature. Eight L2 writers participated in the study. Their open-ended comments after using the tool were qualitatively analyzed and classified in thematic categories. The results showed that the developed tool brought about beneficial effects that genre writing pedagogy explicitly aims to achieve. Implications are discussed in terms of the potential role that a tool like AWSuM could play in the teaching and learning of technology-enhanced genre writing.This study was made possible by a Grant-in-aid for Scientific Research (No. 17H02369)from the Japan Society for the Promotion of Science

Mizumoto Atsushi

水本 篤

Kansai University Repository

Initial Evaluation of AWSuM: A Pilot Study著者 Mizumoto Atsushijournal orpublication titleVocabulary Learning and Instructionvolume 6number 2page range 46-51year 2017-12権利 (C) authorURL http://hdl.handle.net/10112/13023Please cite this article as: Mizumoto, A. (2017). Initial Evaluation of AWSuM: A Pilot Study. Vocab-ulary Learning and Instruction, 6 (2), 46–51. doi: 10.7820/vli.v06.2.MizumotoVocabulary Learning and Instruction Volume 6, Issue 2, December 2017 http://vli-journal.orgInitial Evaluation of AWSuM: A Pilot StudyAtsushi MizumotoKansai Universitydoi: http://dx.doi.org/10.7820/vli.v06.2.MizumotoAbstractThis article reports on a pilot study to evaluate usability, effectiveness, and impact of Academic Word Suggestion Machine (AWSuM), a web-based, innovative tool, powered by the combination of rhetorical moves and lexical bundles with an auto-complete feature. Eight L2 writers par-ticipated in the study. Their open-ended comments after using the tool were qualitatively analyzed and classified in thematic categories. The results showed that the developed tool brought about beneficial effects that genre writing pedagogy explicitly aims to achieve. Implications are discussed in terms of the potential role that a tool like AWSuM could play in the teaching and learning of technology-enhanced genre writing.1 Academic Word Suggestion MachineAcademic Word Suggestion Machine (AWSuM) is an online tool that pro-vides support for English academic research writers through an approach that integrates move analysis (Swales, 1990, 2004), particularly prominent in ESP (English for specific purposes), with the lexical bundles used in corpus research (Biber, Conrad, & Cortes, 2004; Biber, Johansson, Leech, Conrad, & Finegan, 1999). AWSuM auto-suggests high-frequency lexical bundles for each move within a section of a paper in a specific discipline. AWSuM (http://langtest.jp/awsum/) has been available free of charge since February 2016.Figure 1 shows a screenshot of AWSuM. Compared with other reference tools, AWSuM includes the following advantages:•	 It can display frequent lexical bundles within a section of a paper in a spe-cific discipline according to rhetorical structures (i.e., moves).•	 It can suggest the most frequent lexical bundles through an auto-complete user interface.•	 It displays the words before and after the search terms, thus making it easier to check for collocations and formulaic sequences in a specific discipline.•	 It includes a concordance feature, thus allowing it to display how words are actually used in the context of discipline-specific research articles.Due to these features, AWSuM is significantly more useful than conven-tional reference tools such as dictionaries. We have also prepared a user’s manual (http://langtest.jp/awsum/manual/AWSuM-Manual_E.pdf), which provides com-plete details about the features implemented in the tool.Vocabulary Learning and Instruction, 6 (2), 46–51.Mizumoto: Initial Evaluation of AWSuM 47From the beginning of its development, AWSuM has been based on the proof-of-concept of the bundle-move connection approach proposed by Cortes (2013). The bundle-move connection approach integrates two corpus-based ap-proaches to the analysis and teaching of discourse structure: top-down and bot-tom-up (Biber, Connor, & Upton, 2007). The top-down approach focuses on macro-textual features such as moves. Thus, the analytical framework for the particular communicative functions (i.e., discourse units) is prepared first, and the analysis is then conducted based on the predetermined discourse unit types. On the contrary, the bottom-up approach focuses on linguistic features such as lexico-grammatical patterns. That is, the corpus investigation of vocabulary or grammar and grouping of its functions in texts comes first, and the discourse unit types are then identified based on linguistic criteria or groupings. In genre-based writing instruction, it is generally considered that it is the combination of a top-down approach to genre analysis and a bottom-up approach to corpus analysis that further helps raise learner awareness of rhetorical functions and linguistic features (Charles, 2007; Cortes, 2007; Cotos, 2009; Flowerdew, 2015).Genre-based pedagogy is centered around rhetorical consciousness-raising (Hyland, 2004; Tardy, 2009), and technology can better address discipline-specificity and individualization of genre-based writing instruction (Cotos, 2014). Thus, in recent years, this line of corpus-based and genre-based writing instruction has been further Figure 1. A Screenshot of the Web-Based Writing Support Tool “AWSuM.”Vocabulary Learning and Instruction, 6 (2), 46–51.Mizumoto: Initial Evaluation of AWSuM48enhanced with the aid of technology (Anthony & Lashkia, 2003; Cotos, Huffman, & Link, 2015). AWSuM was designed to further this growth in technology-enhanced genre writing pedagogy. It is a data-driven and theory-based practical writing support tool for research articles (see Mizumoto, Hamatani, & Imao, in press, for details).This article reports on a pilot study with L2 writers in Japan in order to evaluate its usability, effectiveness, and impact of AWSuM.2 MethodAWSuM was pilot-tested with eight L2 writers in Japan: six undergraduate students (five females, one male) majoring in English who used the tool to write their Bachelor’s theses in applied linguistics, and two graduate students (both fe-males) writing their M.A. theses in applied linguistics. They participated in this small pilot study on a voluntary basis.Their proficiency, based on their self-report of several proficiency test scores, was from B2 (independent user) to C1 (proficient user) in the Common European Framework of Reference (CEFR) for Languages (Council of Europe, 2001). Given their intermediate level of proficiency in English, the participants were considered appropriate as a target population sample (i.e., novice or non-native English re-searchers). None of them had written an RA prior to their thesis writing. Their theses followed the Abstract and Introduction, Methods, Results, Discussion, and Conclusion (IMRD/C) patterns, and consulting the developed tool as a reference resource would greatly help them with their linguistic difficulties. Thus, the par-ticipants’ needs matched the purpose of the current pilot testing. No training was given to the participants except for explaining the concept of the tool.The six undergraduates provided their feedback on an open-ended response sheet after using the developed tool for 2 months. The two graduate students were asked, for 5 months, to keep an online log of their search terms, open-ended com-ments, and feedback each time they used the tool while writing their manuscripts.3 ResultsThe open-ended comments from all eight participants at the end of pilot testing came to a total of 284 entries, which were manually analyzed and clas-sified into thematic categories. This analysis was conducted by the author of this article and a collaborative researcher, according to the following catego-ries formulated from the literature on evaluating learners’ responses to data-driven learning (DDL) tools (Mizumoto, Chujo, & Yokota, 2016; Yoon, 2011): (1) noticing of lexico-grammatical patterns (discovery), (2) confirmation of lex-ico-grammatical patterns (reference), (3) benefits unique to this tool, (4) greater confidence and autonomy in L2 RA writing, and (5) suggestions for improve-ment. The Kappa coefficient for the inter-rater reliability of the coding was 0.83, showing a very good level of agreement. Where disagreement occurred, a deci-sion was made by consensus.Excerpts of characteristic comments as well as overall frequencies for each cat-egory are provided in Table 1. The open-ended user comments suggest that the tool Vocabulary Learning and Instruction, 6 (2), 46–51.Mizumoto: Initial Evaluation of AWSuM 49Table 1. User Feedback on AWSuM (Categories, Frequency, and Excerpts)(1) Noticing of lexico-grammatical patterns (Discovery) 67/284 entries (23.6%)  -  I was able to learn the appropriate verb for a given noun. [e.g., excerpt # => excerpt # illustrates]  -  I noticed a number of verbs suitable for a certain context. [e.g., these studies have => these studies have shown]  -  I was able to learn the appropriate preposition for a given noun. [e.g., the Cronbach’s alpha => the Cronbach’s alpha for]    -  I was able to learn the specific nouns following a given adjective by setting the left-hand range. [e.g., has been a controversial => has been a controversial topic]  -  I was able to learn how technical terms are used. [e.g., Bonferroni => Bonferroni multiple comparison](2) Confirmation of lexico-grammatical patterns (Reference) 67/284 entries (23.6%)    -  I was able to confirm that the phrase I came up with or wanted to use is frequently used. [e.g., procedure was followed]    - I was able to confirm phrases for which I was unsure of their usage. [e.g., is defined as]  -  I was able to check if a phrase I had used in a different genre could be used in research articles. [e.g., the reason why]  - I was able to check how an unfamiliar word like a synonym is used in research articles.  - I was able to check how sentences could be constructed by viewing more suggestions.(3) Benefits unique to this tool 58/284 entries (20.4%)    -  Setting the section and the move helped me to find more appropriate phrases in the context. [e.g., university students who were majoring]  -  When I did not know how to begin a sentence, I looked at the “most frequent n-grams” in the move.  -  I came across a more appropriate phrase in the “most-frequent n-grams” when I set the section and move and tried to search for a different phrase.  -  This tool suggests frequently-used phrases for a given move, which made me realize how important those phrases are.    - By fixing the left-hand range, I was able to find a more appropriate phrase in the context.(4) Greater confidence and autonomy in L2 RA writing 39/284 entries (13.7%)    -  I can write my paper with confidence because the tool suggests appropriate academic phrases.  - I can write my paper with a more professional tone, compared with writing without any help.  - As I use the tool, I may be able to learn many structural patterns and use them in my writing.  -  The more I used the tool, the more smoothly I was able to write. I want to keep using the tool.  -  I was able to feel assured that the phrases I had never used in my writing were actually used in other published papers. (5) Suggestions for improvement 53/284 entries (18.7%)  -  I felt I was not able to search for phrases well. It may take time to get used to using the tool adequately.  -  It would be better if more example sentences were suggested because I did not know how to begin a sentence.    - Lexical bundles should be arranged and suggested according to their meanings.  - It would be better if a dictionary or thesaurus were included in the tool.  -  While the concordance lines were useful, I was not sure which phrases would be most appropriate.Vocabulary Learning and Instruction, 6 (2), 46–51.Mizumoto: Initial Evaluation of AWSuM50was useful for discovering and confirming lexico-grammatical patterns in research article moves in applied linguistics. Users also reported unique benefits of this tool, which most likely led to the development of greater confidence and autonomy in the users’ L2 RA writing. These findings suggest that AWSuM indeed has the beneficial effects we set out to achieve in developing it. At the same time, users’ feedback pointed to the difficulties they faced in making the most of using the tool (see “Suggestions for improvement” in Table 1). These difficulties will be addressed in future studies.Taken together, although limited in terms of the number of participants, this pilot study provided valuable information about users’ perceptions of the AWSuM’s usability, effectiveness, and impact, as well as suggestions for its further improvement.4 ConclusionCorpus-based approaches to the analysis and teaching of discourse struc-ture have increasingly become the established norm in language teaching and learning. One of the most advanced applications of such corpus-based approaches is their use in technology-enhanced genre L2 writing pedagogical tools (Anthony & Lashkia, 2003; Cotos, 2014). Against this background, AWSuM was designed to further the genre-based technology by developing a data-driven and theo-ry-based practical writing support tool, which combines rhetorical moves and lexical bundles. The current pilot study showed that preliminary user feedback was mostly positive. Although larger-scale studies will be necessary to test the utility of AWSuM in detail, this pilot study confirmed that AWSuM could play a facilitating role in technology-enhanced genre writing teaching and learning.AcknowledgementsThis article is based on Mizumoto, Hamatani, and Imao (in press). This study was made possible by a Grant-in-aid for Scientific Research (No. 17H02369) from the Japan Society for the Promotion of Science. I would like to thank the anonymous reviewers of Vocabulary Learning & Instruction for their insightful comments and suggestions on the manuscript.ReferencesAnthony, L., & Lashkia, G.V. (2003). Mover: A machine learning tool to assist in the reading and writing of technical papers. IEEE Transactions on Profes-sional Communication, 46, 185–193. doi:10.1109/TPC.2003.816789Biber, D., Connor, U., & Upton, T.A. (2007). Discourse on the move: Using corpus analysis to describe discourse structure. Amsterdam, The Netherlands: John Benjamins.Biber, D., Conrad, S., & Cortes, V. (2004). If you look at...: Lexical bundles in uni-versity teaching and textbooks. Applied Linguistics, 25, 371–405. doi:10.1093/applin/25.3.371Biber, D., Johansson, S., Leech, G., Conrad, S., & Finegan, E. (1999). Longman grammar of spoken and written English. London, England: Longman.Vocabulary Learning and Instruction, 6 (2), 46–51.Mizumoto: Initial Evaluation of AWSuM 51Charles, M. (2007). Reconciling top-down and bottom-up approaches to graduate writing: Using a corpus to teach rhetorical functions. Journal of English for Academic Purposes, 6, 289–302. doi:10.1016/j.jeap.2007.09.009Cortes, V. (2007). Exploring genre and corpora in the English for academic writ-ing class. The ORTESOL Journal, 25, 8–14.Cortes, V. (2013). The purpose of this study is to: Connecting lexical bundles and moves in research article introductions. Journal of English for Academic Pur-poses, 12, 33–43. doi:10.1016/j.jeap.2012.11.002Cotos, E. (2009). Designing an intelligent discourse evaluation tool: Theoreti-cal, empirical, and technological considerations. In C.A. Chapelle, G.H. Jun, & I. Katz (Eds.), Developing and evaluating language learning materials (pp. 103–127). Ames, IA: Iowa State University. Retrieved from https://www.apling.engl.iastate.edu/tsll/2008/12IADEElena.pdfCotos, E. (2014). Genre-based automated writing evaluation for L2 research writing: From design to evaluation and enhancement. New York, NY: Palgrave Macmillan.Cotos, E., Huffman, S., & Link, S. (2015). Furthering and applying move/step con-structs: Technology-driven marshalling of Swalesian genre theory for EAP pedagogy. Journal of English for Academic Purposes, 19, 52–72. doi:10.1016/j.jeap.2015.05.004Council of Europe. (2001). Common European Framework of Reference for Lan-guages: Learning, teaching, assessment. Cambridge, England: Cambridge University Press.Flowerdew, L. (2015). Using corpus-based research and online academic corpora to inform writing of the discussion section of a thesis. Journal of English for Academic Purposes, 20, 58–68. doi:10.1016/j.jeap.2015.06.001Hyland, K. (2004). Genre and second language writing. Ann Arbor, MI: University of Michigan Press.Mizumoto, A., Chujo, K., & Yokota, K. (2016). Development of a scale to mea-sure learners’ perceived preferences and benefits of data-driven learning. ReCALL, 28, 227–246. doi:10.1017/S0958344015000208Mizumoto, A., Hamatani, S., & Imao, Y. (in press). Bundle-move connection ap-proach: Applying its concept to the development of an online writing sup-port tool for research articles. Language Learning. Retrieved from http://mizumot.com/files/LL-AWSuM.pdfSwales, J.M. (1990). Genre analysis: English in academic and research settings. Cambridge, England: Cambridge University Press.Swales, J.M. (2004). Research genres: Explorations and applications. Cambridge, England: Cambridge University Press.Tardy, C.M. (2009). Building genre knowledge. West Lafayette, IN: Parlor Press.Yoon, C. (2011). Concordancing in L2 writing class: An overview of research and issues. Journal of English for Academic Purposes, 10, 130–139. doi:10.1016/j.jeap.2011.03.003

Initial Evaluation of AWSuM: A Pilot Study

https://kansai-u.repo.nii.ac.jp/?action=repository_action_common_download&item_id=752&item_no=1&attribute_id=19&file_no=1

Initial Evaluation of AWSuM: A Pilot Study

Abstract

Similar works

Full text

Available Versions

Kansai University Repository