Nakatsuhara, Fumiyo

English

University of Bedfordshire Repository

Centre for Research in English Language Learning and AssessmentAnalysing multi-person discourse in group speaking tests: How do test-taker characteristics, task types and group sizes affect co-constructed discourse in groups?CRELLAFumiyo NakatsuharaUniversity of BedfordshireResearch BackgroundCRELLADesirability of the group oral tests1) Quick and efficient way of testing spoken language (Bonk & Ockey, 2003)2) Less burden on examiners (Folland & Robertson, 1976); more consistent test-administration (Ockey, 2001) 3) Symmetrical contribution to the interaction (Van Lier, 1989); eliciting richer language samples (ffrench, 2003);34) Students’ positive perception (Fulcher, 1996; Ockey, 2001); possible positive washback effect (Hilsdon, 1995)Current practice of group oral tests• The Hong Kong AS Level Examination• The College English Test, China• The Kanda English Proficiency Test (KEPT), JapanImpact of test-taker characteristicsCo-constructed nature of speaking test performance(McNamara,1996)  RaterScale/Criteria → RatingGenderAcquaintanceship Cultural backgroundL1 Personality 4PerformanceInterlocutor Task(inc. other candidate) CandidateProficiency level(e.g. O’Sullivan, 2000; Berry, 2004; Norton, 2005; Ockey, 2006; Van Moere & Bonk, 2004)Results are often mixed in terms of the direction of the effectsSome more issues to be investigated• Impact of test-taker characteristics on discourse– Most studies on group tests have examined scores (e.g. Van Moere & Bonk, 2004; Ockey, 2006), and how people talk in group tests is unclear • Impact of task types5– Most studies have employed a free discussion format, and only a little is known about other group testing tasks.• Impact of group sizes– Grouping students into groups of 3 is better than groups of 4 [G4 1 person tends to be left out] (Ojima, 2005). Socio-cognitive framework for validating speaking testsTest taker characteristics Extraversion-level Oral proficiency-levelContext ValiditySetting: Task Cognitive ValidityWeir (2005) AdministrationDemands: Linguistic Interlocutor - Number Internal processResponse: Conversational StylesResearch QuestionsRQ1: Impact of two test-taker characteristics in general Are conversational styles in groups affected by a test-taker’s own and his/her group members’ extraversion- and proficiency-levels?RQ2: Impact of two test-taker characteristics & task types Do test-takers’ extraversion- and oral proficiency-levels have different influences on conversational styles among 7different task types? RQ3: Impact of two test-taker characteristics & group sizes Do test-takers’ extraversion- and oral proficiency-levels have different influences on conversational styles in groups-of-three participants as against groups-of-four? If there is any influence/difference, how & why does it occur?Methods of Data Collection & Data AnalysisCRELLAData Collection Participants: 269 Japanese high school students Grouping: – Grouping students into groups of 3 or 4 as they wish (controlling for acquaintanceship and gender)  Test-taker characteristics:– Extraversion-level: a Japanese version of EysenckPersonality Questionnaire (EPQ) (Iwawaki et al., 1980)– Oral proficiency-level: classroom teacher’s assessment Tasks:1. Information-gap task more closed/more structured2. Ranking task3. Free discussion task more open/less structured9 Your class has 20,000 yen to spend by next March. It was agreed to use this money to buy a camera. Your classmates asked your group to decide what camera to buy. You have information about Camera C and D, and your group members have information about Camera A, B, E and F.  (1) Exchange all the information you have and (2) decide which camera your class is going to buy. (Note: you can use all 20,000 yen, but you may want to save some money for other things.)  When you have finished discussing, please tell me which camera you would buy and why you want to buy it.  Camera A        Camera B         Camera C         Camera D       Camera E        Camera F    10  Price (yen) Weight (gram) Type of camera Other characteristics Camera A     Camera B     Camera C 16,000 91 Digital camera  You can take movies. (2 Mega pixels) Camera D 600 90 Disposable camera You can take 27 photos with flash. Camera E     Camera F           It is said that a good high school teacher should have the following qualities.                                                             Knowledge of subject                        Clear speaking voice                            Ability to organize class                                      11        (1) Discuss how important these qualities are.  (2) Decide which three qualities are the most important, and give the reasons. When you have agreed on three qualities, please tell me which three things are the most important and why you thought so.  Intelligence                   Pleasant appearance            Enthusiasm for teaching               Sense of humor Fairness          Clear writing        Love of students ♡ ♡ ♡ Your friend, Ken, will have the first date with his girlfriend tomorrow.  He came to your group and asked a question.  Should boys pay all the costs for dates? Here are some opinions from other friends. Boys and girls are equal. Neither girls nor boys have the right to ask the other to pay the bill. Many girls like boys who are generous. But, if I keep paying for all the dates, I will be poor and I won’t be able to see my girlfriend any more. 12                                                                                                                                         Please discuss your opinions so that you can give him suggestions on what he should do tomorrow.  When you have finished discussion, please tell me what you, as a group, would like to suggest him. Some boys like paying for dates. If they pay, they can usually plan the date and impress girls. Generally speaking, boys eat a lot!  Data AnalysisQuantitative Analysis (Multiple Regression) Predictors (IVs)– Extraversion-level: Japanese EPQ (0-20)1. Self (own) score2. Self-excluded group mean3. Self-excluded group Std.Dev.– Oral proficiency-level: classroom teacher’s assessment (0-5)4. Self (own) score5. Self-excluded group mean6. Self-excluded group Std.Dev. Measures of Conversational Styles (DVs) Goal-Orientation: measured by Topic initiation Interactional Contingency: measured by Topic ratification Quantitative Dominance: measured by The amount of talk(Van Lier, 1989; Young & Milanovic, 1992; Young, 1995; Kormos, 1999)Qualitative Analysis (Conversation Analysis)To interpret and elaborate the quantitative results 13Quantitative & Qualitative ResultsCRELLA MR: Model Summary [Overall] The given model (with six IVs) accounted for 14% of the variance in topic initiation and 21% of the variance in the amount of talk, but nottopic ratification. DV R Square Sig. 15Topic initiation .144 .000 Topic ratification .004 .955 The amount of talk .208 .000 MR: Main Finding 1 [Overall]● A more extraverted/more proficient test-taker initiated more topics and talked more, especially when they were grouped with less extraverted/less proficient group members.16How & why? Scaffolding behaviour in expert/novice asymmetric interactions[Excerpt 1] 4008 (E:15, P:4) 4024 (E:0, P:2) 4032 (E:16, P:3)1 4008: So, let’s discuss about qualities ((looking at 4024 and 4032)). (.5) First one, 2              knowledge of subjects? (.5) What do you think about it? ((looking at 4032))3    4032: Uh, Huh huh [uh4    4008:                       [Wha(h)t?5    (.8)6    4032: Uh: uh::: uh: uh I think knowledge of subject is very important.177    4008: Yeh ((nodding)), of course.8    4032: Uh huh9    (1.0)104008: How about clear speaking voice? ((gesturing towards 4024))11   4024: I think this is more important (.5) for us to clear speaking voice.12   4008: uh huh ((nodding)) Use of body language (e.g. putting a hand towards to the next speaker, raising a hand, clapping hands) (a) to involve quiet members (b) to solve interactional problems (c) to draw an attention  Tolerance of silence[Excerpt 2] 3013 (E:8, P:3) 3017 (E:14, P:3) 3028 (E:7, P:3)1    3017: Uh: I organize, ability to organize class is, a- what mean ability to182              organize class?3    (6.0)4 3017: Wha(h)t does mean ability to organize class?5    (4.0)6 3017: Organize mean soshiki? Soshiki?7    3028: ((nodding))8    (3.0)9 3017: I (.5) I e? most important is love of student, but it is MR: Main Finding 2 [Task-type comparison]The proficiency-level variables were influential in all tasks.The extraversion-level variables were more influential in more open tasks (the ranking and the free discussion tasks), but not in the info-gap task.19How & why?Info-gap Task Compulsory info exchange Making all test-takers talk Information order forcing the interactional order & role[Excerpt 3] 5001 (E:8, P:3), 5006 (E:14, P:3), 5007 (E:16, P:3)5    5001: I know about camera A. Uh It’s price is uh:15   5007: I have camera C. This camera is sixteen thousand yen. :2021  5006: Camera E is two: two thousand ye- uh? :25  (3.0) ((5006 & 5007 looking at the prompt card, while occasionally throwing a glance at 5001))26  5001: uh27  (2.0) 28   5006: huh huh295001: What do you like uh? What do you uh:: would you like , what what30              would you like to, Midori?Ranking & Free Discussion Tasks Personalising the given topics  (a) to justify opinions (b) to persuade other group members (c) to involve quiet speakers into the personalised storiesFree Discussion Task The discourse agenda was not formulated ● The liveliness of the interaction differed greatly (e.g. involvement of jokes)● Difficult to use well-functioning sequence openers (in Expert/Novice asymmetrical interaction)[Excerpt 4] 1071 (E:16, P:2) 1077 (E:18, P:3) 1081 (E:6, P:1) 1083 (E:4, P:0)211 1071: Do you think? What do you think? [unspecific opener]2     1083: Huh huh huh huh3     (1.5)4 1077: What do you think? [unspecific opener]5     (3.0)6 1071: If you you:: (.5) you go to restaurant, restaurant with a boy, (.5) ah you7               pay or money pay8     1081: Pay [me?9 1071: [pay or boy pay? Which do you like? [specific opener]10   1081: Half and half.MR: Main Finding 3 [Group-size comparison]The proficiency-level variables were influential in both group sizes.The extraversion-level variables were more influential in groups of 4 than in groups of 3.22How & why? Collaborative atmosphere in groups of 3Mitigating the effect of extraversion variables Joint utterance completion in groups of 3 Avoidance behaviour in groups of 4 [Excerpt 5] Group of 4, 5045 (E:0, P:3) 5046 (E:16, P:4) 5047 (E:14, P:4) 5049 (E:1, P:3)1 5046: What do you think? ((making deliberate eye contact with 5049))2 5049: Me too.23[Excerpt 6] Group of 4, 3002 (E:3, P:3) 3022 (E:5, P:1) 3026 (E:12, P:)   3032 (E:12, P:5)1 3032: Do you have any any (   ) anything else?2     (8.0)3     3026: huh huh Ryoko?4 3022: Do you think about clear speaking voice, Azumi?In addition… The turn-taking was sometimes mechanical in groups of 4[Excerpt 7] Specifying turn-taking order by gesture in groups of 41     2104: Have you ever been (.) have you ever going to date, date?2     (1.0)3 2106: ((indicating to take turns in a counter-clockwise way))[Excerpt 8] Irrelevant use of “How about you?” in Groups of 41 5002: How about you, Maya?2 5001: Ah:: I I think clear writing is uh is important, because uh my teacher :6 5001: S(h)o I think clear writing is important. How about you, Midori?7 5007: Uh:: I think love of students is important, because when ah teacher :105007: How about you, Yukari?Incompatibility between talking naturally in groups of 4 and talking in groups of 4 in oral tests The test interaction does not allow “schisming” (Egbert, 1997)   unconsciously avoiding the simultaneous talk   inducing the unnatural way of turn-takingConclusionCRELLA A test-taker’s characteristics, his/her group members’ characteristics, task types, group sizesaffected the resulting test-takers’ discourse in group oral tests.  the interactionalist view of construct definition (e.g. Brown, 2005)  Greater attention should be paid to task types & group size [in general] A more extraverted/more proficient test-taker initiated more topics and talked more, especially when they were grouped with less extraverted/less proficient group members. [task] The extraversion-level variables were systematically more influential in more open tasks. 26 If extraversion-level variables are not within the test construct, the info-gap task could be the most preferable…  However, if group oral tests are to test communication ability, extraversion-level should be part of the test construct (considering the reasons for the impact).  The information order in the prompt card occasionally determined the interactional role27Problematic! [group size] Interactions in groups of 3 seem more suitable for group oral tests.  Groups of 4 could make it more difficult to elicit ratable speech from introverted test-takers. Incompatibility between talking naturally in groups of 4 and talking in groups of 4 in oral tests Grouping test-takers into groups of 4 might not always provide a suitable environment where test-takers could display their communication ability!! To refine ‘interactional communication’ rating 28scales for the co-constructed interaction  Not ‘shared scoring’ (e.g., May, 2007) Rating scales that take the dynamics of the interaction process into account (Chalhoub-Deville & Deville, 2005: 826)  use of a descriptive rater report as well as the rating scoreCentre for Research in English Language Learning and AssessmentThank you very much!CRELLAFumiyo NakatsuharaUniversity of BedfordshireFumiyo.Nakatsuhara@beds.ac.uk

Analysing multi-person discourse in group speaking tests: how do test-taker characteristics, task types and group sizes affect co-constructed discourse in groups?

http://uobrep.openrepository.com/uobrep/bitstream/10547/623704/2/research-seminar-may-9th-2012.pdf

Analysing multi-person discourse in group speaking tests: how do test-taker characteristics, task types and group sizes affect co-constructed discourse in groups?

Abstract

Similar works

Full text

Available Versions

University of Bedfordshire Repository