The ACER Longitudinal Literacy and Numeracy Study is set within the conceptual framework of developmental assessment. This paper describes work done on the development of a numeracy scale, and demonstrates how the progressive achievement of the LLANS cohort of students can be reported on that scale and subscales

Forster, Margaret

Meiers, Marion

Stephanou, Andrew

English

ACEReSearch

Assessing and reporting students’numeracy growthThe ACER Longitudinal Literacy and Numeracy Studyis set within the conceptual framework ofdevelopmental assessment. The feature ofdevelopmental assessment which distinguishes it fromother forms of assessment is:that the intention of developmental assessment is toobtain an estimate of a student’s current location on aprogress map as a guide to the kinds of learningexperiences likely to be most useful at that stage inthe student’s learning and as a basis for monitoringgrowth over time (Masters & Forster 1997, pp. 1–2).In a paper presented to the 1999 ACER ResearchConference, Improving Literacy Learning, the earlystages of work on the development of a literacy scalewere described (Meiers & Forster 1999). The presentpaper will describe the work which has been donesince then on the development of a numeracy scale,and will demonstrate how the progressiveachievement of the LLANS cohort of students can bereported on that scale and subscales.Central to developmental assessment is the use ofprogress maps, or continua describing increasinglevels of achievement. These progress maps orcontinua provide frames of reference for monitoringthe development of individuals or groups. At differentpoints in time, estimates can be made of a student’slocation on the continuum, and changes in locationprovide measures of growth over time. The LLANS will provide the empirical evidence forconstructing a numeracy scale, that is: a scale based on a measure of each participatingstudent’s achievement. Each student’s responses toLLANS tasks [will be] used to construct the scale sothat the location of [numeracy] skills [will be] basedon students’ observed performances on the[numeracy] tasks. The method used to construct thescale allows achievement measures to be interpretedin terms of the skills typical of students at variouslevels of achievement (Meiers & Forster 1999).Once the LLANS numeracy scales have beendeveloped, it will then be possible to map theachievement of the whole cohort of participatingstudents, subgroups, or individuals at several points intime.The Longitudinal Literacy andNumeracy StudyThe key research question to be investigated in theLLANS is: What is the nature of literacy and numeracyACER Research Conference 2000Improving Numeracy Learning: What does the research tell us?38Constructing scales for reporting growth in numeracy: theACER Longitudinal Literacy and Numeracy StudyAndrew Stephanou, Marion Meiers & Margaret Forster Australian Council for Educational ResearchAndrew Stephanou is a researcher at the Australian Council for Educational Research.  He has taught physics for 17 years at secondary andtertiary levels.  He has extensive experience in large scale testing programs (HSC/VCE Physics and literacy/numeracy statewide testing inAustralia) and research in higher education (at the University of Melbourne).  Currently his main interest is in educational assessment andpsychometrics, more precisely using Rasch Measurement in the construction and description of educational variables.  His work includes theconstruction of phenomenographic categories calibrated with Rasch Measurement on a single scale for the preparation of an instrument formeasuring conceptual understanding in physics.Marion Meiers came to ACER in 1998 with wide-ranging professional experience at all levels of education, particularly in the professional fieldsof literacy and the teaching of English. She has played leadership roles in the contexts of teaching, professional development, curriculum andassessment, and policy implementation. At ACER she is project director of the ACER Longitudinal Literacy and Numeracy Study. She alsodirected the DETYA-funded cross–sectoral project, the Successful Interventions Literacy Research. She has played a major role incoordinating the 1999 and 2000 ACER Research Conferences, Improving Literacy Learning (1999) and Improving Numeracy Learning, (2000).Prior to joining the ACER, she was consultant to the Literacy Section in DEETYA, Canberra, in 1997; and Executive Liaison Officer of theAustralian Literacy Federation from 1993 - 1997.She was a member of the Expert Group for the development of the national literacybenchmarks. She has a particular interest in curriculum development and assessment of Year 12 English, and chaired the accreditation panelfor the new VCE English Language Study. She has been State Reviewer for VCE English since 1992. She also teaches the third year B Edprogram in Secondary English Method at RMIT University.Margaret Forster is a Senior Research Fellow at ACER. Prior to joining ACER, while undertaking post graduate studies in special education, sheworked as a special resource teacher.  Since 1989 she has been a member of the humanities item writing team, working on a number of testingprograms including the NSW Basic Skills Testing Program, the Queensland Core Skills Testing Program, the Victorian General AchievementTest and the Western Australian Monitoring Standards in Education Program. Ms Forster conceptualised and co-authored the firstDevelopmental Assessment Resource for Teachers – DART English (upper primary), and supervised the development of the DART middleprimary English and DART upper primary mathematics kits.  She also supervised the development of materials for use in the 1996 NationalSchool English Literacy Survey (NSELS).  Margaret is co-author of the NSELS report Mapping Literacy Achievement and of the AssessmentResource Kit (ARK) materials.  From 1995 – March 2000 she led the Humanities (Primary) test development team. She directs the researcharea, Assessment and Reporting, within the ACER core program. At present on study leave in the USA, Margaret contributed to the initial workon this paper.development amongst Australian school children? Oneway of responding to this question is to develop a setof scales describing growth in literacy and numeracy.As a national longitudinal study, the LLANS creates anopportunity to develop achievement scales which willdescribe growth from the first year of schoolingthrough to the stage when students make thetransition to secondary school. A national sample of students was selected from anAustralia-wide sample of 100 schools. Ten studentswere randomly selected from class lists provided at thebeginning of the 1999 school year by the 100 schools inthe project, creating a total initial sample of 1000students. As far as possible, students who havechanged schools have been retained in the study.Where students have moved to other schools, theircontinuation in the study has been negotiated with theprincipal of the new school. Around 900 students fromsome 140 schools are now participating in the study. Comprehensive data on the literacy and numeracygrowth of the students is being collected each yearfrom two sources: common tasks developed at ACERand work samples selected from the students’ normalclassroom work. A range of background data on theschool, teachers and student variables is also collectedannually from a set of questionnaires. This will enableanalyses to be made in relation to various subgroupsincluding gender, ESL learners, language backgroundother than English, time spent reading, watchingtelevision and using computers at home, and so on. An important aspect of the methodology is the role ofteachers as partners in the study. In these first years ofthe study, the students’ own teachers have worked in aone-to-one interview-like context to administer andrecord students’ oral responses to the common tasks.The item writers faced the challenge of designing tasksto be administered easily by teachers working one-to-one with students. Each set of instructions wascarefully worded to make the requirements absolutelyclear to both teacher and student, thus ensuring thereliability of the assessments. In designing thecommon tasks it was also essential to take account oftime, acknowledging the practicalities of managingone-to-one assessments in the classroom. Therefore,the focus was on essential aspects of emerging literacyand numeracy.The LLANS numeracy assessment tasksThree sets of common tasks have now been completedby students at three key stages of schooling: in the firstand final terms of the first year at school (1999) and inthe first term of their second year of school (2000). Afourth set of common tasks will be administered in thelast term of the 2000 school year. Four broad aspects of numeracy have beeninvestigated in each of the sets of common tasks:• number• space • measurement • chance and dataThe tasks designed for the first two years of primaryschool have required students to answer questionsorally, while teachers assess and record theirresponses. Wherever possible, hands-on aids such asrods, counters, shapes, coloured stars, pipe cleanersand match sticks have been provided to supportstudents in responding to the tasks.The item writers have designed tasks of variousdifficulties in each of the four broad aspects. Forexample, in the space strand, the tasks included:Start of school (Term 1, 1999)• placing an object upside down, on top of, in frontof, behind, under etc;• identifying shared and different attributes ofshapes;• identifying shapes with same colour and differentattributes; and• naming a geometric figure (square, circle, triangle).End of first year at school (Term 4, 1999)• dentifying the first, third and last object in a line;• naming the fourth position and placing an objectbetween the second and third position;• counting objects; and• naming a rectangle.Start of second year at school (Term 1, 2000)• reading a simple map to locate adjacent animalsand counting animals on the map;• using arrows to determine direction of a path;• identifying the first stopping point on a path andidentifying animals missed by path;• using tiles to copy map onto a grid;• making a square (using match sticks) then makinga larger square;• making a triangle then making a larger triangle;and• indicating where to cut shapes in half (kite, fishshapes).The marking guide (categorisation of children’sresponses) is included with the tasks, and the teacherjudges the child’s responses against the markingguide. Precise instructions have been provided forteachers to follow so that the tasks are, as far aspossible, administered under standard conditions.ACER Research Conference 2000Improving Numeracy Learning: What does the research tell us?39Counting (end of first year at school)Pet Shop (start of second year at school)The marking guide is coded for data entry, and theentered data is then analysed.Constructing the LLANS scales: theprocessRasch measurement allows us to display theperformance of children and the difficulty of tasks onthe same interval scale, in the same units ofmeasurement. High on the scale we see the bestperformances and the most difficult tasks. Low on thescale we find the poor performances and the easiesttasks. The three LLANS surveys completed so far containcommon items that allow the calibration of all tasksused so far to be displayed on the same scale. Thedifficulty of a task for which responses have beenmarked either correct or incorrect is represented by theposition of its threshold on the scale. Children abovethe threshold are more likely to be correct and childrenbelow are more likely to be incorrect. A similarexplanation is given for tasks rated in more than twocategories.The calibration of the tasks on the scale is followed byan analysis of fit to check the extent to which thesetasks target the same latent trait. Misfits in Raschmeasurement are a source of information on theperformance of children. All misfits are considered andexplanations sought. In examining the result of theanalysis of fit, some collapsing of the categories inwhich children's responses had been assigned becomesnecessary either because there is insufficient dataavailable for a sufficiently accurate calibration, orbecause adjacent categories are not clearly andmeaningfully discerned. For example, two categoriesmay be too close along the continuum and the locationof their thresholds overlap considering the error ofmeasurement.In figure 2, four of the LLANS early numeracy scaleitems are used to illustrate our methodology fordescribing the variable constructed with the tasks fromthe first three surveys. The relative positions of thethresholds on the interval scale are those calibratedwith the data. Item 2PPC is more difficult than item1PT3, as indicated by their threshold values.Children’s responses for item 3RP2, and also item1OU, have been classified into three categories,therefore their difficulty is shown by two thresholds.The calibration takes into account the different abilitiesof the children to whom the items have beenadministered. If, for example, an item has beenanswered correctly by 60% of the children at Survey 1and another item by 60% of the children at Survey 3,then the second item must be more difficult becausethe same percentage of better performing children can answer it correctly. The relative difficultyof the items applies to children anywhere on the scale.It can be seen that children find more difficulty inrecognising that there are three more fish thanbutterflies in a set of six fish and three butterflies(2PPC) than in creating their own pattern after havingbeen shown an example of a repetitive pattern (1PT3).The description of the measured variable is a lengthyprocess in which common features in the categories ofitems belonging to the same part of the scale areidentified. Regions of the scale, partly overlapping,with qualitatively different and meaningful descriptionare formed. The description of these regionsconstitutes the description of the measured variable.The process would be similar to a verbal description ofthe temperature scale. The region around 0ºC wouldbe described differently from the region around 15ºCdegrees, etc.Reporting growth on the LLANS numeracyscalesThe construction and description of suitable variablesfor showing the variation in the skills children develop during their early years at school makes itpossible to show the rate at which children developvarious skills. EquipmentPet shopmap andyellow tilesInstructions andquestionsTip the tiles out of thebag near the child.One of these tiles fitsexactly on a small cage,put a tile on the dogcage.How many tiles do youneed to fit exactly onthe snakeÕs cage?DonÕt do it yet. What doyou think?Tell the child to use thetiles if they donÕt know orguess incorrectly.Marking guideTwo, answeredwithout puttingtiles on grid.Two, answeredafter putting tileson grid.OtherNo attemptRecordEquipment18 countersInstructions andquestionsTip out all counters infront of child.Count these and tell mehow many there are.Marking guideEighteenSeventeen ornineteenAny other numberNo attemptRecordACER Research Conference 2000Improving Numeracy Learning: What does the research tell us?40ACER Research Conference 2000Improving Numeracy Learning: What does the research tell us?41																											 			!						"				#				#														$$"%&													#				#	Figure 1. The LLANS early numeracy scale and four of the tasks used to describe it. The two items on the left are dichotomous and the twoon the right are polytomous (three categories).| • | • | •ACER Research Conference 2000Improving Numeracy Learning: What does the research tell us?42$$"%&	, - Figure 2. The LLANS Early Numeracy scale and an example of a growth pathFigure 2 shows how rate of growth over time can bereported at the individual child level and for selectedgroups of children. It can be shown how rate ofgrowth depends on prior achievement through realexamples.Multilevel modelling of the measures of children’sperformance on the LLANS scales will allow theidentification of factors affecting growth.AcknowledgementsThe development of the numeracy items was carriedout by Prue Anderson, Wendy Bodey, Lynne Darkinand Eve Recht from the Australian Council forEducational Research.ReferencesMasters, G. & Forster, M. (1997). Developmental Assessment,ACER Assessment Resource Kit (ARK), Australian Council forEducational Research, Camberwell.Meiers, M. & Forster, M. (1999). The ACER LongitudinalLiteracy and Numeracy Study (LLANS), paper presented toACER Research Conference: Improving Literacy learning –What does the research tell us?, Adelaide, 18–19 October1999.

Constructing Scales for Reporting Growth in Numeracy : The ACER Longitudinal Literacy and Numeracy Study

https://research.acer.edu.au/cgi/viewcontent.cgi?article=1008&amp;context=monitoring_learning

Constructing Scales for Reporting Growth in Numeracy : The ACER Longitudinal Literacy and Numeracy Study

Abstract

Similar works

Full text

Available Versions

ACEReSearch