Bruce, Catherine  D.

Ford, John

Ross, John  A.

English

University of Calgary Journal Hosting

The Alberta Journal of Educational Research Vol. 53, No. 4, Winter 2007, 430-433Research NoteJohn A. RossOntario Institute for Studies in Education, University of TorontoJohn FordKawartha Pine Ridge District School BoardandCatherine D. BruceTrent UniversityNeeds Assessment for the Developmentof Learning ObjectsIn a review of 125 needs assessments conducted in educational settings, Witkin(1994) identified multiple deficiencies: mono-method bias (use of a single datasource to identify needs), confusion between solutions (instructional actions)and needs (gaps in student performance), use of unsystematic procedures, andoverreliance on data from a single group. We attempted to avoid theseproblems in selecting the focus for CLIPS being developed for lower-achievingmathematics students in grades 7-10. In our study, CLIPS (Critical LearningInstructional Paths Support) are learning objects: short multimedia programsfocused on specific learning objectives. In this research note we describe howwe applied systematic needs assessment procedures (Witkin & Altschuld,1995) to focus five CLIPS. Figure 1 summarizes our strategy.Step 1: Link to a formative assessment systemWe framed CLIPS development in a formative assessment system to ensurethat the CLIPS fitted the structure of the curriculum and met the criteria forformative assessment (Black & Wiliam, 1998). Fit means that (a) there is adiagnostic procedure for placing students on a developmental continuum; (b)the CLIPS provide instruction for moving students from one level of the con-tinuum to another; and (c) the instructional strategy embedded in the CLIPScomplements standards-based mathematics teaching (National Council ofTeachers of Mathematics, 2000).We used the grade 6 diagnostic tests of PRIME: Professional Resources andInstruction for Mathematics Educators (Thomson Nelson, 2005). These tests placestudents on a developmental continuum for mathematics that has been empiri-John Ross is a professor of curriculum, teaching, and learning and Head of the Institute’s fieldcentre in Peterborough, ON. His research interests are school change, mathematics education,student assessment, and program evaluation. He can be reached at and Field Centre, OISE/UTTrent Valley Centre, Box 719, 1994 Fisher Drive, Peterborough, ON K9J 7A1.ross@oise.utoronto.ca; tel: 705-742-9773, ext. 2293; fax: 705-742-5104.John Ford is Vice-Principal of Port Hope High School in Port Hope, Ontario. His researchinterests are in mathematics education and teacher professional development.Catherine Bruce is an assistant professor of education and professional learning. Her researchinterests focus on mathematics education, professional learning, and qualitative methods.430cally validated using samples of Canadian students (Small, McDougall, Ross, &Ben Jaafar, 2006). PRIME provides a conception of the curriculum organizedaround a small set of concepts and skills (e.g., for the Number and Operationsstrand there are five organizing concepts and three organizing skills). For eachcurriculum outcome there are five phases of performance. For each level of thecontinuum, PRIME provides examples of developmentally appropriate in-structional strategies to be used with Standards-based texts (Small, 2005a,2005b).Step 2: Select learning objectives that are poorly achievedIn 2005-2006 we collected data from underachieving grade 7 and 8 students(Ross, Ford, & Xu, 2006). We identified 21 problem types for Number andOperations in which student achievement was consistently low. All problemsinvolved fractions.Step 3: Select learning objectives that students believe they need to learnWe assembled pairs of examples for each of the 21 problem types. Studentsused a 1-6 scale to show how confident they were about their ability to solveeach pair of problems. The scale was anchored by not confident at all (1) and veryconfident (6). Each pair of problems was flashed on an overhead screen. It wasessential that the problems not be visible on the screen for long enough for thestudents to solve them. We were measuring students’ self-efficacy beliefs, thatis, their expectations that they could solve each problem type (Bandura, 1997).Following Schunk (1996), we started with a two-second display, but this wasnot enough time to process both items in the pair. After considerable ex-perimentation, we found that the time taken by the teacher to read each pairaloud was sufficient for students to determine whether they could successfullycomplete the problem pairs.We conducted an exploratory factor analysis (principal axis with Varimaxand Promax rotation) on 405 grade 7-10 students’ responses. There was a singlefactor solution: the first eignenvalue=12.541, accounting for 60% of thevariance; second eigenvalue=0.984. We concluded that the 21 items wereFormative Assessment SystemStudentAchievementStudent Perceptionof NeedTeacherKnowledgeCLIPSAgendaFigure 1. Procedures for development of CLIPS agenda.Needs Assessment for the Development of Learning Objects431measuring a single construct (confidence in solving fraction problems) rather thana cluster of themes. We calculated the mean confidence level for each item pairfor the total sample, for students who were in the lowest quartile of self-reported mathematics ability, and for males and females. We found that meanconfidence levels were very high for all students, including those in the lowestquartile. Confidence was higher for boys than girls, although there were nogender achievement differences in Ross et al. (2006).Step 4: Select learning objectives that teachers believe to be importantWe focused on two dimensions of teacher knowledge: (a) Teachers’ pedagogi-cal content knowledge: which areas of the curriculum were most likely to causelearning problems and generate misconceptions. We provided 33 grade 7-10teachers with the same set of problem pairs and asked them, “How muchdifficulty do students in your course have in mastering the learning expecta-tion shown in this pair of items?” (b) Teachers’ disciplinary knowledge: know-ledge of mathematics content and the substantive structure that organizes thiscontent. We asked them to estimate for each problem pair, “How important isit to produce additional CLIPS to help students master the learning expectationshown in this pair of items?”We added the two teachers’ scores together to rank the problem pairs fromhighest to lowest priority. Teachers rated the top 10 problem types in exactlythe same order for difficulty and importance. In contrast, there were substan-tial differences between students and teachers. For example, the item rated asthe highest priority by students (i.e., they were least confident about theirability to deal with problems of this type) was only the 10th highest priority forteachers.Step 5: Set CLIPS AgendaA team of eight expert teachers used the teachers’ and students’ data to identifythe learning objectives to be addressed by the CLIPS. The decision-makingprocess was not a straightforward application of an algorithm, but a thoughtfuldiscussion of all the sources of information. In general the teacher team startedwith the teachers’ results and then factored in the students’ confidence results.After exploring several ways of combining the data, the team decided that theNumber and Operations CLIPS would focus on two themes: representingfractions and equivalent fractions.Lessons LearnedOur procedure provided a rigorous foundation for CLIPS development andhad other benefits: teachers’ ownership increased by their participating in thedata-collection; we discovered that students, especially boys, overestimatedtheir competence in solving fractions; the 21 problem pairs provided items forthe summative evaluation of the CLIPS.AcknowledgmentThe research was funded by the Ontario Ministry of Education and the Social Sciences andResearch Council of Canada. The views expressed in the article are not necessarily those of theMinistry or the Council.ReferencesBandura, A. (1997). Self-efficacy: The exercise of control. New York: W.H. Freeman.J.A. Ross, J. Ford, and C.D. Bruce432Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education,Principles, Policies and Practices, 5(1), 7-74.National Council of Teachers of Mathematics. (2000). Principles and standards for schoolmathematics. Reston, VA: Author.Ross, J.A., Ford, J., & Xu, Y. (2006). Programming remediation and intervention for students inmathematics (PRISM): Final report. Peterborough, ON: OISE/UT Trent Valley Centre.Schunk, D.H. (1996). Goal and self-evaluative influences during children’s cognitive skilllearning. American Educational Research Journal, 33, 359-382.Small, M. (2005a). Number and operations: Background and strategies. Toronto, ON: Thomson Nelson.Small, M. (2005b). Number and operations: Guide to using the developmental map. Toronto, ON:Thomson Nelson.Small, M., McDougall, D., Ross, J.A., & Ben Jaafar, S. (2006). PRIME developmental maps: Researchstudy. Toronto, ON: Nelson.Thomson Nelson. (2005). Number and operations: Diagnostic tests. Toronto, ON: Author.Witkin, B. (1994). Needs assessment since 1981: The state of the practice. Evaluation Practice, 15(1),17-27.Witkin, B., & Altschuld, J. (1995). Planning and conducting needs assessments: A practical guide.Thousand Oaks, CA: Sage.Needs Assessment for the Development of Learning Objects433

Needs Assessment for the Development of Learning Objects

https://journalhosting.ucalgary.ca/index.php/ajer/article/download/55307/42349

Needs Assessment for the Development of Learning Objects

Abstract

Similar works

Full text

Available Versions

University of Calgary Journal Hosting