Accessed 48,156 times on https://pareonline.net from June 17, 2001 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right

Forgette-Giroux, Renée

Simon, Marielle

English

ScholarWorks@UMass Amherst

Practical Assessment, Research, and Evaluation Volume 7 Volume 7, 2000-2001 Article 18 2000 A rubric for scoring postsecondary academic skills Marielle Simon Renée Forgette-Giroux Follow this and additional works at: https://scholarworks.umass.edu/pare Recommended Citation Simon, Marielle and Forgette-Giroux, Renée (2000) "A rubric for scoring postsecondary academic skills," Practical Assessment, Research, and Evaluation: Vol. 7 , Article 18. DOI: https://doi.org/10.7275/bh4d-me80 Available at: https://scholarworks.umass.edu/pare/vol7/iss1/18 This Article is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Practical Assessment, Research, and Evaluation by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact scholarworks@library.umass.edu. A peer-reviewed electronic journal.Copyright is retained by the first or sole author, who grants right of first publication to Practical Assessment, Research & Evaluation. Permission is granted to distribute this article fornonprofit, educational purposes if it is copied in its entirety and the journal is credited. PARE has the right to authorize third party reproduction of this article in print, electronic anddatabase forms.Volume 7, Number 18, June, 2001 ISSN=1531-7714A RUBRIC FOR SCORING POSTSECONDARY ACADEMIC SKILLSMarielle Simon  & Renée Forgette-Giroux,  Faculty of Education, University of OttawaToday’s assessment of postsecondary academic skills must take into account their comprehensive nature and theirmultiple facets (Biggs, 1995; Sadler, 1989). In this regard, the use of rubric is more likely to provide qualitative,meaningful, and stable appraisals than are traditional scoring methods. The stability of assessment results, however,rests on the scale’s ability to lead to a common and uniform interpretation of student performance. The assessment ofpostsecondary academic skills on the basis of such a scale offers several advantages. First, it presents a continuum ofperformance levels, defined in terms of selected criteria, towards to full attainment or development of the targeted skills.Second, it provides qualitative information regarding the observed performance in relation to a desired one. Third, itsapplication, at regular intervals, tracks the student’s progress of his or her skill mastery. Finally, the choice of ratherbroad universal criteria extends the application to several contexts. Despite its merits, however, the use of a generic descriptive scale at the postsecondary level is relatively recent and somedifficulties need to be addressed. This paper has three objectives: a. to present the nature of a generic rubric used to assess postsecondary academic skills, b. to describe a preliminary application in a university setting, and c. to discuss observed related issues from a research point of view.Nature of the rubricThe rubric for scoring academic skills is essentially qualitative and descriptive in nature and relies on criterion-referenced perspectives. It serves to appraise academic competencies such as the ability to critique, to produce scholarlywork, to synthesize, and to apply newly acquired principles and concepts. It requires the use of criteria that best describeactual student products in a postsecondary setting. The criteria form the left-hand column of the two-way table formatand the horizontal continuum contains headings indicating four increasing levels of performance towards competencymastery (Wiggins, 1998).The use of the scale involves the acts of scoring, interpreting, and judging. (Forgette-Giroux, & Simon, 1998; Simon, &Forgette-Giroux, 2000). Scoring occurs when one identifies, within the scale, and for each criterion, the cell descriptionthat most closely matches the observed performance. The interpretation consists of locating the column that bestdescribes the level of skill mastery. Judging means comparing the identified or observed performance level to apredetermined standard level.Context of ApplicationThe rubric discussed in this paper has evolved over the past five years but the latest, most generic version, was usedwithin four graduate and two undergraduate level courses. Course enrollment varied from three to 30 students for atotal of approximately 100 students. The courses were taught by the two authors, both experts in measurement andevaluation in education, and their topics related to research methodology or assessment. Given their theoretical nature,all courses were organized to assist students in their development and mastery of a single, carefully formulatedacademic skill, such as the ability to critically analyze a variety of research studies in education, to write a researchproposal or report in education, and to assess student learning using current assessment methods and principles.Students were asked to assemble a portfolio that included scholarly works such as critiques, proposals, essays,manuscripts (Forgette-Giroux, et al., 1998). Practical assignments such as lesson plans, tests, performance assessments,were always accompanied by a structured critique. Students used the scale to self-assessed their portfolio for formativeand summative purposes.In this specific context, the performance levels, or anchors, are labeled as good, very good, excellent, and exceptional, toconform to the university approved grading scale. The five criteria are: Relevance, scope, accuracy, coherence, anddepth. These criteria are commonly applied to scholarly writing by most manuscript review processes (NCME, PARE)1,as are other attributes such as clarity, rigor, appeal, and strength of argument. The five criteria are also those found inthe curriculum scoring rubrics mandated by the regional educational jurisdiction. The latter were to be learned by thePage 1 of 4 1Simon and Forgette-Giroux: A rubric for scoring postsecondary academic skillsPublished by ScholarWorks@UMass Amherst, 2000training teachers at the undergraduate level and eventually used in their future teaching environment.During the repeated application of the scale to the various university level courses, three concerns arose that have alsobeen noticed elsewhere, and which continue to interest researchers. The following sections describe these difficulties andpresent their tentative treatment in this particular university setting.Scale levels (anchors) identificationWhen the stages of development or mastery of the targeted skills are not empirically grounded, the initial identificationof the scale levels is often arbitrarily determined. Also, when courses are given for the first time, the lack of student worksamples further complicates the scale level identification process. Some researchers define scale levels and criteria in apost hoc fashion, such as was the case with the National Assessment of Educational Progress (Burstein, Koretz, Linn,Sugrue, Novak, Baker, & Lewis Harris, 1995/1996). The difficulty with this approach is that it is context specific andstudents cannot be made aware of these parameters prior to the assessment. An alternative procedure is to select workfrom the student at hand, that is typical of the upper levels of the scale or of the standard level. Wiggins (1998) suggeststhat, given clear parameters around the intended use of the rubric, those criteria that make the most sense are chosenwith an understanding that they may be constantly adjusted based on exemplary performances. In the universitycontext described here, the first version of the scale was developed around the expected student performance at the levelof excellence. As the course progressed, performance exemplars of that level were identified, distributed among thestudents, and used to refine the scale.Specificity of descriptorsFor the scale to be generic enough to be applied in a variety of university courses, the descriptors need to refer to aspread of performances at each level. On the other hand, there is a risk that these statements may be too general andthus lead to inconsistent interpretation of the data. In the study reported here, the descriptors were formulated based oncriteria associated with the development of valued academic skills that are relatively independent of the course contents.These skills tend to combine declarative and procedural knowledge with scholarly writing. The universality andpertinence of the selected criteria in terms of academic and practical perspectives extended the applicability of thedescriptors to a variety of courses at both undergraduate and graduate levels and ensured student endorsement. Inaddition, the formative assessment at least once during the course, allowed the students and their professor to mediatescale interpretation in order to produce stable results. Despite its early stages of development, the scale yielded averagepercent agreements of 75 % between professor and student ratings.Qualitative rating versus quantitative scoringStudent bodies and administrative pressures stress the attribution of a letter grade or a quantitative score to ratingsobtained using the descriptive scale. In assigning a score, the rubric loses its ability to provide detailed and meaningfulinformation about the quality of the performance as reflective of a specific level of skill mastery. Within the studycontext, the university administration required the presentation of a letter grade. Its scale equates Exceptional with theletter A+, Excellent with A/A-, Very good with B+/B and Good with C+. Throughout each course, assessment results werecommunicated to the students primarily using descriptive statements based on the rubric, but a final letter grade had tobe assigned at the end of the course for official transcript purposes. It is interesting to note that, in adopting the scale fortheir own courses, colleagues typically experienced the need to quantify their assessment using complex algorithms,medians, modes, averages. In doing so, they easily lost track of the object of assessment. It would appear therefore, thatthe transition toward a purely qualitative approach within certain administrative constraints, takes repeatedapplications, discussion, and much self-reflection.Discussion and ConclusionThe rubric was initially conceived as a substitute for the numerical scale that became obsolete and unstable in itstraditional application, particularly when assessing complex skills through performance assessments. Its usefulness inhigher education, therefore, largely depends on its ability to lead to meaningful and stable assessment results.Relevancy and consistency of results refer to validity and reliability issues. Among some of the design considerationsput forward by Arter (1993) in the selection of good criteria when constructing rubrics for performance assessments, themost relevant to postsecondary contexts are (a) the need for universal attributes, (b) the means for assessing bothholistically and analytically, and (c) the identification of the main components of the object of assessment. Moskal andLaydens (2000) have proposed practical ways to address these issues. They equate evidence related to content with theextent to which the rubric relates to the subject domain, and construct-related evidence to the conceptualization of acomplex internal skill. Criterion-related evidence, meanwhile, serves to indicate how well the scoring criteria matchthose found in practice. Given this rubric’s generic nature and the focus on the assessment of academic skills, primaryattention must be given to the production of construct-related evidence. This was achieved by linking the scale’s criteria,anchors, and descriptors to the nature of the skill addressed by the rubric and expressed in terms of a single learningobjective.Interrater and intrarater aspects of reliability were greatly improved by attaching the rubric to the course outline andby clarifying its various components and use early in the course, by enabling the students to access high qualityPage 2 of 4 2Practical Assessment, Research, and Evaluation, Vol. 7 [2000], Art. 18https://scholarworks.umas .edu/ are/vol7/iss1/18DOI: https://doi.org/10.7275/bh4d-me80exemplars, by providing regular qualitative feedback, by inviting the students to take part in mediation duringformative assessments, and by requesting them to justify, in writing, their self-assessment based on specific references totheir portfolio. It was important that this written rationale clearly support their perceived level of achievement. Writtensupport of scoring decisions by the professor was also expected.Given the exploratory nature of the study, many questions arise. However, five are of particular interest from bothpractical and research perspectives:1. Would a scale based on a combination of the post hoc approach, of theoretical foundations of academicskills, and of samples of student work lead to increased validity?2. Would a scale based on a combination of both qualitative and quantitative components lead to evengreater consistency of results when assessing academic skills?3. Is there an optimal class size to which this rubric can be applied most efficiently?4. What type of teaching style is more likely to fit with the effective use of the scale?5. Would the scale be as useful in courses focusing on content rather than skill development?Research and dialogue on the obstacles and advantages of this approach are definitely needed to achieve some balanceand to assist professional educators in addressing these issues when using the rubric within their own courses. Anotherdimension in need of further investigation would be to obtain evidence of convergent and discriminant validity. Finally,a rigorous, larger scale validation study of the universality of the criteria is also warranted if the scale is to become awidespread, valuable and valued tool in the assessment of postsecondary academic skills.Footnotes1 See http://ncme.ed.uiuc.edu/pubs/jem_policy.ace and http://pareonline.net/Review.htmReferencesArter, J. (1993). Designing scoring rubrics for performance assessments: The heart of the matter. Portland, OR.: Northwest Regional EducationLaboratory. (ERIC Document Reproduction Service No. ED 358 143)Biggs, J. (1995). Assessing for learning: Some dimensions underlying new approaches to educational assessment. The Alberta Journal ofEducational Research, XLI(1), 1-17.Burstein, L., Koretz, D., Linn, R., Sugrue, B., Novak, J., Baker, E.L., & Lewis Harris, E. (1995/1996). Describing performance standards: Validityof the 1992 National Assessment of Educational Progress achievement level descriptors as characterizations of mathematics performance.Educational Assessment, 3(1), 9-51.Forgette-Giroux, R., & Simon, M. (1998). L’application du dossier d’apprentissage au niveau universitaire. Mesure et évaluation en éducation,20(3), 85-103.Moskal, B.M., & Leydens, J.A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, 7(10).Available online: http://pareonline.net/getvn.asp?v=7&n=10Sadler, R. (1989). Specifying and promulgating achievement standards. Oxford Review of Education, 13(2), 191-209.Simon, M., & Forgette-Giroux, R. (2000). Impact of a content selection framework on portfolio assessment at the classroom level. Assessment inEducation: Principles, Policy and Practice, 17(1), 103-121.Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco: Jossey-BassPublishers. APPENDIXDescriptive scale: EDU5499 Current methods of student assessment in teaching and learning (graduate level course).Learning objective: To be able to critically analyze the technical qualities of own assessment approaches.Name: ________________________    Date:_______________Assessment formative Assessor:  selfsummative professorCRITERIA PERFORMANCE LEVELS (ANCHORS)GOOD VERY GOOD EXCELLENT EXCEPTIONALRELEVANCEPortfolio components addressdirectly the learning objective.Portfolio components are notnecessarily related to thelearning objective.Portfolio components aresomehow linked to thelearning objective.Portfolio components aredirectly linked to the learningobjective.The portfolio reflects exemplaryhigh relevance to the learningobjectivePage 3 of 4 3Simon and Forgette-Giroux: A rubric for scoring postsecondary academic skillsPublished by ScholarWorks@UMass Amherst, 2000SCOPEAll aspects of the learning objectiveand recommended readings arecovered within the portfolio.The portfolio partially reflectsthe various components of thelearning objective.Most elements of the learningobjective and recommendedreadings are covered withinthe portfolio.All aspects of the learningobjective and all recommendedreadings have been dealt withwithin the portfolio.The portfolio incorporatestreatment of elements beyondthe scope of the class andrecommended references.ACCURACYCurrent concepts, terms, principles,and conventions are used correctlyand with clarity throughout theportfolio.Learned concepts, terms,principles, and conventionsare more or less used correctlythroughout the portfolio.The portfolio shows precisionin the use of current concepts,terms and principles butrelevant conventions notalways followed.The portfolio reflects correctand clear use of terms,concepts, principles, andconventions.The portfolio demonstratesclear, correct, precise, andconcise use of terms, concepts,principles and conventions.COHERENCEElements within and across theportfolio are logically andstructurally linked together. Ideasare interconnected and arepresented in a consistent fashionthroughout the portfolio.Elements and ideas arepresented in a disconnected,rather piecemeal fashion.Elements are somehowlinked together but reflectsome inconsistency across theportfolio.Evidence of structural andinternal consistency within andto some extent, across theportfolio.The portfolio is highly andtightly organized. Ideas,concepts and principles arepresented in a consistentfashion across the portfolio.DEPTHThe portfolio reflects a personalposition supported by a richanalysis of relevant and highquality references.The portfolio presents aposition highly dependent ona superficial analysis ofreferences.The portfolio presents aposition supported by someanalysis of relevantreferences.The portfolio reflects a personalposition based on a deep andthorough analysis of relevantreferences.The portfolio presents apersonal position based on anintegration of relevant and highquality references. Descriptors: *Rubrics; Scoring; Student Evaluation; Test ConstructionCitation: Simon, Marielle & Renée Forgette-Giroux (2001). A rubric for scoring postsecondary academic skills. Practical Assessment,Research & Evaluation, 7(18). Available online: http://PAREonline.net/getvn.asp?v=7&n=18.Page 4 of 4 4Practical Assessment, Research, and Evaluation, Vol. 7 [2000], Art. 18https://scholarworks.umass.edu/pare/vol7/iss1/18DOI: https://doi.org/10.7275/bh4d-me80

A rubric for scoring postsecondary academic skills

https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1101&amp;context=pare

A rubric for scoring postsecondary academic skills

Abstract

Similar works

Full text

Available Versions

ScholarWorks@UMass Amherst