Despite the multiple benefits of performance-based assessment, the biggest challenge is the subjectivity in

scoring. Rubrics are recommended to reduce its subjectivity and make scoring more reliable. However,

educators tend to shy away from utilizing performance-based assessment due to the challenges faced in

designing good rubrics. Collaborative efforts have often been advocated in developing good quality rubrics

especially for tertiary level, however how academics from various disciplines can work together have not

been dealt sufficiently. This qualitative study was carried out to collaboratively develop oral presentation

and written report assessment rubrics that can be used as a guideline for developing task specific rubrics by

lecturers across disciplines of study. The first stage involved seven experienced lecturers who met

regularly to develop the initial drafts. This was followed by a survey utilising open-ended questionnaire

items conducted with 60 lecturers to gather their feedback on the rubrics developed in the first stage.

While highlighting the comprehensiveness of the criteria included in the rubrics and its value as a useful

point of reference, the respondents suggested improvements on the performance descriptors. The initial

draft was further improved based on the feedback and shared at the university level digital platform

Joseph, Ramanair

Soubakeavathi, Rethinasamy

Unimas Institutional Repository

DEVELOPING RUBRICS FOR PERFORMANCE BASED ASSESSMENT THROUGH COLLABORATIVE EFFORTSouba Rethinasamy1Joseph Ramanair1Freddy Yeo21Faculty of Language and Communication2 Faculty of Resource Science and TechnologyUniversiti Malaysia Sarawak 8-9 September 2021Universiti Malaysia Sarawak,Kota Samarahan, Sarawak,Malaysia.Introduction❖ Assessment is a crucial integral part of teaching and learning❖ Among the different types of assessments, the multiple benefits of alternative assessment which includes performance based assessment are often highlighted especially in relation to Education 4.0 as well as 21st century learning skills❖ However, the value of performance based assessments is clouded by subjectivity in scoring❖ To help improve objectivity in scoring, the development and use of rubrics are highly recommended❖ According to Gallardo (2020, p. 16), “rubrics have evolved in the last years as a powerful support tool  to make judgments about students’ learning in several disciplines.”❖ The usage of rubrics are currently among the most popular innovations in education (Andrade, 2001). ❖ Rubrics, when properly developed, promotes teaching and learning by setting clear goals, offer targeted input and direct the evaluation process (Ghaffar et al., 2020)IntroductionWhat are rubrics?❖ scoring tool or guide that is used to evaluate the quality of students’ responses to a given task❖ contains a list of criteria, performance levels and descriptions of levels of performance quality for the listed criteria (Brookhart, 2018; Cargas et al., 2017)performance descriptors performance descriptors performance descriptors performance descriptors performance descriptors performance descriptors performance descriptors performance descriptors performance descriptors performance descriptors IntroductionTypes  of rubrics❖ Generally, there are two main types of rubrics: holistic rubrics and analytic rubrics.❖ Holistic rubrics provide a general overview of what is expected in a given task. The criteria for evaluation are considered together. ❖ Analytic rubrics is often  highly recommended because it enables the student’s performance to be matched to the appropriate level of performance for each criterion, providing students with specific guidance and feedback related to each criterion and helps assessors to assess consistently (Andrade & Brookhart, 2020).❖ Analytic rubrics can serve as teaching learning  tool as well as an assessment tool. For instructors/assessors➢ help instructors to develop clear and attainable learning objectives for their students➢ help instructors communicate to students the specific requirements and acceptable performance standards for a given task ➢ assist in evaluating students’ learning achievement for that particular task. ➢ enables instructors to better understand what to assess according to specific criteria and implements them accordingly (Beyreli & Ari, 2009) ➢ serve as documentation for students’ achievement ➢ provides instructors with a tangible form of accountability➢ Assist in providing feedback to students➢ rubric to improve the quality of evaluation by teachersFor students➢ help students understand what is required in an assignment, how it will be graded ➢ give students clearer idea of the important elements that they should pay attention to➢ help students understand what are the expected targets or desired performance required from their learning serve as a guideline for students to follow and address in order to meet the expectation of the assessment task➢ helps students make reliable judgments about their own work, do appropriate revisions and improvements➢ help students monitor and assess their progress as they work towards the stated goals➢ helps facilitate students self-assessment helps students produce a better product, perform at their best possible➢ help students focus on what they've learned, how to apply the input they have received and assesses how far they have comes ❖ Benefits of Rubrics• Many previous research has claimed the benefits of using a rubrics as an evaluation tool (Andrade et al., 2008; Becker, 2016; Brookhart, 2018; Ene & Kosobucki, 2016; Andrade & Du, 2005; Jonsson & Svingby, 2007; Laurian & Fitzgerald, 2013; Rezaei & Lovorn, 2010; Sundeen, 2014).Research Problem❖ According to Gallardo (2020), “rubrics design is a topic of upward interest in the educational process around the world” (p. 74). ❖ Despite the benefit of rubrics, educators tend shy away from using performance based assessment due to the challenges faced in designing good scoring rubrics. ❖ While many studies have investigated the effect of using scoring rubrics on students’ performance and assessors’ rating judgement and rating accuracy, only limited studies have focused on the construction of rubrics  (Zhao et al., 2021). ❖ Collaborative effort has often been advocated especially for tertiary level in developing rubrics that can be used as guidelines to developing task specific rubrics.  However, how faculty members can work together in developing good quality guideline rubrics have not been dealt sufficiently (Gallardo, 2020)IntroductionPurpose of the study➢ to provide a detailed description of how rubrics can be developed collaboratively by academics to serve as guidelines for designing task specific rubrics by academic from various disciplinesDraft the rubrics Gather lecturers’ feedback on the draftAnalyse and  discuss the feedback from the lecturersMake the relevant improvements to the rubricsMake the guideline rubrics available via online platform Stage 1Stage 2Stage 3Stage 4Stage 5➢ qualitative study➢ consisted of 5 stagesMethodStage 1• Involved 5 experienced lecturers from different disciplines of study with 15-20 years experience in teaching learning and assessment • meet every weekly to discuss and develop the first draft of the rubrics• Before Meeting➢ the team gathered sample rubrics for oral presentation from their faculties and internet and studied the samples.  • During Meeting➢ discussed and listed the criteria that should be included for the benefit across various faculties➢ drafted the performance descriptors for the highest level for each criterion were written.  ➢ the descriptors for the lower levels were written for each criteria➢ the descriptors in the sample rubrics were referred to and the appropriate words/phrases were identified. ➢ for the less important criteria but need to be included to maintain the comprehensiveness of the guideline rubrics, 3-4 levels of performance indicators were included instead of 5 levels. ➢ After the draft was completed, the team met again for the final time to go through the descriptors and refined them before the next stage.Example of rubrics databases ⚫ i-rubrics -https://www.rcampus.com/indexrubric.cfm⚫ Rubistar -http://rubistar.4teachers.org/index.php⚫ Rubrics for All -https://www.forallrubrics.com/Stage 2• A survey questionnaire consisting of open ended questions was sent out to lecturers. • A total of 60 lecturers for various faculties responded. Respondents NoFaculty of Applied and Creative Arts 8Faculty of Cognitive Sciences 9Faculty of Economics and Business 9Faculty of Engineering 6Faculty of Language and Communication9Faculty of Medicine and Health Sciences3Faculty of Resource Science and Technology9Faculty of Social Sciences 7Total 60InstrumentThe assessment rubrics for Oral Presentation designed to serve asguideline for educators to adapt for their own specific assessments.Please review the oral presentation rubrics. Your review could includebut not limited to the following;➢ Relevance of the criteria included➢ Sufficiency of the levels➢ Clarity of the performance descriptors➢ Suggestions for improvementStage 3 • Analyse and discuss the feedback from the respondents Feedback from the respondents• Overall, the criteria included are relevant for assessing oral presentation• 5 levels are included for the important criteria and 3 levels for the less important but still relevant to oral presentation• The performance descriptors are clearly written and using simple words that can be easily understood by both students and lecturers.Stage 3 • Analyse and discuss the feedback from the respondents “the criteria listed are clear and relevant for assessing oral presentation assessment”“The criteria are appropriate to the knowledge and skills that are evaluated in oral presentation”“Rubrics provide a clear set of criteria. Each criterion is obvious enough to be comprehensible for the students and is relevant to the task. Some of the criteria are divided into sub criteria.  This is useful as this will allow for a more consistency in scoring - more objective, and fairer evaluation by the educators”Sample excerpts from respondents Criteria Levels“performance levels are also realistic, ranging from the lowest, a wide middlerange and the highest being excellent”“The gradation or levels in the rubrics given are clear and precise. It is easily understandable and the breakdown in each level is useful as this will allow for a more objective, and fairer evaluation amongst the educators”Stage 3 • Analyse and discuss the feedback from the respondents Sample excerpts from respondents Performance Descriptors“the descriptors are well constructed with simple language that will not mislead the students when working on their tasks.”“Each performance level shows clear and concrete differences between the levels”  “The descriptors for each of the criteria are clear, detailed and measurable. spelled out using plain English that is relatively easy to comprehend”“The adoption of plain English in rubrics is commendable as the assessors can come from different backgrounds with different levels of prior knowledge and experiences”“The quality of the performance level matches the level of importance of each criterion in this guideline. The major criterion have more detailed descriptors,”Stage 3 • Analyse and discuss the feedback from the respondents “The clothing the presenter is wearing will influence the way the information delivered is received and the audience’s perception of both the presenter and the presentation”“The students’ attire on the presentation day is important - it indicates that they respect the occasion and the audience ”Sample excerpts from respondentsSuggestions for improvement CriteriaInclude Attire“Although students are presenting in a classroom environment, their presentation tasks would normally be based on a specific situation given in the assignment. show that they are sensitive and care about their audience and occasion.”➢ Suggestion accepted and the criterion attire is included in the rubrics. Attire • non-appropriate• not suitable/untidy• acceptable • Reasonably suitable/tidy• smart casual• appropriately dressed• suitable and tidy Stage 3 • Analyse and discuss the feedback from the respondents “By making the student make proper citations and references of the sources, the students are being trained to be responsible for the integrity of the knowledge that they have gained.”“giving credit wherever it is due to the sources that are used to create the oral presentation content is important. ”Sample excerpts from respondentsSuggestions for improvement CriteriaInclude ‘Citation’➢ Suggestion accepted and ‘citation’ included as a sub-criterion for contentCitation • sources cited are mostly unreliable and irrelevant • some of the sources cited are reliable and relevant• citations are incorrectly written• most of the sources cited are reliable and relevant• mostly correctly written• All sources are reliable and relevant• Correctly written Stage 3 • Analyse and discuss the feedback from the respondents “a good presenter not only uses effective eye contact, he/she also needs to constantly attract the attention of the audience with appropriate body language such as simple hand gestures or even moving around from time to time to actively engage audiences seated at different parts of a setting.”“divide ‘Non-verbal communication’ into two (2) distinct sub-criteria – namely ‘Eye contact’ and ‘Body language’ some presenters are good at engaging eye contact with their audience, but they are lacking with the body language and gestures to enhance their articulation and visualisation.”Sample excerpts from respondentsSuggestions for improvement CriteriaSeparate ‘eye-contact’ and ‘body movement’➢ Suggestion accepted and ‘citation’ included as 2 different sub-criteria under Delivery Stage 3 • Analyse and discuss the feedback from the respondents “enthusiasm is important because audience will not pay attention the presenter if they present in an unenthusiastic or boring way. However, it should be a sub-criterion under the criteria for delivery”Sample excerpts from respondentsSuggestions for improvement CriteriaPlace  ‘enthusiasm’ under ‘Delivery’➢ Suggestion accepted and ‘enthusiasm’ included as a sub-criterion under  Delivery Stage 3 • Analyse and discuss the feedback from the respondents “there are a few criteria that have only 3-4 levels whereas the rest have 5 levels and this may be seen as incomplete and can cause confusion. Suggest to add a brief explanation under the rubric to justify the rubric design”Sample excerpts from respondentsSuggestions for improvement Level of PerformanceAdd  ‘note’➢ Suggestion accepted and ‘note’ includedNote: only 3-4 levels of descriptions included for the minor criteria Stage 3 • Analyse and discuss the feedback from the respondents “Under the criterion ‘knowledge of topic, the descriptor includes understanding of topic-it is redundant as the ability to provide accurate and clear points and providing great details and relevant examples are enough to measure the presenter’s understanding of the topic”Sample excerpts from respondentsSuggestions for improvement Performance descriptormodify  descriptor for sub-criterion ‘knowledge of topic’➢ Suggestion accepted and ‘level of understanding of topic’ deleted from the descriptor for Stage 3 • Analyse and discuss the feedback from the respondents “The levels will be clearer it is labelled as excellent, good, average, fair, poor) or numbered 5 for excellent and 1 for ‘poor’. Although the current oral presentation rubric has labelled the two ends of the continuum, “Inadequate” and “Excellent”, it is recommended that each level of performance be labelled. Sample excerpts from respondentsSuggestions for improvement Not acceptedAdd  ‘quantity’➢ Suggestion not accepted – the rubrics is meant to serve as a guideline.  The labels and quantity can be added by lecturers to suit their specific assessment task. Stage 3 • Analyse and discuss the feedback from the respondents “While it is the students’ responsibility to ensure all of their AV materials can be presented well, technical errors can also happen due to external factors such as virus in the computer or software incompatibility. It is unfair for the students to be marked down if the technical error happens due to these external factors.”Sample excerpts from respondentsSuggestions for improvement Not acceptedExclude‘technical matters’➢ Suggestion not accepted – the expert members felt that it is better to include it in the guideline rubrics as some of the lecturers may need to include this criterion to ensure that the students check that their presentation materials/devices are working before their presentation.Stage 4 Make the relevant improvements to the rubricsStage 5 Make the guideline rubrics available via online platform (refer to Appendix)Conclusion➢ This study was to provide a detailed description of how rubrics can be developed collaboratively by academics to serve as guidelines for designing task specific rubrics by academic from various disciplines➢ The designing of the rubrics is supported by feedback data from respondents➢ Future study could investigate how the guideline rubrics is utilised by the academics in constructing task specific rubrics and how it affects the student’s performance and the lecturers’ rating judgement