Engineering programs often feature units that contain a semester-long laboratory project, in which students complete an extended piece of work throughout the full duration of the semester. This paper presents an alternative assessment approach called “Milestone-Based Marking”. As students make incremental progress they can claim incremental marks, and are able to receive incremental feedback on their progress. Each of the milestones is rated for difficulty – Easy, Standard, Hard or Challenging. Easy milestones require less effort than Hard milestones, providing students with a clear guide as to how best to invest their time and effort. This approach changes the nature of the assessment from a purely summative process to a largely formative process. This approach has been used successfully across a number of units, with students indicating that they believe that the approach is fair, and that it better supports their learning

Lindsay, Euan

English

espace@Curtin

E D Lindsay, Milestone-Based Assessment: An Alternative Continuous Assessment Strategy for Laboratory Learning Outcomes  Proceedings of the 2008 AaeE Conference, Yeppoon, Copyright © E D Lindsay, 2008 1Milestone – Based Assessment: An Alternative Continuous Assessment Strategy for Laboratory Learning Outcomes   Euan Lindsay Curtin University of Technology, Perth, Australia e.lindsay@curtin.edu.au  Abstract: Engineering programs often feature units that contain a semester-long laboratory project, in which students complete an extended piece of work throughout the full duration of the semester.  This paper presents an alternative assessment approach called “Milestone-Based Marking”.  As students make incremental progress they can claim incremental marks, and are able to receive incremental feedback on their progress.  Each of the milestones is rated for difficulty – Easy, Standard, Hard or Challenging.  Easy milestones require less effort than Hard milestones, providing students with a clear guide as to how best to invest their time and effort.  This approach changes the nature of the assessment from a purely summative process to a largely formative process.  This approach has been used successfully across a number of units, with students indicating that they believe that the approach is fair, and that it better supports their learning.  Introduction The Mechatronic Engineering degree program at Curtin University of Technology has a strong hands-on focus, with students involved in semester long design and build activities in most semesters of the course.  These activities are embedded in semester long units where the focus is upon applying what the students have learned, and in producing functional solutions to real-world (or near real-world) problems.  Historically, the students had been assessed through a number of in-class demonstrations of their progress, and through the submission of reports throughout the semester, with each report covering one of the phases of the project’s development.  Although often referred to as continuous assessment, a more accurate description would be periodic assessment, as the assessment occurs only at distinct points throughout the semester. The way in which students are assessed affects the way in which they engage with their learning (Albon 2003).  Deadlines ultimately serve two purposes – they motivate the students to work, and they ensure that everything doesn’t happen at once.  Setting multiple small deadlines throughout the semester (eg phase one in week three, phase two in week six) causes the students to cluster their work around these smaller deadlines instead of around one large deadline at the end of the semester.  These means that progress is made more regularly, but it also means that there are (potentially) highly stressful marking situations more often as well. A particular difficulty in the assessment of ongoing laboratory projects is the in-class demonstration.  Students always want to be the last to demonstrate, so that they have a few extra precious moments to get things working, but not every group can go last.  Leaving the demonstration until late also means that there is often no second chance to demonstrate – if the project isn’t working at that exact moment, then the students won’t be rewarded with marks. This “single opportunity” problem also applies to the marking of progress reports.  It is frustrating as an academic to see through their report that students have almost grasped a concept, but not quite.  This leaves the academic with a choice of some kind of partial mark, or an extended resubmission – which may introduce other workload and equity issues of its own. The key motivation in this synchronized approach is that the students should achieve project milestones by the given deadline.  Without regular progress, the overall completion of the project is E D Lindsay, Milestone-Based Assessment: An Alternative Continuous Assessment Strategy for Laboratory Learning Outcomes  Proceedings of the 2008 AaeE Conference, Yeppoon, Copyright © E D Lindsay, 2008 2less likely, and regular milestones keep the project on track.  The reason for the deadline is to provide a latest time by which students should have achieved particular outcomes.  There is no pedagogical reason not to reward the students if they have successfully achieved the outcomes earlier.   To do this, the assessment scheme has to allow students the opportunity to earn marks and gain feedback all throughout the semester.  This is what the Milestone based assessment system was developed to achieve.  Milestone based assessment The initial implementation of the milestone based assessment was straightforward.  Each demonstration and report had a number of marks associated with it, each for a particular learning outcome.  The marks for each outcome were made explicit, and the list distributed to the students in advance, with instructions to claim the marks from the laboratory demonstrators or the lecturer whenever they had achieved the milestone.  The submission deadline for the demonstration and reports became the last, rather than only, opportunity at which students could claim these milestones. Once this framework was established, further expansions of the concept were possible, such as rating the milestones for difficulty, offering multi-part milestones, and linking milestones through each other as prerequisites. Rating for Difficulty Not all milestones are equally difficult, or equally valuable.  Some require significant understanding and reflection; others are straightforward simple tasks.  By providing an indication to the students as to which milestones are which, the students have more information with which to plan their work.  For instance, a ±10% error margin may be acceptable in the project specification, but a  ±1% is preferable.  If the accuracy milestones are rated for difficulty, students who find themselves within the 10% margin can then decide whether they wish to invest the time and effort to achieve the smaller tolerance, and thus the additional mark, or whether to focus their energies elsewhere. Four difficulty categories were chosen for the milestones: Easy, Standard, Hard and Challenging.  The intention is that all students should achieve all of the Easy marks.  Standard milestones determine who passes and who fails; a student who is able to complete all the standard milestones should successfully complete the unit.  The Hard marks are intended to separate the passing students from the honours students, and the Challenging marks separate the excellent from the exceptional.  Accordingly, the milestones were allocated to these difficulty categories in the following ratios: Easy 20%, Standard 40%, Hard 20% and Challenging 20% (see Table 1).  Each of the difficulty levels was allocated a shape to provide a simplified visual representation of the category.  These shapes were chosen to be consistent with the Australian Government’s Classification Board symbols for rating movies (Classification 2008). E D Lindsay, Milestone-Based Assessment: An Alternative Continuous Assessment Strategy for Laboratory Learning Outcomes  Proceedings of the 2008 AaeE Conference, Yeppoon, Copyright © E D Lindsay, 2008 3 Table 1: Difficulty Ratings Difficulty Symbol Mark Allocation Easy  20% Standard  40% Hard  20% Challenging 20%  Many students did not realise their familiarity with these classifications until it was made explicit in class.  Once it was made explicit, however, the shape-based naming convention in fact became the dominant paradigm.  Milestones were referred to by their shapes, rather than by their difficulties:  Don’t put in too much work for an easy mark became “it’s just a triangle”  You’re at the border line between getting the mark and not – “you have a square, but not a hexagon” An unintended consequence of the selection of shapes was that the hardest marks were classified as Diamonds – which themselves are valuable.  This led to some of the most ambitious and motivated students proudly identifying themselves as Diamond Hunters, which served as a valuable motivational tool.  Multi-part Milestones Some of the more significant milestones had multiple marks allocated to them.  When combined with the difficulty ratings, this allowed for more subtle distinctions to be made in what was expected from the students. Some multiple-part milestones consisted of two (or more) milestones of the same difficulty.  This indicated to the students that there is a task that needs to be repeated.  For example, the vehicles are controlled through differential steering – each side of the vehicle has its own drivetrain, both of which need to be constructed.  Completion of each drivetrain constitutes its own milestone, with both milestones having the same difficulty level. Other multiple-part milestones consisted of two (or more) milestones of differing difficulty.  This allowed for differing levels of understanding and achievement to be assessed.  Making a circuit work could be rated as a Standard milestone; but the discussion of why it works could be a Hard milestone.  Similarly normal operation of a system may be Standard, but robustly handling unexpected inputs could be rated Challenging.  In this way the students understand the different depths of achievement (and thus different levels of effort) required to earn both marks, and can choose to allocate their efforts accordingly. Pre-requisites Pre-requisite links were identified between some of the milestones, requiring students to complete some tasks before completing others.  This encouraged students to develop planning skills, and also E D Lindsay, Milestone-Based Assessment: An Alternative Continuous Assessment Strategy for Laboratory Learning Outcomes  Proceedings of the 2008 AaeE Conference, Yeppoon, Copyright © E D Lindsay, 2008 4helped scaffold their progress through the work.  The usual pre-requisite chain is planning-results-reflections, although other combinations have been implemented.  The impact Overall, the Milestone based marking scheme has been overwhelmingly positive, from both a student and a staff perspective.  Students report that the approach provides them with a clear understanding of what is expected of them, and staff find that this clearer expectation allows for a stronger focus upon the learning, rather than upon the assessment. Milestone based assessment has been used successfully in a range of project units in the Mechatronic Engineering program:  The design and build of an autonomous line-following vehicle  PLC control of pneumatic cylinders for parts handling  PID control of servomotors through a computer interface The milestone based approach has also allowed for a number of reflective practice milestones to be incorporated into the projects.  Milestones related to anticipating problems are including in the planning stages; milestones related to reflection upon their learning – “What was your biggest mistake?” – are incorporated in the later stages.  In this way deeper learning outcomes are encouraged and achieved in the students. Student Satisfaction The units that incorporate Milestone-Based Assessment perform very well upon Curtin University of Technology’s end-of-semester online evaluation of teaching.  Responses to items such as “The learning outcomes in this unit are clearly identified” and “Feedback on my work in this unit helps me to achieve the learning outcomes” are often at or near 100% agreement.  In addition to this general feedback on the unit, the open-ended questions were very positive towards the milestone-based marking scheme: Milestone marking reinforces the learning objectives in the lab work motivated (some types of) people to acheive the more 'outstanding' marks (since they were explicitly defined and listed as difficult) “New marking scheme is great, really lets you know what’s expected and where to direct studies.  I would recommend that this be used by all lecturers.” In addition to the general support for the approach, students also valued the hidden benefit of the Milestone-based approach:  it changes the nature of the assessment from a purely summative process to a largely formative process.  Students whose performances are borderline are given specific feedback about what they need to do to reach the expected competency levels, and the appreciate this focus: “Most helpful is the depth of feedback, comments on how the work could be improved.” The overall result is that a strong majority of students believe that the assessment is a fair measure of their learning, and that the feedback they receive actively contributes to this learning process.  They also overwhelmingly report that they are reflecting on their learning and becoming more independent learners. Staff Satisfaction It is not just the students that are happier with this assessment approach.  The milestone approach has also improved staff satisfaction.  The milestone based approach represents a change to a mastery learning paradigm.  Rather than only allowing students a single attempt at assessment, they are able to attempt as often as is required, gaining feedback each time, until they reach the required level of competency. E D Lindsay, Milestone-Based Assessment: An Alternative Continuous Assessment Strategy for Laboratory Learning Outcomes  Proceedings of the 2008 AaeE Conference, Yeppoon, Copyright © E D Lindsay, 2008 5This ability to say no, and to have the student re-attempt the assessment, removes one of the more stressful aspects of marking – making decisions in borderline cases.  If a student is borderline in a written report, the marker has to decide whether they get the mark, or a half mark, or no mark, and needs to apply this consistently.  If a student is borderline for a milestone, the marker provides feedback on what is required to clearly demonstrate what is needed to achieve outcome, and the student is given another attempt.  As well as being a better approach educationally, this also lowers the stress levels involved in marking. This approach also supported the involvement of demonstrators in the laboratory, and also widened the range of possible demonstrators to include undergraduate students who had completed the units previously.  Students can be uncomfortable with the marking process, particularly when confronted with borderline calls as discussed above.  The difficulty ratings on the milestones allowed for the demonstrators to avoid the more stressful decisions – the lecturer was responsible for assessing challenging milestones, whilst the demonstrators could happily assess easy and standard ones.  This allowed for the lecturer to focus on the higher order outcomes, and empowered the demonstrators to handle the more straightforward outcomes such as “the circuit works”. Conclusion The Milestone-based marking scheme has improved the learning process in the laboratory.  Its key advantages are clarity of expectations, improved feedback about whether those expectations are being met, and the freedom to choose how to meet those expectations.  This combination of factors ensures that the focus of the learning process is squarely upon the student, rather than upon the series of laboratory reports they are expected to write.  This approach has also shown itself to be transferrable to a range of different projects. The students are happier and more productive with this approach, and they feel that it supports the development of their independent learning skills.  The teaching staff are happier with the approach, with the clear expectations at different difficulty levels allowing for more meaningful and timely feedback to be provided to the students.  Students learn more; they are happier; and the staff are less stressed.  Overall Milestone based marking has shown itself to be a significant improvement in the assessment of semester-long design & build laboratory projects. References Albon, R. J (2003) Assessment drives the learning: Raising the bar.  Paper presented at the Second Assessment and Learning Conference, Nov 24-25, Adelaide, Australia Classification (2008) The Classification Board and Classification Review Board, Australian Government, www.classification.gov.au/special.html , last accessed July 25th, 2008   Copyright © 2008 E D Lindsay: The authors assign to AaeE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced.  The authors also grant a non-exclusive licence to AaeE to publish this document in full on the World Wide Web (prime sites and mirrors) on CD-ROM and in printed form within the AaeE 2008 conference proceedings. Any other usage is prohibited without the express permission of the authors. 

Milestone - based assessment: an alternative continuous assessment strategy for laboratory learning outcomes

https://espace.curtin.edu.au/bitstream/20.500.11937/27446/2/131065_PID%2520131065%2520%27Milestone%27%2520Euan%2520Lindsey.pdf

Milestone - based assessment: an alternative continuous assessment strategy for laboratory learning outcomes

Abstract

Similar works

Full text

Available Versions

espace@Curtin