The core second level course in Software Engineering for Computer Science students at the University of Kent has, for many years, included a substantial design-and-build group project component. In recent years, this has been enhanced to address a number of issues crucial to students' development as reflective, professional practitioners. These include: . group allocation and formation . appropriate technical and non-technical goal-setting within the context of the taught material. These goals are used both to guide practice and as a basis for assessment. . students' critical evaluation of their own and others' achievements, in the context of moderated self and peer assessment. Although the work is assessed as a group, it has proved possible to account for variations between individuals' efforts using a novel questionnaire based approach initially developed by the University of Exeter. Problems (for both staff and students) in the implementation of these enhancements to the group learning and educational experience are addressed, and identified beneficial solutions are described. The value of this approach is situated firstly in terms of changes to the students' approach to subsequent, less constrained, project work, and secondly against the stated learning outcomes of the project; their improved technical and professional practices

Utting, Ian

English

Kent Academic Repository

Negotiated Assessment Criteria and Peer Assessment in SoftwareEngineering Group Project Work: A Case StudyIan UttingComputing LaboratoryUniversity of Kent at Canterbury, UKe-mail: I.A.Utting@ukc.ac.ukThe core second level course in Software Engineering for Computer Science students at theUniversity of Kent (UKC) has, for many years, included a substantial design-and-build groupproject component. In recent years, this has been enhanced to address a number of issuescrucial to students’ development as reflective, professional practitionersProblems (for both staff and students) in the implementation of these enhancements to grouplearning and the educational experience are addressed, and identified beneficial solutions aredescribed. The value of this approach is situated firstly in terms of changes to the students’approach to subsequent, less constrained, project work, and secondly against the statedlearning outcomes of the project; their improved technical and professional practices.1 IntroductionIn common with most undergraduate Computer Science programmes (in the UK), theUniversity of Kent at Canterbury (UKC) runs a small-group design-and-build softwaredevelopment project as part of the second year of its three year degree programme. In ourcase, this project is primarily in support of the core Software Engineering unit. Moreunusually, the Computer Science degrees at UKC also include a group project in the final yearof the programme.Over recent years, we have seen both an increase in student numbers, and an increasingsophistication of the types of work undertaken by students in their final year project. Inresponse to these issues, we have adopted a number of strategies designed to increase thevalue of the project for students both in terms of the technical outcomes of the work and in theextent to which we can encourage them to reflect on their maturation as professionalpractitioners in software engineering.Particularly, we have enhanced those aspects of the work concerned with:• Group allocation and formation.• Technical and non-technical goal setting within the context of the taught material. Thesegoals are used both to guide practice and as a basis for assessment.• Students’ critical evaluation of their own and others’ achievements, in the context ofmoderated self and peer assessment.Although the work is assessed as a group, it has proved possible to account for variationsbetween individuals’ efforts using a novel questionnaire based approach initially developed atthe University of Exeter.Many of the techniques we have drawn upon in this work have been identified by a nationalproject on “Effective Project work in Computer Science” (project EPCoS), in which theauthor is a partner.  EPCoS's goals are “To identify, make explicit and systematise existingbest practices in Computer Science student project methods and techniques in order to makeexisting knowledge and experience readily accessible for the achievement of thresholdstandards in Computer Science graduates” (Fincher, 1998).2 Group AllocationSensible allocation of scarce resources against competing requirements is one of the majorlessons we would like students to learn from this project work. As such, we allow them toselect groups and allocate tasks for themselves, but place a number of constraints in their way.The project that the students undertake has changed over time but always asks them toundertake the development of a small software system, focussing on the developmentprocesses involved rather than the end result. The systems are typically well specified(sometimes formally in, e.g., Z) and fall fairly readily into two halves. Each group of studentsdevelops one half of the system, integrating their work with that of another group, who aredeveloping the other half.This is the second piece of group in the software engineering course. Before attempting it,students have spent some 10 weeks working on a professional-issues case study undertaken ingroups of about a dozen (Fincher & Utting, 1998). The fact that these (large) groups havebeen formed for an entirely different purpose – to reflect on professional (non-technical)responsibilities, and to generate a short group report each week – makes the task of dividingresources between the two halves of the development project an interesting one.In a study of the factors affecting students’ choice of the ways to break down their large-scalegroups, Thorn (1998) observed that, despite the taught material in the course on resourceallocation, the major factor influencing the students’ choice was how much they “liked” theother members of the group. Nonetheless, in a survey of students' experiences of this workand its relevance to their final year projects, “experience of working in a group” and“allocation of tasks to group members” were the most-quoted advantages of having done thework.3 Negotiating project goals and assessment criteriaIt has often been observed that students are focussed strongly on software development as acode-writing activity. This is particularly problematic in the teaching of software engineeringwhere limitations of available student time and effort make it difficult for them to developsoftware of a scale and complexity sufficient to require the techniques being taught. Surveysover time at UKC have suggested that, working from a simple specification of a smallprogram, final year CS students write 15 lines of code per assessment hour. With a totalavailable time per course of 120 hours, this means that even if nothing is done except codingthe average student could produce only 1800 lines of code. A program of this size hardlyrequires the sort of design and project management techniques suitable for large-scalesystems. Thus we need to constrain students into using apparently less appropriate techniques,in order to give them (at least some) practice in them before they move on to systems wheretheir use will be critical.By the time that they come to start the project component of this course, the students havealready been introduced to the basic principles of managing software development. Thisallows us to negotiate with them (in a fairly abstract manner) the kinds of tasks that they willhave to undertake in order to successfully complete the project, and identify the products ofthe project in which evidence for the completion of those tasks will be found. As part of thisprocess, conducted as a group discussion, we also look at the relative weights of the differenttasks identified (in terms of both scale and complexity), which leads to the weight they willeach be given in assessment. Although the outcome of this discussion varies (marginally)from year to year, a typical result is shown in Table 1.Table 1: Assessment criteria with weightingsSection Group 1 Group 2Planning 10 10Quality Management 10 10Design and Design Documentation 15 15Implementation 15 20User Documentation 15 0Testing 15 15Interface Specification 0 10Records of Meetings 10 10Integration 10 10Total 100 100An example of the description of what evidence might be found for these activities, andwhere, is given below:Quality Assurance and ManagementEvidence for these activities will mainly be found in the records of project meetingsand also in the delivered QA documentation.In particular, it should be evident that:• The responsibilities of group members for quality issues have been identified.• Checks on the quality of deliverables have been performed and the resultsdocumented.• Standards (for coding and documentation) have been specified and adhered to.• Configuration management and source code control have been used whereappropriate, to keep track of different code and document versions, and toidentify those responsible for their modification.The act of having negotiated such criteria with the students both encourages them tounderstand the real worth of the individual tasks (especially "implementation"), and providesthem with early guidance on the way in which their work will be judged. The fact that theassessment criteria are aligned with software engineering “good practice” is of immensebenefit to the learning process.4 Encouraging critical reflection – peer and self assessmentGiven the inevitable delay between submission of work for assessment and its return, and thecompeting calls on students’ attention, even the best crafted feedback can easily lose itsrelevance, at least until the next time similar work must be performed, and by then students’understanding of what they had done and why has often faded. For this reason, many studentsfocus their attention on the mark that they are eventually awarded, and much of the value offeedback in developing their understanding of their work is lost.In order to address these problems we instituted a "peer assessment" approach to theassessment of this project work. Getting students to evaluate the work of others is importantin developing their own ideas as well as for the feedback they receive. Knowing that they willhave to judge the work of others against the developed criteria also serves to keep themfocussed on those criteria in their own work.The process adopted requires groups of students who have themselves undertaken a particularhalf of the project to meet and assess the work of another such group, writing a short report aswell as providing marks. A member of staff then moderates the marks before returning them.This moderation process reveals no significant difference between the marks awarded bystudents and by staff, and no significant correlation between the mark eventually awarded to aparticular group and the accuracy of that groups’ marking of the work of others.In early versions of this process, following the return of marked work there was a periodduring which the assessed group could appeal against the assessment, and a meeting wasconvened at which both sides could present their case for (and against) the assessment beforea binding agreement was reached. Typically, this route was taken by some 15% of groups.Partly this was a result of the number of criteria against which the work had been assessed –with more criteria, there is more room for detailed disagreements – but more frequently it wasa result of an unreasonable expectation of the accuracy of assessment. During the moderationprocess, a discrepancy of ±5% overall was judged as acceptable – well within the bounds oferror to be expected when work on such a scale is marked by more than one assessor. Suchvariation is normally hidden from students, relying on their trust of the judgement of staff.Such trust is clearly suspended when students are judged by their peers, even though thevariation in mark is no wider than would be expected from multiple staff assessors.In the current version of the process, groups are asked to submit (along with their projectdeliverables) a self-assessment of their own work against the negotiated criteria. This wasinstituted to assist in their reflection, to give them more of a basis against which to judge thework of others, and to provide input to the moderation process. Unexpectedly, it also has theeffect of reducing the number of groups who are dissatisfied with the mark awarded them bytheir peers from 15% to 5% (one group). So marked is this effect that even the addedcomplication of a substantial cash prize for the best project group resulted in no increase inthe number of disagreements.4.1 Group and individual marksFor simplicity's sake, a single mark is awarded to an entire group. Normally, this isappropriate, as all members of the group have, to a first approximation, contributed equally tothe success of the project. In occasional cases, however, a particular individual hasspectacularly failed to do so. In this past, this has been catered for by a penalty clauseallowing a group to allocate a mark of 0% to one (or more!) of their members on the groundsof their non-contribution, thus giving them some leverage during the course of the work. Sucha decision must be taken unanimously by the group, including the individual concerned. Thisdraconian step has only once been taken, and then against a student who was in the earlystages of deciding to withdraw from the University. There has been, to date, no mechanismfor rewarding conspicuously effective contributions to the work of a group.In order to provide a less dramatic mechanism for accounting for the differential input ofparticular students, we are adopting (from Spring 1999) a mechanism identified in projectEPCoS based on practice at the University of Exeter (Milne, 1998). Each member of a groupfills in a small questionnaire assessing the contribution of all the groups' members (includingthemselves) against their expected contribution to each of the assessment criteria and to thework of the group as a whole. This will allow the identification of individuals who haveunder- or over-performed, and appropriate modification can then be made to the overall markawarded to that individual. The intention is not to differentiate in any great detail betweengroup members, but only to recognise particularly strong or weak contributions by varying themark by ±5 to 10%, thus encouraging a more even application of effort by the group, andameliorating the occasional bad feeling among those who feel that they have "carried" othermembers.5 ConclusionsThe project work described addresses a number of issues in students’ development as softwareengineering practitioners. At base, it represents a fairly typical example of such work aspractised in undergraduate Computer Science programmes. A number of mechanisms havebeen described by which students can be encouraged to reflect on the affective goals of suchwork, rather than on the superficial goals of producing working software:• The use of groups formed for a separate purpose as means of providing constraints onavailable resources.• The negotiation with students of assessment criteria (and appropriate weightings) derivedfrom taught software engineering principles.• The use of these criteria to drive students’ efforts during the work, discouraging them fromadopting simpler (but inappropriate) strategies.• The use of (group) self- and peer-assessment to encourage reflection on the workproduced and the principles learnt.• The (proposed) use of individual self- and peer-evaluation to modify marks awarded togroups, recognising the differential input of individuals without detailed tracking of effort.6 AcknowledgementsProject EPCoS is funded by the Higher Education Funding Councils for England (HEFCE)through their “Fund for the Development of Teaching and Learning” programme.The prize for the best overall performance in the project is provided by Philips ResearchLaboratories in Redhill, UK.REFERENCESSally Fincher, Effective project work in computer science: Project EPCoS. In Mette Knudsenand Torben "Sopper" Vinther, editors, Project Work in University Studies, volume 2, pp195-203. Roskilde University, Denmark, September 1997.Sally Fincher and Ian Utting, Entraining students in professional issues: challenging theirstructures of knowledge. In 6th Improving Student Learning Symposium: ImprovingStudent Learning Outcomes, September 1998.K. Thorn, A Window on Group Formation Factors. In Mike Holcombe et al., editors, Projectsin the Computing Curriculum, pages 217-224. Springer-Verlag, July 1998.W. Milne, Moderation Using Student Input, Project EPCoS internal document,http://www.cs.ukc.ac.uk/national/EPCOS/data_archive/bundles/bundle1.htm, 1998.Ian Utting is a Senior Lecturer in the Computing Laboratory at the University of Kent at Canterbury in the UK,and Director of the UKC Authorized Academic Java Campus. His research interests are in the engineering oflarge-scale distributed systems, and in the teaching of Computer Science. He teaches Object Oriented Design,Software Engineering and Distributed Systems. He is currently working as a member of the HEFCE FDTL-funded project EPCoSThe Computing Laboratory at UKC is ranked as one of the top 10 in the UK for the teaching of ComputerScience, and has recently become the first Sun AAJC in Europe.

A Window on Group Formation Factors.

Effective project work in computer science: Project EPCoS.

Entraining students in professional issues: challenging their structures of knowledge.

Moderation Using Student Input, Project EPCoS internal document, http://www.cs.ukc.ac.uk/national/EPCOS/data_archive/bundles/bundle1.htm,

Negotiated assessment criteria and peer assessment in software engineering group project work: A case study

http://kar.kent.ac.uk/21848/1/negotiated_assessment_criteria_utting.pdf

Negotiated assessment criteria and peer assessment in software engineering group project work: A case study

Abstract

Similar works

Full text

Available Versions

Kent Academic Repository