Smailes, Joanne

English

Northumbria University Research Portal

„Problem‟ Subject?  Reconsider your approach to Computer Based Assessment:  A case study in Business Modelling  Joanne Smailes, Northumbria University What makes a  subject  a problem?   (Warwick and Ottewell, 2004)  Data Analysis within Business Courses Core component in the curriculum Yes Students perceive the subject as more difficult that other subjects Yes Students dismiss content as irrelevant Yes Prior student experience may have an adverse impact on percep-tions   Yes results  in student fear or  nervousness of ability Studied by large numbers from diverse backgrounds  Yes including those where English is a  second language Policies reduce contact time   Yes to around 10% of notional  student workload Subject paradigm differs to other paradigms within the subject  Yes Assessment for Learning?  Assessment has a “profound influence” on quality and quantity of engagement with student learning tasks -  Gibbs and Simpson (2004-5) Assessment is not only what students regard as important but often takes centre stage in the management of their time - Brown (2001) In order for assessment to be at its most effective it should provide a powerful formative function and  reduce the domination of the summative function  - McDowell et al. (2005)   Continuous assessment as less anxiety provoking system.  IF!  non-threatening, give a realis-tic workload, help pace student learning yet allow for „slow‟ learning  and includes plenty of regular formative feedback - (Rust 2002) Blackboard VLE was introduced at Northumbria University in February 2001.  Author carefully  designed a number of “formative self-tests” using  question formats such as multiple answer, fill-in-the blank, matching etc. to ensure coverage of high –order understanding in  data analysis curriculum.  These tests became the foundational elements in investigating two computer-based assessment  strategies (over two student cohorts).   Using CAA purely as a formative assessment resource  Using CAA as a combined formative and summative strategy. The data analysis module runs over a two-semester period. Using the existing databank of questions  as a starting point,  ten tests were designed to be distributed evenly spaced across this time period.  All start and end dates automatically controlled by the Blackboard VLE.  For the summative cycle, students were informed that three of the ten tests would be randomly selected as a summative mark. How can assessment be made both  efficient and effective ?   Computer-aided assessment stands out as the most cited  practical and strategic solution (Harvey and Mogey, 1999)   Large student numbers 800+  Could be set up to allow for „slow‟ learning  (Rust, 2002)  Suitable bridge between formative and summative  (Bull and McKenna, 2004)  Feedback is required within “minutes” to be most effec-tive (Cowan, 2003)  Anonymity, formative purpose timeliness crucial factors  (Weaver, 2003)    Quantity and Distribution of Effort Student Response to Feedback 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%12345UK International1. I would learn more if I received more feedback 2. The feedback helped me to understand things better * 3. The feedback showed me how to do better next time 4. I used the feedback to go back over what I have done in the test * 5. I tended to only read the marks * (* Statistical difference also found in item scores, where p<0.05) 020406080100% agreeingA B C D E FFormative and Summative Formative onlyA. On the module I got plenty of feedback on how I was doing B. The feedback came very quickly C. There was hardly any feedback on the tests  D. When I got things wrong or misunderstand them I didn‟t receive  much guidance in what to do about it. E. I would learn more if I received more feedback F. Whatever feedback I got came back too late to be useful Feedback During the formative only cycle, every group taking part indicated that they needed to be “forced” into participating in the formative quizzes: “The quizzes need to contribute to the final mark” “Participation in tests should be enforced” “The quizzes should count towards your final mark”  Previous A Previous B Formative & Summative Formative only Mean 47.9 49.7 58.1 54.8 Std. Deviation 19.2 22.5 20.4 20.3 Median 46 47 64 60 Module Results  RECOMMENDATIONS The use of multiple-choice questions in objective tests should be kept to a bare minimum.  Fill-in the blank, matching and ordering question types are more effective in testing high order thinking skills.  In line with student request, a reward or penalty based system for participation should be introduced.   Whenever a quiz is used feedback for incorrect answers must be supplied.  Importantly this must be given in the form of advice on how common errors and misinterpretations may occur. The instantaneous feedback should be used to direct the students to a variety of other forms of learning (e.g. a variety of text books, websites, additional quizzes, discussion boards).  It is felt that no feedback is necessary for cor-rect answers. When technology permits, avoid the use of grades, marks and supplying the “correct” answer. Encourage students to become involved and work with peers.  For example, asking students to suggest questions or activities for inclusion within the quizzes could do this.  Ask students to suggest learning resources, which they found useful and incorporate these into the  feedback provided. References   Brown, G. (2001). Assessment: A Guide for Lecturers. York, LTSN Generic Centre. Bull, J. and C. McKenna (2004). Blueprint for Computer-Assisted Assessment, London, Routledge Farmer. Cowan, J. (2003). "Assessment for learning - giving timely and effective feedback." Exchange Spring 2003(4): 21-22. Gibbs, G. and C. Simpson (2003). Measuring the response of students to assessment: the Assessment Experience Questionnaire.  11th International Improving Student Learning Symposium,, Hinckley, England. Gibbs, G. and C. Simpson (2004-5). "Does your assessment support your students' learning?" Learning and Teaching in Higher Education  1(1): 3-31. Gibbs, G., C. Simpson, et al. (2003). Improving student learning through changing assessment - a conceptual and practical framework. EARLI, Padova. Harvey, J. and N. Mogey (1999). Pragmatic issues when integrating technology into the assessment of students. Computer-Assisted  Assessment in Higher Education,. S. Brown, P. Race and J. Bull, Kogan: 7-19. McDowell, L., K. Sambell, et al. (2005). Assessment for Learning Current Exemplars of Practice from the Centre of Excellence in Teaching and Learning. Newcastle upon Tyne, Northumbria University. Rust, C. (2002). "The impact of assessment on student learning.  How can the research literature practically help to inform the  development of departmental assessment strategies and learner-centred assessment practices?" Active Learning 3(2): 145-158. Warwick, P. and R. Ottewill (2004). "How can 'problem subjects' be made less of a problem." Teaching in Higher Education 9(3): 337-347. Weavers, C. (2003). "What do students want from assessment." Exchange Spring 2003(4): 12-13. 11 Conditions of Assessment to Support Student Learning As reported in (Gibbs, Simpson et al. 2003)  Formative only Formative &  Summative  1. Assessed tasks capture sufficient study time and effort    2. These tasks distribute student effort evenly across topics and weeks   3. These tasks engage students in productive learning activity   4. Assessment communicates clear and high expectations to students   5. Sufficient feedback is provided   6. The feedback is provided quickly enough to be of use to students   7. Feedback focuses on learning rather than on marks or student themselves    8. Feedback is linked to the purpose of the assignment and to criteria   9. Feedback is understandable to students, given their sophistication   10. Feedback is received by students and attended to   11. Feedback is acted upon by students to improve their work or their learning.   Differences across Cycles    International v UK     

Assessed tasks capture sufficient study time and effort

Assessment communicates clear and high expectations to students

Assessment: A Guide for

Feedback focuses on learning rather than on marks or student themselves

Feedback is acted upon by students to improve their work or their learning. Differences across Cycles International v

Feedback is linked to the purpose of the assignment and to criteria

Feedback is received by students and attended to

Feedback is understandable to students, given their sophistication

Sufficient feedback is provided

The feedback is provided quickly enough to be of use to students

These tasks distribute student effort evenly across topics and weeks

These tasks engage students in productive learning activity

Problem subject? : reconsider your approach to computer based assessment : a case in business modelling

Northumbria Research Link

Northumbria Research Link
Citation: Smailes, Joanne (2006) Problem subject? : reconsider your approach to computer based 
assessment  :  a  case  in  business  modelling.  In:  1st  Pedagogical  Research  in  Higher  Education 
(PRHE) conference - 'Pedagogical Research : Enhancing student success', 2-3 May 2006, Liverpool. 
URL: 
This version was downloaded from Northumbria Research Link: http://nrl.northumbria.ac.uk/3197/
Northumbria University has developed Northumbria Research Link (NRL) to enable users to access 
the University’s research output. Copyright © and moral rights for items on NRL are retained by the 
individual author(s) and/or other copyright owners.  Single copies of full items can be reproduced, 
displayed or performed, and given to third parties in any format or medium for personal research or 
study, educational, or not-for-profit purposes without prior permission or charge, provided the authors, 
title and full bibliographic details are given, as well as a hyperlink and/or URL to the original metadata 
page. The content must not be changed in any way. Full items must not be sold commercially in any  
format or medium without formal permission of the copyright holder.  The full policy is available online: 
http://nrl.northumbria.ac.uk/pol  i cies.html  
This  document  may differ  from the  final,  published version of  the research  and has been made 
available online in accordance with publisher policies. To read and/or cite from the published version 
of the research, please visit the publisher’s website (a subscription may be required.)
                        
„Problem‟ Subject?  
Reconsider your approach to Computer Based Assessment:  
A case study in Business Modelling  
Joanne Smailes, Northumbria University 
What makes a  subject  a problem?   
(Warwick and Ottewell, 2004)  
Data Analysis within 
Business Courses 
Core component in the curriculum Yes 
Students perceive the subject as more difficult that other subjects Yes 
Students dismiss content as irrelevant Yes 
Prior student experience may have an adverse impact on percep-
tions  
 Yes 
results  in student fear or  
nervousness of ability 
Studied by large numbers from diverse backgrounds  
Yes 
including those where English is a  
second language 
Policies reduce contact time  
 Yes 
to around 10% of notional  
student workload 
Subject paradigm differs to other paradigms within the subject  Yes 
Assessment for Learning? 
 
Assessment has a “profound influence” on quality and quantity of engagement with student 
learning tasks -  Gibbs and Simpson (2004-5) 
Assessment is not only what students regard as important but often takes centre stage in 
the management of their time - Brown (2001) 
In order for assessment to be at its most effective it should provide a powerful formative 
function and  reduce the domination of the summative function  - McDowell et al. (2005)   
Continuous assessment as less anxiety provoking system.  IF!  non-threatening, give a realis-
tic workload, help pace student learning yet allow for „slow‟ learning  and includes plenty of 
regular formative feedback - (Rust 2002) 
Blackboard VLE was introduced at Northumbria University in February 2001.  Author carefully  
designed a number of “formative self-tests” using  question formats such as multiple answer, fill-in-the 
blank, matching etc. to ensure coverage of high –order understanding in  data analysis curriculum.  
These tests became the foundational elements in investigating two computer-based assessment  
strategies (over two student cohorts).  
 Using CAA purely as a formative assessment resource 
 Using CAA as a combined formative and summative strategy. 
The data analysis module runs over a two-semester period. Using the existing databank of questions  
as a starting point,  ten tests were designed to be distributed evenly spaced across this time period.  All start and end dates automatically controlled 
by the Blackboard VLE.  For the summative cycle, students were informed that three of the ten tests would be randomly selected as a summative 
mark. 
How can assessment be made both 
 efficient and effective ?  
 
Computer-aided assessment stands out as the most cited  
practical and strategic solution (Harvey and Mogey, 1999) 
 
 Large student numbers 800+ 
 Could be set up to allow for „slow‟ learning  
(Rust, 2002) 
 Suitable bridge between formative and summative  
(Bull and McKenna, 2004) 
 Feedback is required within “minutes” to be most effec-
tive (Cowan, 2003) 
 Anonymity, formative purpose timeliness crucial factors  
(Weaver, 2003)   
 
Quantity and Distribution of Effort 
Student Response to Feedback 
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%
1
2
3
4
5
UK International
1. I would learn more if I received more feedback 
2. The feedback helped me to understand things better * 
3. The feedback showed me how to do better next time 
4. I used the feedback to go back over what I have done in the test * 
5. I tended to only read the marks * 
(* Statistical difference also found in item scores, where p<0.05) 
0
20
40
60
80
100
%
 a
g
re
e
in
g
A B C D E F
Formative and Summative Formative only
A. On the module I got plenty of feedback on how I was doing 
B. The feedback came very quickly 
C. There was hardly any feedback on the tests  
D. When I got things wrong or misunderstand them I didn‟t receive  
much guidance in what to do about it. 
E. I would learn more if I received more feedback 
F. Whatever feedback I got came back too late to be useful 
Feedback 
During the formative only cycle, every group taking part 
indicated that they needed to be “forced” into 
participating in the formative quizzes: 
“The quizzes need to contribute to the final mark” 
“Participation in tests should be enforced” 
“The quizzes should count towards your final mark” 
 Previous A Previous B Formative & 
Summative 
Formative 
only 
Mean 47.9 49.7 58.1 54.8 
Std. Deviation 19.2 22.5 20.4 20.3 
Median 46 47 64 60 
Module Results 
 
RECOMMENDATIONS 
The use of multiple-choice questions in objective tests should be kept to a bare minimum.  Fill-in the blank, matching and ordering question 
types are more effective in testing high order thinking skills.  
In line with student request, a reward or penalty based system for participation should be introduced.   
Whenever a quiz is used feedback for incorrect answers must be supplied.  Importantly this must be given in the form of advice on how 
common errors and misinterpretations may occur. The instantaneous feedback should be used to direct the students to a variety of other 
forms of learning (e.g. a variety of text books, websites, additional quizzes, discussion boards).  It is felt that no feedback is necessary for cor-
rect answers. 
When technology permits, avoid the use of grades, marks and supplying the “correct” answer. 
Encourage students to become involved and work with peers.  For example, asking students to suggest questions or activities for inclusion 
within the quizzes could do this.  Ask students to suggest learning resources, which they found useful and incorporate these into the  
feedback provided. 
References  
 
Brown, G. (2001). Assessment: A Guide for Lecturers. York, LTSN Generic Centre. 
Bull, J. and C. McKenna (2004). Blueprint for Computer-Assisted Assessment, London, Routledge Farmer. 
Cowan, J. (2003). "Assessment for learning - giving timely and effective feedback." Exchange Spring 2003(4): 21-22. 
Gibbs, G. and C. Simpson (2003). Measuring the response of students to assessment: the Assessment Experience Questionnaire.  
11th International Improving Student Learning Symposium,, Hinckley, England. 
Gibbs, G. and C. Simpson (2004-5). "Does your assessment support your students' learning?" Learning and Teaching in Higher Education  
1(1): 3-31. 
Gibbs, G., C. Simpson, et al. (2003). Improving student learning through changing assessment - a conceptual and practical framework. 
EARLI, Padova. 
Harvey, J. and N. Mogey (1999). Pragmatic issues when integrating technology into the assessment of students. Computer-Assisted  
Assessment in Higher Education,. S. Brown, P. Race and J. Bull, Kogan: 7-19. 
McDowell, L., K. Sambell, et al. (2005). Assessment for Learning Current Exemplars of Practice from the Centre of Excellence in Teaching 
and Learning. Newcastle upon Tyne, Northumbria University. 
Rust, C. (2002). "The impact of assessment on student learning.  How can the research literature practically help to inform the  
development of departmental assessment strategies and learner-centred assessment practices?" Active Learning 3(2): 145-158. 
Warwick, P. and R. Ottewill (2004). "How can 'problem subjects' be made less of a problem." Teaching in Higher Education 9(3): 337-347. 
Weavers, C. (2003). "What do students want from assessment." Exchange Spring 2003(4): 12-13. 
11 Conditions of Assessment to Support Student Learning 
As reported in (Gibbs, Simpson et al. 2003) 
 Formative only Formative & 
 Summative  
1. Assessed tasks capture sufficient study time and effort    
2. These tasks distribute student effort evenly across topics and weeks   
3. These tasks engage students in productive learning activity   
4. Assessment communicates clear and high expectations to students   
5. Sufficient feedback is provided   
6. The feedback is provided quickly enough to be of use to students   
7. Feedback focuses on learning rather than on marks or student themselves  
  
8. Feedback is linked to the purpose of the assignment and to criteria   
9. Feedback is understandable to students, given their sophistication   
10. Feedback is received by students and attended to   
11. Feedback is acted upon by students to improve their work or their learning.   
Differences across Cycles 
 
 
 
International v UK  
 
 
 


http://nrl.northumbria.ac.uk/id/eprint/3197/1/Smailes%20J%20-%20Problem%20subject%20-%20poster%20for%20NRL.pdf

Problem subject? : reconsider your approach to computer based assessment : a case in business modelling

Abstract

Similar works

Full text

Available Versions

Northumbria University Research Portal

Northumbria Research Link