Joining the dots: Conditional pass and programmatic assessment enhances recognition of problems with professionalism and factors hampering student progress

<p>Abstract</p> <p>Background</p> <p>Programmatic assessment that looks across a whole year may contribute to better decisions compared with those made from isolated assessments alone. The aim of this study is to describe and evaluate a programmatic system to handle student assessment results that is aligned not only with learning and remediation, but also with defensibility. The key components are standards based assessments, use of "Conditional Pass", and regular progress meetings.</p> <p>Methods</p> <p>The new assessment system is described. The evaluation is based on years 4-6 of a 6-year medical course. The types of concerns staff had about students were clustered into themes alongside any interventions and outcomes for the students concerned. The likelihoods of passing the year according to type of problem were compared before and after phasing in of the new assessment system.</p> <p>Results</p> <p>The new system was phased in over four years. In the fourth year of implementation 701 students had 3539 assessment results, of which 4.1% were Conditional Pass. More in-depth analysis for 1516 results available from 447 students revealed the odds ratio (95% confidence intervals) for failure was highest for students with problems identified in more than one part of the course (18.8 (7.7-46.2) p < 0.0001) or with problems with professionalism (17.2 (9.1-33.3) p < 0.0001). The odds ratio for failure was lowest for problems with assignments (0.7 (0.1-5.2) NS). Compared with the previous system, more students failed the year under the new system on the basis of performance during the year (20 or 4.5% compared with four or 1.1% under the previous system (p < 0.01)).</p> <p>Conclusions</p> <p>The new system detects more students in difficulty and has resulted in less "failure to fail". The requirement to state conditions required to pass has contributed to a paper trail that should improve defensibility. Most importantly it has helped detect and act on some of the more difficult areas to assess such as professionalism.</p

The explication of quality standards in self-evaluation

Education aiming at students’ competence development asks for new assessment methods. The quality of these methods needs to be assured using adapted quality criteria and accompanying standards. As such standards are not widely available, this study sets out to examine what level of compliance with quality criteria stakeholders consider satisfactory. Two professional education programmes specified the implicit standards they applied in a self-evaluation procedure designed to evaluate the quality of their Competence Assessment Programs (CAPs). They specified similar cut-off scores, but different descriptive standards. Analysis revealed that this was due to theIR experience with competence-based education and the quality of their own CAP, but influences of the selected method and the understanding of the quality criteria were also found. As such, the specified standards are local, but meaningful for the programmes’ quality assurance. Implications for self-evaluation and standard-setting procedures are discussed

Does self-evaluation of CAP quality lead to improvements in assessment and curriculum

Technological developments and computerisation influence many jobs. The goal of this study is to identify to what extent employees need an increased and/or different understanding of science and technology to function in their jobs than is currently taught in schools. Whereas previous research tended to focus on very general skills (e.g., problem solving, communication) or long detailed lists of content knowledge, the focus of this study is on competences such as modelling and visualising. Companies most influenced by technological developments were selected in cooperation with National Centres of Expertise on vocational education. They were asked in which companies (1) more and/or different scientific and technological competences, and (2) creativity and flexibility are required. We focused on jobs at the level of senior secondary vocational education, as these jobs are most prone to being outsourced or taken over by computers. Employees working in the different companies were asked what knowledge and skills they use during their work, for example when working with machines and computers. Preliminary results show the influence of technological developments on almost all jobs. Required knowledge and skills are for example: data-analysis, the use of graphs and thinking beyond one's own job. The need for flexibility, creativity and insight in ‘black boxes' seems to depend on the level of education. Theory and aimsThe increased use of information- and communication technology and the influence of scientific and technological developments have caused many countries to redefine the key competences for adequately functioning professionals. This study specifically focuses on the needs of future employees: what competences in the domains of science and technology do they need to function adequately in their jobs, now and the future?Previous studies tend to focus on very general skills such as problem solving and communication (e.g., Holbrook &amp; Rannikmae, 2007), which do not provide any specification of what should be taught. Other studies provide long and detailed lists of content to be taught in schools (e.g., AAAS, 1993). These lists run the risk of quickly becoming outdated as the amount of technical information is doubling every two years (Binkley et al., 2010). This begs the question whether we can identify competences that are general in the sense that we may expect them to stay valuable for a long time, but at the same time not too general to offer directions for curricula. This study tries to answer this question by interviewing employees in companies that are subject to technological developments. In the literature, two domains provide input to this study. First, a number of studies focus on 21st century-skills, lifelong learning competences, or key skills. For example, Binkley et al. (2010) analysed curriculum and assessment frameworks around the world, and identified ten competences, including creativity, critical thinking and ICT literacy. Voogt and Pareja-Roblin (2010) and Dede (2009) reviewed different reports about 21st century skills, generally relying on private/business initiatives. These reports mention skills such as communication, ICT literacy, social awareness, and creativity. Problematic is, however, that explicit links to educational levels are missing and the educational community hardly participates in this debate. These reports thus provide little direction with regard to the content of education. Second, we reviewed the literature about knowledge and skills used in the workplace. Here, hardly any studies exist on the use of science and technology at the workplace. There are, however, studies on mathematics that are general enough to be useful for science and technology as well (e.g., Bakker et al., 2006; Pozzi et al., 1998). These studies show the importance of: (1) knowing what processes are ‘hidden' in computers or machines, and (2) analysing relationships between variables, based on quantitative data. Data and MethodsParticipantsThe context of this study was vocational education in the Netherlands, preparing students for a job at levels ranging from assistant worker to middle management. This middle-level job was chosen, as it is likely to be most affected by technological changes, while employees are not specifically educated in this domain (Levi &amp; Murnane, 2005). To identify companies most influenced by technological developments, interviews were conducted with National Centres of Expertise, who develop national qualification profiles for the different branches. Six interviews were conducted with representatives from: animal and plant care, car mechanics, audician/optician, graphical design, commercials/presentation, and nursing/care. Seven interviews were conducted with employees of these branches: a farmer using robots, employees of Ford cards, a company developing logistical systems, an ICT desk, an outsourcing company, and an audician. In November/December 2010, more interviews will be conducted in different branches, and nursing specifically. InterviewsRepresentatives of the branches were asked to describe developments with regard to: (1) the amount of science and technology, (2) the content of science and technology, and (3) flexibility and creativity needed to function on the job. Depending on their function, the employees were asked:- what kind of machines and computer programs they work with;- if they need to know the ‘invisible processes' inside;- how they are trained to work with new machines and computers;- what they do in case of an unexpected outcome or problem.All participants were asked to describe job situations in which scientific and technological knowledge and skills are used. Preliminary resultsFull results, including the remaining interviews and more examples from concrete job situations, are presented at the conference.- Required knowledge and skills: most employees need some basic knowledge of science and technology. For example, audicians need knowledge of the auditory organs.- Flexibility and creativity: at lower levels, employees are not expected to suggest improvements or solve non-standard problems. An exception is ‘defence/tank' mechanics. They need to know the exact working of tank engines, enabling creativity in emergencies.- Insight in black boxes: opinions seem to differ here. In general, employees can work with machines without knowing ‘what is going on inside'. For example, graphical designer can use software to make sketches without knowing how colours mix. Theoretical and educational significance. Most jobs seem to be influenced by computerisation and technology, leading to changed requirements in terms of knowledge and skills, creativity and insight in black boxes. The identification of these changed requirements could guide curriculum adaptations, better preparing young people for their future jobs. More research seems warranted on the use of science and technology at the workplace (following studies on mathematics). This study is a first step in this direction

Development of engineering identity in the engineering curriculum in Dutch higher education : an explorative study from the teaching staff perspective

How do engineering students develop a professional identity during the course of the curriculum? What are the development mechanisms and important conditions? In an exploratory study among teachers the authors tried to find out whether the development of engineering identity can be understood by using the theoretical models of Ibarra and Sullivan. The results showed that the development of engineering identity is boosted during the internship in industry in the third year. Furthermore, the theoretical models could be recognised in the findings and four interaction types in internships, with significant differences in effect on identity development, could be identified if two dimensions were used: (1) industry supervisors’ perception of students; (2) the professional responsibility awarded. Some recommendations for curriculum improvement could be made; for example, introducing projects into the curriculum to stimulate identity development from the start. Further research is required on students’ preparation for internships

Reflection during portfolio-based conversations

This study aims to explore the relationship between the occurrence of reflection (and non-reflection) and thinking activities (e.g., orientating, selecting, analysing) during portfolio-based conversations. Analysis of 21 transcripts of portfolio-based conversations revealed that 20% of the segments were made up of reflection (content reflection (6%), process reflection (10%), and premise reflection (4%)). The thinking activities comparing, analysing and concluding occurred significantly more often during reflection than during non-reflection. Orientating on the task, selecting and describing, occurred significantly less often during reflection. The outcomes show that the occurrence of certain thinking activities can be an indication of reflection

Determining the quality of Competence Assessment Programs: A self-evaluation procedure

As assessment methods are changing, the way to determine their quality needs to be changed accordingly. This article argues for the use Competence Assessment Programs (CAPs), combinations of traditional tests and new assessment methods which involve both formative and summative assessments. To assist schools in evaluating their CAPs, a self-evaluation procedure was developed, based on 12 quality criteria for CAPs developed in earlier studies. A self-evaluation was chosen as it is increasingly used as an alternative to external evaluation. The CAP self-evaluation is carried out by a group of functionaries from the same school and comprises individual self-evaluations and a group interview. The CAP is rated on the 12 quality criteria and a piece of evidence is asked for to support these ratings. In this study, three functionaries from eight schools (N = 24) evaluated their CAP using the self-evaluation procedure. Results show that the group interview was very important as different perspectives on the CAP are assembled here into an overall picture of the CAP's quality. Schools seem to use mainly personal experiences to support their ratings and need to be supported in the process of carrying out a self-evaluation

Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks

Because learning and instruction are increasingly competence-based, the call for assessment methods to adequately determine competence is growing. Using just one single assessment method is not sufficient to determine competence acquisition. This article argues for Competence Assessment Programmes (CAPs), consisting of a combination of different assessment methods, including both traditional and new forms of assessment. To develop and evaluate CAPs, criteria to determine their quality are needed. Just as CAPs are combinations of traditional and new forms of assessment, criteria used to evaluate CAP quality should be derived from both psychometrics and edumetrics. A framework of 10 quality criteria for CAPs is presented, which is then compared to Messick's framework of construct validity. Results show that the 10-criterion framework partly overlaps with Messick's, but adds some important new criteria, which get a more prominent place in quality control issues in competence-based education