Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper

Background: Reliable and valid written tests of higher cognitive function are difficult to produce, particularly for the assessment of clinical problem solving. Modified Essay Questions (MEQs) are often used to assess these higher order abilities in preference to other forms of assessment, including multiple-choice questions (MCQs). MEQs often form a vital component of end-of-course assessments in higher education. It is not clear how effectively these questions assess higher order cognitive skills. This study was designed to assess the effectiveness of the MEQ to measure higher-order cognitive skills in an undergraduate institution. Methods: An analysis of multiple-choice questions and modified essay questions (MEQs) used for summative assessment in a clinical undergraduate curriculum was undertaken. A total of 50 MCQs and 139 stages of MEQs were examined, which came from three exams run over two years. The effectiveness of the questions was determined by two assessors and was defined by the questions ability to measure higher cognitive skills, as determined by a modification of Bloom's taxonomy, and its quality as determined by the presence of item writing flaws. Results: Over 50% of all of the MEQs tested factual recall. This was similar to the percentage of MCQs testing factual recall. The modified essay question failed in its role of consistently assessing higher cognitive skills whereas the MCQ frequently tested more than mere recall of knowledge. Conclusion: Construction of MEQs, which will assess higher order cognitive skills cannot be assumed to be a simple task. Well-constructed MCQs should be considered a satisfactory replacement for MEQs if the MEQs cannot be designed to adequately test higher order skills. Such MCQs are capable of withstanding the intellectual and statistical scrutiny imposed by a high stakes exit examination.Edward J Palmer, Peter G Devit

Limitations of student-driven formative assessment in a clinical clerkship. A randomised controlled trial

Background Teachers strive to motivate their students to be self-directed learners. One of the methods used is to provide online formative assessment material. The concept of formative assessment and use of these processes is heavily promoted, despite limited evidence as to their efficacy.Methods Fourth year medical students, in their first year of clinical work were divided into four groups. In addition to the usual clinical material, three of the groups were provided with some form of supplementary learning material. For two groups, this was provided as online formative assessment. The amount of time students spent on the supplementary material was measured, their opinion on learning methods was surveyed, and their performance in summative exams at the end of their surgical attachments was measured.Results The performance of students was independent of any educational intervention imposed by this study. Despite its ready availability and promotion, student use of the online formative tools was poor.Conclusion Formative learning is an ideal not necessarily embraced by students. If formative assessment is to work students need to be encouraged to participate, probably by implementing some form of summative assessment.Edward J Palmer and Peter G Devit

Exploring the impact of cumulative testing on academic performance of undergraduate students in Spain

The final publication is available at Springer via http://dx.doi.org/10.1007/s11092-014-9208-zFrequent testing provides opportunities for students to receive regular feedback and to increase their motivation. It also provides the instructor with valuable information on how course progresses, thus making it possible to solve the problems encountered before it is too late. Frequent tests with noncumulative contents have been widely analysed in the literature with inconclusive results. However, cumulative testing methods have hardly been reported in higher education courses. This paper analyses the effect of applying an assessment method based on frequent and cumulative tests on student performance. Our results show that, when applied to a microeconomics course, students who were assessed by a frequent, cumulative testing approach largely outperformed those assessed with a single final exam.Doménech I De Soria, J.; Blázquez Soriano, MD.; De La Poza, E.; Muñoz Miquel, A. (2015). Exploring the impact of cumulative testing on academic performance of undergraduate students in Spain. Educational Assessment, Evaluation and Accountability. 27(2):153-169. https://doi.org/10.1007/s11092-014-9208-zS153169272Adelman, HS, & Taylor, L. (1990). Intrinsic motivation and school misbehaviour some intervention implications. Journal of Learning Disabilities, 23, 541–550.Biggs, J, & Tang, C. (2007). Teaching for quality learning at university 3rd edn. Open University Press.Boston, C. (2002). The concept of formative assessment. Practical Assessment Research & Evaluation 8.Brown, GA, Bull, J, Pendlebury, M. (1997). Assessing Student Learning in Higher Education, 1st edn. Routledge.Cano, MD. (2011). Students’ involvement in continuous assessment methodologies: a case study for a distributed information systems course. IEEE Transactions on Education, 54, 442–451.Casem, ML (2006). Active learning is not enough. Journal of College Science Teaching, 35.Chen, J, & Lin, TF. (2008). Class attendance and exam performance a randomized experiment. The Journal of Economic Education, 39, 213–227.Chickering, AW, & Gamson, ZF. (1987). Seven principles for good practice in undergraduate education. American Association for Higher Education Bulletin, 39, 3–7.Crooks, TJ. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58, 438–481.De Paola, M, & Scoppa, V. (2011). Frequency of examinations and student achievement in a randomized experiment. Economics of Education Review, 30, 1416–1429.Deck, W. (1998). The effects of frequency of testing on college students in a principles of marketing course, PhD thesis, Virginia Polytechnic Institute and State University. Virginia: Blacksburg.Dempster, FN. (1991). Synthesis of research on reviews and tests. Educational Leadership, 48, 71–76.Dochy, F. (2008). The Edumetric Quality of New Modes of Assessment: Some Issues and Prospects. Assessment, Learning and Judgement in Higher Education. Dordrecht: Springer Netherlands.Eikner, AE, & Montondon, L. (2001). Evidence on factors associated with success in intermediate accounting I. Accounting Educators’ Journal 13.Emerson, TLN, & Mencken, KD. (2011). Homework to require or not? online graded homework and student achievement Perspectives on Economic Education Research 7.Fulkerson, F, & Martin, G. (1981). Effects of exam frequency on student performance, evaluations of instructor, and test anxiety. Teaching of Psychology, 8, 90–93.Furnham, A, & Chamorro-Premuzic, T. (2005). Individual differences and beliefs concerning preference for university assessment methods. Journal of Applied Social Psychology, 35, 1968–1994.Gibbs, G, & Simpson, C. (2005). Conditions under which assessment supports students’ learning Learning and Teaching in Higher Education 1 (August 5, 2011)3–31.Haberyan, KA. (2003). Do weekly quizzes improve student performance on general biology exams?. The American Biology Teacher, 65, 110–114.Kling, N, McCorkle, D, Miller, C, Reardon, J. (2005). The impact of testing frequency on student performance in a marketing course. Journal of Education for Business, 81, 67–72.Kuh, GD (2003). What we’re learning about student engagement from NSSE Change 35.Kuo, T, & Simon, A. (2009). How many tests do we really need. College Teaching, 57, 156–160.Leeming, FC. (2002). The exam-a-day procedure improves performance in psychology classes. Teaching of Psychology, 29, 210–212.Lumsden, KG, Scott, A, Becker, WE. (1987). The economics student reexamined Male-female differences in comprehension. Journal of Economic Education, 18, 365–375.Marriott, P. (2009). Students’ evaluation of the use of online summative assessment on an undergraduate financial accounting module. British Journal of Educational Technology, 40, 237–254.Marriott, P, & Lau, A. (2008). The use of on-line summative assessment in an undergraduate financial accounting course. Journal of Accounting Education, 26, 73–90.McNabb, R, Pal, S, Sloane, P. (2002). Gender differences in educational attainment. the case of university students in england and wales. Economica, 69, 481–503.Miller, F. (1987). Test frequency, student performance and teacher evaluation in the basic marketing class. Journal of Marketing Education, 9, 14–19.Nicol, DJ, & Macfarlane Dick, D. (2006). Formative assessment and self-regulated learning, A model and seven principles of good feedback practice. Studies in Higher Education, 31, 199–218.Nowell, C, & Alston, RM. (2007). I thought I got an A! Overconfidence across the economics curriculum. The Journal of Economic Education, 38, 131–142.Race, P (1995). The art of assessing 1 New Academic 4.Scriven, M. (1967). The Methodology of Evaluation, vol 1 (pp. 39–83). Chicago: Rand McNally.Skinner, BF. (1974). About behaviorism. New York: Alfred A Knopf.Taras, M. (2005). Assessment - summative and formative - some theoretical reflections. British Journal of Educational Studies, 53, 466–478.Trotter, E. (2006). Student perceptions of continuous summative assessment. Assessment & Evaluation in Higher Education, 31, 505–521.Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher Education, 45, 477–501

Quantum Measurement Theory in Gravitational-Wave Detectors

The fast progress in improving the sensitivity of the gravitational-wave (GW) detectors, we all have witnessed in the recent years, has propelled the scientific community to the point, when quantum behaviour of such immense measurement devices as kilometer-long interferometers starts to matter. The time, when their sensitivity will be mainly limited by the quantum noise of light is round the corner, and finding the ways to reduce it will become a necessity. Therefore, the primary goal we pursued in this review was to familiarize a broad spectrum of readers with the theory of quantum measurements in the very form it finds application in the area of gravitational-wave detection. We focus on how quantum noise arises in gravitational-wave interferometers and what limitations it imposes on the achievable sensitivity. We start from the very basic concepts and gradually advance to the general linear quantum measurement theory and its application to the calculation of quantum noise in the contemporary and planned interferometric detectors of gravitational radiation of the first and second generation. Special attention is paid to the concept of Standard Quantum Limit and the methods of its surmounting.Comment: 147 pages, 46 figures, 1 table. Published in Living Reviews in Relativit

SH3 Domain-Peptide Binding Energy Calculations Based on Structural Ensemble and Multiple Peptide Templates

SH3 domains mediate signal transduction by recognizing short peptides. Understanding of the driving forces in peptide recognitions will help us to predict the binding specificity of the domain-peptide recognition and to understand the molecular interaction networks of cells. However, accurate calculation of the binding energy is a tough challenge. In this study, we propose three ideas for improving our ability to predict the binding energy between SH3 domains and peptides: (1) utilizing the structural ensembles sampled from a molecular dynamics simulation trajectory, (2) utilizing multiple peptide templates, and (3) optimizing the sequence-structure mapping. We tested these three ideas on ten previously studied SH3 domains for which SPOT analysis data were available. The results indicate that calculating binding energy using the structural ensemble was most effective, clearly increasing the prediction accuracy, while the second and third ideas tended to give better binding energy predictions. We applied our method to the five SH3 targets in DREAM4 Challenge and selected the best performing method

Hsmar1 transposition is sensitive to the topology of the transposon donor and the target

Hsmar1 is a member of the Tc1-mariner superfamily of DNA transposons. These elements mobilize within the genome of their host by a cut-and-paste mechanism. We have exploited the in vitro reaction provided by Hsmar1 to investigate the effect of DNA supercoiling on transposon integration. We found that the topology of both the transposon and the target affect integration. Relaxed transposons have an integration defect that can be partially restored in the presence of elevated levels of negatively supercoiled target DNA. Negatively supercoiled DNA is a better target than nicked or positively supercoiled DNA, suggesting that underwinding of the DNA helix promotes target interactions. Like other Tc1-mariner elements, Hsmar1 integrates into 5′-TA dinucleotides. The direct vicinity of the target TA provides little sequence specificity for target interactions. However, transposition within a plasmid substrate was not random and some TA dinucleotides were targeted preferentially. The distribution of intramolecular target sites was not affected by DNA topology

Technology enhanced assessment in complex collaborative settings

Building upon discussions by the Assessment Working Group at EDUsummIT 2013, this article reviews recent developments in technology enabled assessments of collaborative problem solving in order to point out where computerised assessments are particularly useful (and where non-computerised assessments need to be retained or developed) while assuring that the purposes and designs are transparent and empowering for teachers and learners. Technology enabled assessments of higher order critical thinking in a collaborative social context can provide data about the actions, communications and products created by a learner in a designed task space. Principled assessment design is required in order for such a space to provide trustworthy evidence of learning, and the design must incorporate and take account of the engagement of the audiences for the assessment as well as vary with the purposes and contexts of the assessment. Technology enhanced assessment enables in-depth unobtrusive documentation or ‘quiet assessment’ of the many layers and dynamics of authentic performance and allows greater flexibility and dynamic interactions in and among the design features. Most important for assessment FOR learning, are interactive features that allow the learner to turn up or down the intensity, amount and sharpness of the information needed for self-absorption and adoption of the feedback. Most important in assessment OF learning, are features that compare the learner with external standards of performance. Most important in assessment AS learning, are features that allow multiple performances and a wide array of affordances for authentic action, communication and the production of artefacts

Light interception principally drives the understory response to boxelder invasion in riparian forests

Since several decades, American boxelder (Acer negundo) is replacing white willow (Salix alba) riparian forests along southern European rivers. This study aims to evaluate the consequences of boxelder invasion on understory community in riparian areas. We determined the understory species richness, composition and biomass in boxelder and white willow stands located in three riparian forests, representative of three rivers with distinct hydrological regimes. We investigated correlation of these variables to soil moisture and particle size, main soil nutrient stocks, potential nitrification and denitrification, tree canopy cover and photosynthetic active radiation (PAR) at the ground level. A greenhouse experiment was then conducted to identify the causal factors responsible for changes in the understory. The effect of soil type, PAR level and water level on the growth and the biomass production of Urtica dioica were examined. A lower plant species richness and biomass, and a modification of community composition were observed for boxelder understory in all sites, regardless of their environmental characteristics. The strongest modification that follows boxelder invasion was the decline in U. dioica, the dominant species of the white willow forest understory. These differences were mainly correlated with a lower incident PAR under boxelder canopy. The greenhouse experiment identified PAR level as the main factor responsible for the changes in U. dioica stem number and biomass. Our results indicate that adult boxelder acts as an ecosystem engineer that decreases light availability. The opportunistic invasion by boxelder leads to important understory changes, which could alter riparian ecosystem functioning