An architecture for systematic tracking of skills and competence level progression in computer science

A typical Computer Science degree is three to five years long, consists of four to six subjects per semester, and two semesters per year. A student enrolled in such a degree is expected to learn both discipline-specific skills and transferable generic skills. These skills are to be taught in a progressive sequence through the duration of the degree. As the student progresses through the subjects and semesters of a degree, his skill portfolio and competence level for each skill is expected to grow. Effectively modeling these curriculum skills, mapping them to assessment tasks across subjects of a degree, and measuring the progression in learner competence level is, largely, still an unsolved problem. Previous work at this scale is limited. This systematic tracking of skills and competence is crucial for effective quality control and optimization of degree structures. Our main contribution is an architecture for a curriculum information management system to facilitate this systematic tracking of skill and competence level progression in a Computer Science context

A conceptual model for re ecting on expected learning vs. demonstrated student performance

© 2013, Australian Computer Society, Inc. Educators are faced with many challenging questions in designing an effective curriculum. What prerequisite knowledge do students have before commencing a new subject? At what level of mastery? What is the spread of capabilities between bare-passing students vs. the top-performing group? How does the intended learning specification compare to student performance at the end of a subject? In this paper we present a conceptual model that helps in answering some of these questions. It has the following main capabilities: capturing the learning specification in terms of syllabus topics and outcomes; capturing mastery levels to model progression; capturing the minimal vs. aspirational learning design; capturing confidence and reliability metrics for each of these mappings; and finally, comparing and re ecting on the learning specification against actual student performance. We present a web-based implementation of the model, and validate it by mapping the final exams from four programming subjects against the ACM/IEEE CS2013 topics and outcomes, using Bloom's Taxonomy as the mastery scale. We then import the itemised exam grades from 632 students across the four subjects and compare the demonstrated student performance against the expected learning for each of these. Key contributions of this work are the validated conceptual model for capturing and comparing expected learning vs. demonstrated performance, and a web-based implementation of this model, which is made freely available online as a community resource

Progoss: Mastering the Curriculum

Abstract: In education, we need to design effective degree programs of study that meet authoritative curricula guidelines. This is challenging because of the size of the curriculum and complexity of degree program structures. When dealing with data of this size and complexity, traditional spreadsheets are a clumsy way of storing the data. A database is a better option, especially when the database is accessible over the web. We created ProGoSs to effectively tackle this complexity. ProGoSs is a web-based system that maps curricula learning goals and mastery levels to individual assessment tasks across entire degree programs. ProGoSs enables academics to answer important questions such as: Does our degree teach the essential core defined in a recommended curriculum? Where in our degree are particular parts of the recommended curriculum taught? Does our degree ensure a solid progression in building skills? Where and how do we assess the learning achieved by bare-pass students on particular parts of the recommended curriculum? We present the design and implementation of ProGoSs and report on its evaluation by mapping multiple programming subjects from multiple universities to the ACM/IEEE Computer Science 2013 topics and learning objectives. This includes a mapping to various levels of Bloomâs Taxonomy to capture mastery

Mastering cognitive development theory in computer science education

To design an effective computer science curriculum, educators require a systematic method of classifying the difficulty level of learning activities and assessment tasks. This is important for curriculum design and implementation and for communication between educators. Different educators must be able to use the method consistently, so that classified activities and assessments are comparable across the subjects of a degree, and, ideally, comparable across institutions. One widespread approach to supporting this is to write learning objects in terms of Bloom's Taxonomy. This, or other such classifications, is likely to be more effective if educators can use them consistently, in the way experts would use them. To this end, we present the design and evaluation of our online interactive web-based tutorial system, which can be configured and used to offer training in different classification schemes. We report on results from three evaluations. First, 17 computer science educators complete a tutorial on using Bloom's Taxonomy to classify programming examination questions. Second, 20 computer science educators complete a Neo-Piagetian tutorial. Third evaluation was a comparison of inter-rater reliability scores of computer science educators classifying programming questions using Bloom's Taxonomy, before and after taking our tutorial. Based on the results from these evaluations, we discuss the effectiveness of our tutorial system design for teaching computer science educators how to systematically and consistently classify programming examination questions. We also discuss the suitability of Bloom's Taxonomy and Neo-Piagetian theory for achieving this goal. The Bloom's and Neo-Piagetian tutorials are made available as a community resource. The contributions of this paper are the following: the tutorial system for learning classification schemes for the purpose of coding the difficulty of computing learning materials; its evaluation; new insights into the consistency that computing educators can achieve using Bloom; and first insights into the use of Neo-Piagetian theory by a group of classifiers. © 2013 Copyright Taylor and Francis Group, LLC

Coming to terms with Bloom: An online tutorial for teachers of programming fundamentals

© 2012, Australian Computer Society, Inc. This paper describes a web-based interactive tutorial that enables computer science tutors and lecturers to practice applying the Bloom Taxonomy in classifying programming exam questions. The structure, design and content of the tutorial are described in detail. The results of an evaluation with ten participants highlight important problem areas in the application of Bloom to programming assessments. The key contributions are the content and design of this tutorial and the insights derived from its evaluation. These are important results in continued work on methods of measuring learning progression in programming fundamentals

Over-confidence and confusion in using bloom for programming fundamentals assessment

A computer science student is required to progress from a novice programmer to a proficient developer through the programming fundamentals sequence of subjects. This paper deals with the capturing and representation of learning progression. The key contribution is a web-based interactive tutorial that enables computer science educators to practice applying the Bloom Taxonomy in classifying programming exam questions. The tutorial captures participant confidence and self-explanations for each Bloom [5] classification exercise. The results of an evaluation with ten participants were analyzed for consistency and accuracy in the application of Bloom. The confidence and self-explanation measures were used to identify problem areas in the application of Bloom to programming fundamentals. The tutorial and findings are valuable contributions to future ACM/IEEE CS curriculum revisions, which are expected to have a continued emphasis on Bloom [1]. © 2012 ACM