Web-based system for adaptable rubrics: case study on CAD assessment

[EN] This paper describes the implementation and testing of our concept of adaptable rubrics, defined as analytical rubrics that arrange assessment criteria at multiple levels that can be expanded on demand. Because of its adaptable nature, these rubrics cannot be implemented in paper formats, neither are they supported by current Learning Management Systems (LMS). The main contribution of this work involves the adaptable capability of different levels of detail, which can be expanded for each rubric criterion as needed. Our rubrics platform provides specialized and intuitive tools to create and modify rubrics as well as managing metadata to support learning analytics. As an example of a practical assessment situation, a case study on Mechanical Computer Aided Design (MCAD) systems training is presented. The validation process in this scenario proved the effectiveness of our adaptable rubric platform for supporting formative assessment in a multifaceted and complex field such as MCAD. The system also showed the potential of collecting user interaction metadata, which can be used to analyze the evaluation process and guide further improvements in the teaching strategy.This work was supported by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund, through the ANNOTA project (Ref. TIN2013-46036-C3-1-R).Company, P.; Contero, M.; Otey, J.; Camba, J.; Agost, M.; Pérez Lopez, DC. (2017). Web-based system for adaptable rubrics: case study on CAD assessment. Journal of Educational Technology and Society. 20(3):24-41. http://hdl.handle.net/10251/136958S244120

Empowering Learners for Lifelong Competence Development

Sligte, H. W., & Koper, R. (2008). Empowering Learners for Lifelong Competence Development: pedagogical, organisational and technological issues. Proceedings of the 4th TENCompetence Open Workshop. April, 10-11, 2008, Madrid, Spain: SCO-Kohnstamm Instituut, Amsterdam, The Netherlands.These proceedings consist of the peer reviewed papers presented at the Fourth TENCompetence Open Wokrshop. This workshop was organised by the EU 6th Framework integrated project TENCompetence, and took place in the Hotel Santo Domingo, Madrid, Spain, on the 10th and 11th of April 10 2008. The objective of the workshop was to identify and analyse current research and technologies in the fields that provide design guidelines and evidence for powerful interfaces, interaction and navigation support, and tailormade competence development opportunities for individual learners, teams and organisations. These actors and organisations (will) use open source infrastructures that contain all the services to (further) develop their competences, using all the distributed knowledge resources (including actors), learning activities, units of learning and learning routes/ programmes that are available online. The main theme of this workshop is to provide an overview on current research on support and empowerment of learners in relation to their competence development. The papers are grouped in the following thematic sections: 1) Pilots & Practices; 2) The Integrated Architecture; 3) Group interaction and group learning; 4) Assessment.The work on this publication has been sponsored by the TENCompetence Integrated Project that is funded by the European Commission's 6th Framework Programme, priority IST/Technology Enhanced Learning. Contract 027087 [http://www.tencompetence.org

Enjoying Science in a Singapore Neighbourhood Secondary School: An Oxymoron?

A sample of 238 Secondary 1 students at school-entry level and 274 Secondary 4 and 5 students at school-leaving level in Singapore responded to four questionnaires: Getting to Know You to assess attitudes to science and general views of school; WIHIC to assess perceptions of science classroom environment; and QTI to assess student-teacher relationships. Enjoying science in a Singapore neighbourhood Secondary school was not found to be an oxymoron

A Contribution to Conveying Quality Criteria in Mechanical CAD Models and Assemblies through Rubrics and Comprehensive Design Intent Quantification

Esta investigación examinó el uso de rúbricas de ensamblaje, describiendo su evolución a partir de rúbricas de piezas, y estudió cómo afectan a la autoevaluación de los estudiantes. También se valoró la evaluación de los estudiantes por los instructores, encontrando que, mientras que las rúbricas de ensamblaje fueron parcialmente comprendidas y utilizadas de manera eficiente por los estudiantes, éstas fueron usadas con más éxito por los instructores. En esta investigación se han abordado estrategias diseñadas para mejorar la comunicación de la intención de diseño en modelos CAD, acrecentando así su calidad, con directrices dirigidas a evaluar su eficiencia. Es evidente que se necesitan métricas dirigidas hacia la instrucción de la intención de diseño, ya que la intención de diseño transferida a través de modelos CAD puede realizarse en tres etapas con criterios contradictorios que deben ser equilibrados para llegar a la mejor estrategia de modelado. La investigación ha incluido el desarrollo de un método de validación que demuestra que las rúbricas son dispositivos útiles para garantizar una comunicación consistente de la intención de diseño, y son fundamentales no sólo para evaluar, sino también para comunicar las expectativas del instructor. En esta investigación se examinó cómo definir claramente las cualidades de la intención de diseño para permitir una más fácil evaluación de un ensamblaje CAD. Para todas las dimensiones de la rúbrica, se encontró más concordancia y correlación entre instructores que entre instructores y estudiantes. Existe una correlación moderada/fuerte entre los instructores para las dimensiones de la validez, completitud, concisión y claridad, mientras que existe una ligera correlación para las dimensiones de consistencia e intención del diseño. En segundo lugar, las rúbricas también pueden ser descritas como estáticas o dinámicas. Las rúbricas estáticas, existen sólo en papel, no proporcionan retroalimentación inmediata al educando. Las rúbricas dinámicas realizan cálculos que proporcionan observaciones de evaluación inmediatas al usuario. Además, pueden adaptarse a situaciones específicas dependiendo de la capacidad del usuario. Las rúbricas electrónicas son ideales para rúbricas dinámicas, y permiten el uso y desarrollo de rúbricas adaptativas y adaptables, como se describe a continuación. En tercer lugar, las rúbricas deben ser adaptables lo que debería hacerlas fácilmente comprensibles y fáciles de usar, y adaptativas. Las rúbricas de evaluación se usan cuando un experto determina el progreso pedagógico de un educando, mientras que las rúbricas formativas son empleadas por los propios estudiantes, para trazar su progreso e identificar las deficiencias escolares para las que necesitan apoyo. Las rúbricas se deben refinar y mejorar de forma continuada, en un proceso iterativo y colaborativo, hasta que se alcance un acuerdo satisfactorio, tanto entre evaluadores como entre evaluadores y alumnos. Por ello, se desarrollaron mapas de aserciones que ilustran el modo en que la estrategia de expansión-contracción adapta las rúbricas al progreso del aprendiz de CAD, a la vez que ayudan a comprender las diferentes dimensiones de la rúbrica. Basándose en los experimentos con las rúbricas de ensamblajes, es evidente que las pequeñas diferencias entre los instructores sugieren que la rúbrica de ensamblajes propuesta es lo suficientemente sofisticada como para proporcionar una evaluación acumulativa imparcial del desempeño del alumno. En consecuencia, se puede afirmar con confianza que los evaluadores pueden usarse indistintamente sin sacrificar la precisión. Sin embargo, la rúbrica de ensamblaje posee una eficacia finita para producir una autoevaluación formativa de las habilidades de ensamblaje CAD para nuevos alumnos.This research examined the use of assembly rubrics, described how they evolved from parts rubrics, and studied how they affect student self-evaluation. Instructor assessment of students was also evaluated, finding that while the assembly rubrics were partially understood and effectively used by the students, they were more successfully utilized by the instructors. Strategies designed to improve design intent communication in CAD models, in order to enhance their quality, with guidelines targeted to evaluate efficiency, have been addressed with this research. It is apparent that metrics directed toward the instruction of design intent are needed, since design intent transferred through CAD models can be performed at three stages with competing tradeoffs that must be balanced to arrive at the best modeling strategy. Research included the development of a validation approach that reflects that rubrics are valuable devices to expedite consistent design intent communication, and are vital not only for evaluation, but also for the communication of instructor expectations. This research examined how to clearly define qualities of design intent to enable easier CAD assembly assessment. It has been found that there is more inter-rater agreement and correlation between instructors than between instructors and students, for all rubric dimensions. There is strong to moderate correlation between instructors for the dimensions of validity, completeness, conciseness, and clarity, while slight correlation exists for the dimensions of consistency and design intent. Secondly, rubrics can also be described as being either static or dynamic. Static rubrics, existing in paper form only, do not provide immediate feedback to the learner. Dynamic rubrics perform calculations that provide immediate evaluative observations to the user. Besides, they can be independently adapted to specific situations depending on the capability of the user. Electronic rubrics are ideally suited for dynamic rubrics, and permit the use and development of both adaptable and adaptive rubrics, as described next. Thirdly, rubrics need to be adaptable which should make them easily understood and user-friendly, and adaptive (rubric can change itself, depending on the usage pattern). Evaluative rubrics are used when an expert determines the pedagogical progress of a learner, while formative rubrics are employed by the learners themselves, in order to chart their progress and identify scholastic deficiencies that are in need of remediation. Rubrics must be continually refined and improved, in an iterative, collaborative process, until satisfactory agreement is attained, both between raters, but also between raters and learners. Thus, assertions maps were developed, illustrating how the expand-contract strategy adapts the rubrics to CAD trainee progress, while assisting the understanding of the different rubric dimensions. Based on the assembly rubric experiments, it is apparent that the small differences between instructors suggests that the proposed assemblies rubric is sufficiently sophisticated to furnish an unbiased accumulative assessment of student performance. Accordingly, it can be confidently stated that raters can be used interchangeably without sacrificing accuracy. However, the assembly rubric possesses finite efficacy to produce formative self-evaluation of CAD assembly skills for new learners.Aquesta investigació examinà l'ús de rúbriques de acoblament, descrivint la seua evolució a partir de rúbriques de peces, i estudià cóm afecten a la autoavaluació dels estudiants. També es va valorar la avaluació dels estudiants per els instructors, trobant que, mentre que les rúbriques de acoblament van ser parcialment compreses i fetes servir de manera eficient per els estudiants, van ser usades amb mes èxit per els instructors. En aquesta investigació s'han abordat estrategues dissenyades per a millorar la comunicació de la intenció de disseny en models CAD, creixentat així la seua qualitat, amb directrius dirigides a avaluar la seua eficiència. Es evident que es necessiten mètriques dirigides cap a la instrucció de la intenció de disseny, ja que la intenció de disseny transferida a través de models CAD pot realitzar-se en tres etapes amb criteris contradictoris que deuen ser equilibrats per a arribar a la millor estratègia de modelatge. La investigació ha inclòs el desenvolupament de un mètode de validació que demostra que las rúbriques son dispositius útils per a garantir una comunicació consistent de la intenció de disseny, i son fonamentals no només per a avaluar, però també per a comunicar les expectatives de l'instructor. En aquesta investigació s'examinà com definir clarament les qualitats de la intenció de disseny per a permetre una mes fàcil avaluació de un acoblament CAD. Per a totes les dimensions de la rúbrica, es va trobar mes concordança i correlació entre instructors que no pas entre instructors i estudiants. Existeix una correlació moderada/fort entre els instructors per a les dimensions de la validesa, completesa, concisió i claredat, mentre que existeix una lleugera correlació per a les dimensions de consistència i intenció del disseny. En segon lloc, les rúbriques també poden ser descrites com estàtiques o dinàmiques. Les rúbriques estàtiques, existeixen sòls en paper, no proporcionen retroalimentació immediata a l'educand. Les rúbriques dinàmiques realitzen càlculs que proporcionen observacions de avaluació immediates al usuari. A mes, poden adaptar-se a situacions específiques dependent de la capacitat de l'usuari. Les rúbriques electròniques son ideals per a rúbriques dinàmiques, i permeten l'ús i desenvolupament de rúbriques adaptatives i adaptables, como es descriu a continuació. En tercer lloc, les rúbriques deuen ser adaptables, el que deuria ferles fàcilment comprensibles i fàcils d'usar, i adaptatives. Les rúbriques d'avaluació se usen quant un expert determina el progrés pedagògic de un educand, mentre que les rúbriques formatives son fetes servir per els propis estudiants, per a traçar el seu progrés i identificar les deficiències escolars para a les que necessiten suport. Les rúbriques es deuen refinar i millorar de forma continuada, en un procés iteratiu i col·laboratori, fins que se arriba a un acord satisfactori, tant entre avaluadors como entre avaluadors i alumnes. Por això, es desenvoluparen mapes de assercions que il·lustren el mode en que la estratègia de expansió-contracció adapta les rúbriques al progres del aprenent de CAD, a la vegada que ajuden a comprendre les diferents dimensions de la rúbrica. Basant-se en els experiments amb les rúbriques de acoblaments, es evident que les xicotetes diferencies entre els instructors suggereixen que la rúbrica de acoblaments proposta es lo suficientment sofisticada com per a proporcionar una avaluació acumulativa imparcial del acompliment de l'alumne. En conseqüència, es pot afirmar amb confiança que els avaluadors poden usar-se indistintament sense sacrificar la precisió. No obstant, la rúbrica de acoblaments posseeix una eficàcia finita per a produir una autoavaluació formativa de les habilitats de acoblament CAD per a nous alumnes.Otey, JM. (2017). A Contribution to Conveying Quality Criteria in Mechanical CAD Models and Assemblies through Rubrics and Comprehensive Design Intent Quantification [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/94627TESI

D10.3 - Report with an assessment of the WP results including ID10.7-ID10.11

Kew, C., & Griffiths, D. (2009). D10.3 - Report with an assessment of the WP results including ID10.7-ID10.11. TENCompetence.This report summarises Dissemination work carried out by the TENCompetence Project between months 30 and 42. It includes sections on workshops and events, standardisation of project outcomes, awareness raising and the TENCompetence Foundation, business models and interim sustainability plan.The work on this publication has been sponsored by the TENCompetence Integrated Project that is funded by the European Commission's 6th Framework Programme, priority IST/Technology Enhanced Learning. Contract 027087 [http://www.tencompetence.org

Tracing the creation and evaluation of accessible Open Educational Resources through learning analytics

The adoption of Open Educational Resources (OER) has been continuously growing and with it the need to addressing the diversity of students’ learning needs. Because of that, OER should meet with characteristics such as the web accessibility and quality. Thus, teachers as the creators of OER need supporting tools and specialized competences. The main contribution of this thesis is a Learning Analytics Model to Trace the Creation and Evaluation of OER (LAMTCE) considering web accessibility and quality. LAMTCE also includes a user model of the teacher’s competences in the creation and evaluation of OER. Besides that, we developed ATCE, a learning analytics tool based on the LAMTCE model. Finally, it was carried out an evaluation conducted with teachers involving the use of the tool and we found that the tool really benefited teachers in the acquisition of their competences in creation and evaluation of accessible and quality OER.La adopción de Recursos Educativos Abiertos (REA) ha ido en aumento y con ello la necesidad de abordar la diversidad de necesidades de aprendizaje de los estudiantes. Por ello, los REA deben cumplir con características tales como la accesibilidad web y la calidad. Así, los profesores como los creadores de REA necesitan de herramientas de soporte y competencias especializadas. La principal contribución de la tesis es el modelo LAMTCE, un modelo de analíticas de aprendizaje para hacer seguimiento a la creación y evaluación de REA considerando la accesibilidad web y la calidad. LAMTCE también incluye un modelo de usuario de las competencias del profesor en creación y evaluación de REA. Además, se desarrolló ATCE, una herramienta de analíticas de aprendizaje que está basada en el modelo LAMTCE. Finalmente, se llevó a cabo un estudio con profesores involucrando el uso de la herramienta encontrando que ésta realmente benefició a los profesores en la adquisición de sus competencias en creación y evaluación de REA accesibles y de calidad

Technology and Testing

From early answer sheets filled in with number 2 pencils, to tests administered by mainframe computers, to assessments wholly constructed by computers, it is clear that technology is changing the field of educational and psychological measurement. The numerous and rapid advances have immediate impact on test creators, assessment professionals, and those who implement and analyze assessments. This comprehensive new volume brings together leading experts on the issues posed by technological applications in testing, with chapters on game-based assessment, testing with simulations, video assessment, computerized test development, large-scale test delivery, model choice, validity, and error issues. Including an overview of existing literature and ground-breaking research, each chapter considers the technological, practical, and ethical considerations of this rapidly-changing area. Ideal for researchers and professionals in testing and assessment, Technology and Testing provides a critical and in-depth look at one of the most pressing topics in educational testing today

Mobile learning experience and self-directed learning readiness on mobile task-based activity performance: a case study among postgraduate students

Mobile devices have a wide array of capabilities, including accessing unlimited information and self-learning for students. This study aimed to identify postgraduate students' experiences performing task-based activities on mobile devices and preparing for self-directed learning. This mobile task-based activity is an educational practice that combines education and assessment. The researcher used a targeted sample of 34 postgraduates from a public university in Malaysia. This study used a mixed-method case study design with qualitative and quantitative data for triangulation. The researcher gathered quantitative data through online questionnaires and evaluations based on mobile devices. Questionnaire data were analyzed using descriptive statistics and paired t-sample analyses, while mobile-based assessments were analyzed using a rubric. Qualitative data were collected through learning logbooks, reflective writing, and semistructured interviews and analyzed thematically. The survey results of descriptive statistical analysis have shown that students’ prior knowledge and learning experience using mobile technology is more modest. However, post-study findings prove that students who use laptops and smartphones for task-based activities on mobile devices gain more experience. The Wilcoxon Signed-Ranks test showed that students’ post-test readiness for self-directed learning was more significant than the pre-test. Assessments of mobile devices indicate that students perform at an advanced and competent level. Thus, this study shows that mobile task-based activities can improve students’ learning experiences in a significant way and develop self-directed learning skills. These results could assist further studies on mobile devices for learning and assessment purposes. The implications of this research are to cultivate students' experience of meaningful mobile learning and self-directed learning skills

Diagnosing and remediating mathematical common student errors in e-assessment questions (A case study)

Mistakes and misconceptions in mathematics are as common as those in any real life situation. The errors frequently made by a myriad of students, known as Common Students Errors (CSEs), heavily influence their ability to learn and retain mathematics. Identifying and correcting CSEs on handwritten work is straightforward since teachers have access to the students’ intermediate workings. One major drawback of this approach is the time and effort it takes to mark and to get feedback back to the individual student.In recent years, e-Assessments have become a pivotal method of providing mathematical assessments in education. However, there is increasing concern that e-Assessment cannot act like a human-marker to identify and correct CSEs in mathematical e-Assessment questions. Therefore, how to diagnose and remediate CSEs in e-Assessment questions has long been a question of great interest in research in e-Assessments.This thesis explores a method to diagnose and remediate CSEs in Engineering Mathematics (EM) e-Assessment questions. Firstly, this study contributes to existing knowledge of mathematical CSEs by systematically gathering and taxonomically compiling CSEs in EM e-Assessment questions and producing an interactive book. Secondly, different features of e-Assessment questions, to capture mathematical CSEs and provide enhanced feedback to correct those CSEs, are developed. Thirdly, new light is shed on our understanding of parameter selection restrictions of some mathematical e-Assessment questions which have at least one CSE. Fourthly, it offers some important insights into students’ perceptions on the CSE enhanced feedback by analysing a questionnaire. Finally, it provides strong empirical confirmation that the CSE enhanced feedback has successfully corrected some mathematical CSEs in the majority of students who participated in this study.The outcomes of this thesis contribute in several ways to our understanding of mathematical CSEs and addressing them in e-Assessment questions, and provides a basis for further research