An Evaluation of the Reliability, Validity and Sensitivity of Three Human Mental Workload Measures Under Different Instructional Conditions in Third-Level Education

Although Cognitive Load Theory (CLT) has been researched for many years, it has been criticised for its theoretical clarity and its methodological approach. A crucial issue is the measurement of three types of cognitive load conceived in the theory, and the assessment of overall human cognitive load during learning tasks. This research study is motivated by these issues and it aims to investigate the reliability, validity and sensitivity of three existing self-reporting mental workload instruments, mainly used in Ergonomics, when applied to Education and in particular to the field of Teaching and Learning. A primary research study has been designed and performed in a typical third-level classroom in Computer Science, and the self-reporting mental workload instruments employed are the NASA Task Load Index, the Workload Profile and the Rating Scale Mental Effort. Three instructional design conditions have been designed and employed for the above purposes. The first design condition followed the traditional explicit instruction paradigm whereby a lecturer delivers instructional material mainly using a one-way approach with almost no interactions with students. The second design condition was inspired by the Cognitive Theory of Multimedia Learning whereby the same content, delivered under the first condition, was converted in a multimedia video by following a set of its design principles. The third design condition was an extension of the second condition whereby an inquiry activity was executed after the delivery of the second condition. The empirical evidence gathered in this study suggests that the three selected mental workload measures are highly reliable. Their moderate face validity is in line with the results obtained so far within Ergonomics emphasising and confirming the difficulty in creating optimally valid measures of mental workload. However, the sensitivity of these measures, as achieved in this study, is low, indicating how the three instructional design conditions, as conceived and implemented, do not impose significantly different mental workload levels on learners

Massive Open Online Courses for Employability, Innovation and Entrepreneurship: a Rapid Assessment of Evidence

This report summarises the evidence base regarding the use of open online learning for supporting employability, innovation and entrepreneurship within the European area. It was written as part of the European MOOC Consortium – Labour Markets project (EMC-LM). MOOC platforms in the European MOOC Consortium (EMC) look for solutions to reach better the labour market. This streamlined review of literature draws on scientific literature, project reports, policy documents, case studies and other resources to describe the potential for MOOCs to stimulate and empower organisations to use open education as part of their programs of continuous education (CE) and continuous professional development (CPD) or continuous vocational training (CVT). This report identifies several interstices where the worlds of higher education, vocational educational, training and open online learning come are converging. Evidence is provided for the contention that, through enhancing opportunities for flexible delivery of education, MOOC can innovate the way that we approach degree programmes, lifelong learning, CE and CPD. The information gathered in this report aims to provide a comprehensive overview of relevant material but does not claim to be exhaustive. This report was written as part of the EMC-LM project as a way to establish a shared understanding of the possibilities for collaboration and innovation. The references provided are nonetheless a great starting point for any investigation of the potential of MOOCs for workplace and lifelong learning. Approaches which emphasize the flexible delivery of learning are especially suited to workplace upskilling. For greatest impact and relevance, collaboration should cross disciplinary and professional boundaries, involving a wide range of stakeholders. Employability can be supported through improved management of work transitions; more flexible training options; new routes between education and work; building credibility in CPD; enhancing soft and transversal skills; and developing new mechanisms for authenticating non-formal learning. Innovation can be encouraged by working with greater transparency and sharing; improved dialogue between stakeholders; adopting a reflective attitude towards technology; and through policies which support and motivate new approaches. Entrepreneurship can be moving beyond the knowledge transfer model of entrepreneurial education; drawing stronger connections between theory and practice; promoting collaboration between researchers and practitioners in education and entrepreneurship; and encouraging entrepreneurial culture

DVCL:A Distributed Virtual Computer Lab for Security and Network Education

Teaching networking and IT security in higher education requires a safeplayground for students, where they can safely carry out hands-on exercises.This safe playground is known as a computer lab. Universities have todesign and to provide such a lab with respect to certain criteria, e.g.technical opportunities, educational requirements and demands of thelearners. Since there is no one-size-fits-all lab, the labs will be designed tofit into a certain context and thus have own strengths and weaknesses.In this thesis, we investigate and work with two established labs, whichwere designed for hands-on experiences in networking and it security courses.These labs are predominantly different but have an essential overlap ineducational requirements.One lab is developed by the Open University. It is dedicated for distancelearning. It is based on virtualization and every student is able to startthis lab on his own computer. Students can work out exercises wheneverand wherever they want. A shortcoming however is that students have towork alone, (distant) group work is not possible due to the isolated labarchitecture. This lab is the technical base for our research.The other lab is developed by the Cologne University. It is a physicallab, dedicated for on-campus courses and thus it is not portable. Butstudents can meet in the lab, work in groups and are able to get supportfrom a course advisor, who is also able to verify exercises. A shortcominghowever is that students must be present (they have to travel to theuniversity) and they are dependent on the opening hours of the laboratoryand the availability of the course advisers.In two research parts, we show how such two different lab approachescan be combined and what can be achieved.The first research part is about design issues.Initially, we enable group work in our lab for distance education, sincegroup work is an essential part in on-campus classes. Also remote studentsshould be able to work together. Since the lab is designed as an isolatedsystem, the challenge is to connect two of them on the network level butwithout creating a potential bridge between the isolated and the outsideworld. We achieved this by adding a communication interface to the labarchitecture. This communication interface consists of a ghost host toextract and inject network packets, and a remote bridge endpoint, totransport these packets between remote ghost hosts across an intermediateconnection, e.g. the internet. The developed prototype is called DistributedVirtual Computer Lab (DVCL) and enables to connect two or more distantlabs while preserving the isolated character.The DVCL is then extended and improved by a central authority (CA).While the point-to-point connection of the communication interface canconnect two remote networks in a handy way, more connections requirecareful planning by the students. We show that a CA simplifies the usageof our DVCL for the students (and also for academic staff) and in additionto it avoids administrative configuration errors while connecting remotelabs, e.g. a circular flow which leads to an unusable lab.The first part is completed by two applicability enhancements. Thefirst enhancement covers and resolves security issues in order to pushour prototypical implementation of the DVCL and the CA closer to aproductive learning environment. The second enhancement introduces aGraphical User Interface to increase the usability of the DVCL.The second research part is about educational aspects.In the first part, we assume that working independent from a physicalon-campus lab as well as group work is essential for our students. Ourevaluation of more than 200 students participating in an on-campus networkingcourse shows, that nearly half of the students actually say, thatthey would like to work independently from the university at least partiallyand they would welcome the introduction of an e-learning system. Inaddition, a predominant majority think of working in groups as well asreceiving guidance and feedback as crucial to their learning success. Thisresult justifies and confirms our research and also reveals an additional requirement.The challenge is to provide feedback and guidance to a student, who isworking on an exercise and a human course advisor is not available. Thisis e.g. when students use the DVCL at home in the evening hours. Weshow, that captured network traffic of a lab can give some indication ofwhat a student has already configured according to a certain exercise. Weuse this insight to develop an Electronic Exercise Assistant. This softwareprogram is able to recognize the progress of an exercise and can provideappropriate feedback and support, based on preloaded rules and conditions.This significantly improves the learning situation for students workingremotely in lab. Besides this automatic support, the exercise assistant canverify intermediate and complete solutions of an exercise.The second part is completed by an educational enhancement. Ourevaluation and also own observations show, that a lab is more than aroom with computer and network facilities. Rather it is a social placewhere students e.g. meet, form learning groups, talk and discuss. Weuse these insights and enhance the DVCL to support social interactions.Based on our on-campus lab as source, we model a set of communicational,organizational as well as educational activities and implement them in ourDVCL. The result shows, that our DVCL prototype is no longer a technicalplatform but a virtual place, where students can meet, communicate,arrange learning groups, exchange experiences and work on exercises.This thesis shows that aspects of our two different lab environments canbe combined. Our resulting Distributed Virtual Computer Lab incorporatesstrengths of each source lab. It is a gain for distance teaching as well as foron-campus classes. Remote students are now able to utilize the lab being avirtual classroom, where they can learn in groups, assisted by an electronicadvisor and without the need for a face-to-face meeting. On-campus classescan offer students a new learning environment, where they can learn in aclassroom character without the need to travel to the university

How to design for persistence and retention in MOOCs?

Design of educational interventions is typically carried out following a design cycle involving phases of investigation, conceptualization, prototyping, implementation, execution and evaluation. This cycle can be applied at different levels of granularity e.g. learning activity, module, course or programme. In this paper we consider an aspect of learner behavior that can be critical to the success of many MOOCs i.e. their persistence to study, and the related theme of learner retention. We reflect on the impact that consideration of these can have on design decisions at different stages in the design cycle with the aim of en-hancing MOOC design in relation to learner persistence and retention, with particular attention to the European context

Método para la evaluación de usabilidad de sitios web transaccionales basado en el proceso de inspección heurística

La usabilidad es considerada uno de los factores más importantes en el desarrollo de productos de software. Este atributo de calidad está referido al grado en que, usuarios específicos de un determinado aplicativo, pueden fácilmente hacer uso del software para lograr su propósito. Dada la importancia de este aspecto en el éxito de las aplicaciones informáticas, múltiples métodos de evaluación han surgido como instrumentos de medición que permiten determinar si la propuesta de diseño de la interfaz de un sistema de software es entendible, fácil de usar, atractiva y agradable al usuario. El método de evaluación heurística es uno de los métodos más utilizados en el área de Interacción Humano-Computador (HCI) para este propósito debido al bajo costo de su ejecución en comparación otras técnicas existentes. Sin embargo, a pesar de su amplio uso extensivo durante los últimos años, no existe un procedimiento formal para llevar a cabo este proceso de evaluación. Jakob Nielsen, el autor de esta técnica de inspección, ofrece únicamente lineamientos generales que, según la investigación realizada, tienden a ser interpretados de diferentes maneras por los especialistas. Por tal motivo, se ha desarrollado el presente proyecto de investigación que tiene como objetivo establecer un proceso sistemático, estructurado, organizado y formal para llevar a cabo evaluaciones heurísticas a productos de software. En base a un análisis exhaustivo realizado a aquellos estudios que reportan en la literatura el uso del método de evaluación heurística como parte del proceso de desarrollo de software, se ha formulado un nuevo método de evaluación basado en cinco fases: (1) planificación, (2) entrenamiento, (3) evaluación, (4) discusión y (5) reporte. Cada una de las fases propuestas que componen el protocolo de inspección contiene un conjunto de actividades bien definidas a ser realizadas por el equipo de evaluación como parte del proceso de inspección. Asimismo, se han establecido ciertos roles que deberán desempeñar los integrantes del equipo de inspectores para asegurar la calidad de los resultados y un apropiado desarrollo de la evaluación heurística. La nueva propuesta ha sido validada en dos escenarios académicos distintos (en Colombia, en una universidad pública, y en Perú, en dos universidades tanto en una pública como en una privada) demostrando en todos casos que es posible identificar más problemas de usabilidad altamente severos y críticos cuando un proceso estructurado de inspección es adoptado por los evaluadores. Otro aspecto favorable que muestran los resultados es que los evaluadores tienden a cometer menos errores de asociación (entre heurística que es incumplida y problemas de usabilidad identificados) y que la propuesta es percibida como fácil de usar y útil. Al validarse la nueva propuesta desarrollada por el autor de este estudio se consolida un nuevo conocimiento que aporta al bagaje cultural de la ciencia.Tesi

O impacto da animação e da avaliação automática na motivação para o ensino da programação

Tese de Doutoramento em InformáticaA aprendizagem da programação é uma tarefa complexa que coloca desafios importantes. Nesta dissertação, propõem-se duas abordagens ao processo de ensino nas disciplinas de Programação que visam aumentar a motivação dos alunos e a sua auto-estima. As propostas apresentadas resultam de um estudo cuidado das dificuldades que os alunos sentem ao contactar com a programação de computadores, quer a nível do raciocínio lógico que é necessário para esquematizar a resolução de um qualquer problema, quer a nível da compreensão da linguagem de programação que terá de ser usada para exprimir essa resolução. Percebendo-se que a principal razão assenta na falta de motivação que resulta em parte da falta de confiança e quebra da autoestima necessárias para ultrapassar os impasses, procedeu-se também ao estudo dessa área da psicologia, conforme aqui se relata. Identificados os problemas inerentes à motivação humana e às caraterísticas do processo de resolução de problemas por computador, procuraram-se técnicas que tenham vindo a ser propostas para ajudar os alunos na aquisição de conhecimentos de Programação. Conforme se verifica ao longo do documento focou-se a atenção em duas estratégias: a capacidade do sistema visual humano para rapidamente apreender conceitos e sobretudo processos; a necessidade de os alunos receberem rápido feedback quando se aventuram a resolver sozinhos um problema. No primeiro caso investigaram-se os sistemas de Animação de Programas e no segundo caso os sistemas de Auto-avaliação de Programas. As duas abordagens propostas baseiam-se precisamente numa combinação destas duas estratégias. Dessas abordagens, uma foi alvo de experimentação em sala de aula para se poder aferir o seu real impacto. A outra serviu de base para a proposta de uma plataforma Web para suporte ao processo de ensino/aprendizagem da Programação; o sistema designado por PEP chegou a ser prototipado, conforme é descrito. Para se poder avaliar a qualidade do PEP e até guiar a implementação de uma versão final foram estudados sistemas de aferição de qualidade de software para o ensino, com especial destaque para a framework QEF. Por fim ainda se estudou a importância de incluir no sistema de apoio ao ensino elementos dos jogos que estimulam a motivação e ainda se propôs um enriquecimento do sistema PEP através da inclusão de técnicas de ludificação ou gamificação.Learning programming is a complex task that raises important challenges. In this dissertation, two approaches for teaching Programming courses are proposed aimed at increasing students' motivation and their self-confidence/self-regulation . The proposals presented came out after a deep study of the difficulties that students feel when they are beginning a computer programming course, either at the level of the logical reasoning that is necessary to sketch the resolution of a problem, or at the level of understanding the programming language that has to be used to code this resolution. Realizing that the main reason is based on the lack of motivation derived from the lack of confidence and self-regulation necessary to overcome the troubles, the area of psychology that studies motivation was also researched, as will be discussed in this dissertation. After identifying the problems inherent to human motivation and the characteristics of the computer problem solving process, techniques that have been proposed to help students on learning programming were surveyed. This study was focussed on two strategies: the power of human’s to quickly and easily grasp concepts from static or dynamic visualizations; the positive impact of feedback returned to the students when they write a program on their own. In the first case, Program Animation systems were investigated; and in the second case, Automatic Program Evaluators were studied. The two approaches that outcame from this Ph.D. work are precisely based on a combination of these two strategies. Experiments in the classroom were drawn and conducted to validate the first approach. The second approach inspired the design of a Web-based platform (PEP) to support the teaching / learning process of Programming. A prototype of PEP, based on those guidelines and requirements, was developed by a team of M.Sc. Students as will be described. In order to evaluate PEP’s quality, and to guide its final implementation, software quality assessment systems were studied, with special emphasis on QEF framework. The importance of including, in the educational support systems, elements traditionally used in the context of games to stimulate the motivation was studied and an enrichment of PEP tool through the inclusion of techniques of gamification was still proposed

Automatic generation of software interfaces for supporting decisionmaking processes. An application of domain engineering & machine learning

[EN] Data analysis is a key process to foster knowledge generation in particular domains or fields of study. With a strong informative foundation derived from the analysis of collected data, decision-makers can make strategic choices with the aim of obtaining valuable benefits in their specific areas of action. However, given the steady growth of data volumes, data analysis needs to rely on powerful tools to enable knowledge extraction. Information dashboards offer a software solution to analyze large volumes of data visually to identify patterns and relations and make decisions according to the presented information. But decision-makers may have different goals and, consequently, different necessities regarding their dashboards. Moreover, the variety of data sources, structures, and domains can hamper the design and implementation of these tools. This Ph.D. Thesis tackles the challenge of improving the development process of information dashboards and data visualizations while enhancing their quality and features in terms of personalization, usability, and flexibility, among others. Several research activities have been carried out to support this thesis. First, a systematic literature mapping and review was performed to analyze different methodologies and solutions related to the automatic generation of tailored information dashboards. The outcomes of the review led to the selection of a modeldriven approach in combination with the software product line paradigm to deal with the automatic generation of information dashboards. In this context, a meta-model was developed following a domain engineering approach. This meta-model represents the skeleton of information dashboards and data visualizations through the abstraction of their components and features and has been the backbone of the subsequent generative pipeline of these tools. The meta-model and generative pipeline have been tested through their integration in different scenarios, both theoretical and practical. Regarding the theoretical dimension of the research, the meta-model has been successfully integrated with other meta-model to support knowledge generation in learning ecosystems, and as a framework to conceptualize and instantiate information dashboards in different domains. In terms of the practical applications, the focus has been put on how to transform the meta-model into an instance adapted to a specific context, and how to finally transform this later model into code, i.e., the final, functional product. These practical scenarios involved the automatic generation of dashboards in the context of a Ph.D. Programme, the application of Artificial Intelligence algorithms in the process, and the development of a graphical instantiation platform that combines the meta-model and the generative pipeline into a visual generation system. Finally, different case studies have been conducted in the employment and employability, health, and education domains. The number of applications of the meta-model in theoretical and practical dimensions and domains is also a result itself. Every outcome associated to this thesis is driven by the dashboard meta-model, which also proves its versatility and flexibility when it comes to conceptualize, generate, and capture knowledge related to dashboards and data visualizations