Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

Adaptive Educational Hypermedia based on Multiple Student Characteristics

The learning process in Adaptive Educational Hypermedia (AEH) environments is complex and may be influenced by aspects of the student, including prior knowledge, learning styles, experience and preferences. Current AEH environments, however, are limited to processing only a small number of student characteristics. This paper discusses the development of an AEH system which includes a student model that can simultaneously take into account multiple student characteristics. The student model will be developed to use stereotypes, overlays and perturbation techniques. Keywords: adaptive educational hypermedia, multiple characteristics, student model

Designing for change: mash-up personal learning environments

Institutions for formal education and most work places are equipped today with at least some kind of tools that bring together people and content artefacts in learning activities to support them in constructing and processing information and knowledge. For almost half a century, science and practice have been discussing models on how to bring personalisation through digital means to these environments. Learning environments and their construction as well as maintenance makes up the most crucial part of the learning process and the desired learning outcomes and theories should take this into account. Instruction itself as the predominant paradigm has to step down. The learning environment is an (if not 'the�) important outcome of a learning process, not just a stage to perform a 'learning play'. For these good reasons, we therefore consider instructional design theories to be flawed. In this article we first clarify key concepts and assumptions for personalised learning environments. Afterwards, we summarise our critique on the contemporary models for personalised adaptive learning. Subsequently, we propose our alternative, i.e. the concept of a mash-up personal learning environment that provides adaptation mechanisms for learning environment construction and maintenance. The web application mash-up solution allows learners to reuse existing (web-based) tools plus services. Our alternative, LISL is a design language model for creating, managing, maintaining, and learning about learning environment design; it is complemented by a proof of concept, the MUPPLE platform. We demonstrate this approach with a prototypical implementation and a – we think – comprehensible example. Finally, we round up the article with a discussion on possible extensions of this new model and open problems

A scrutable adaptive hypertext

Fuelled by the popularity and uptake of the World Wide Web since the 1990s, many researchers and commercial vendors have focussed on Adaptive Hypermedia Systems as an effective mechanism for disseminating personalised information and services. Such systems store information about the user, such as their goals, interests and background, and use this to provide a personalised response to the user. This technology has been applied to a number of contexts such as education systems, e-commerce applications, information search and retrieval systems. As an increasing number of systems collect and store personal information about their users to provide a personalised service, legislation around the world increasingly requires that users have access to view and modify their personal data. The spirit of such legislation is that the user should be able to understand how personal information about them is used. There literature has reported benefits of allowing users to access and understand data collected about them, particularly in the context of supporting learning through reflection. Although researchers have experimented with open user models, typically the personalisation is inscrutable: the user has little or no visibility in to the adaptation process. When the adaptation produces unexpected results, the user may be left confused with no mechanism for understanding why the system did what it did or how to correct it. This thesis is the next step, giving users the ability to see what has been personalised and why. In the context of personalised hypermedia, this thesis describes the first research to go beyond open, or even scrutable user models; it makes the adaptivity and associated processes open to the user and controllable. The novelty of this work is that a user of an adaptive hypertext system might ask How was this page personalised to me? and is able to see just how their user model affected what they saw in the hypertext document. With an understanding of the personalisation process and the ability to control it, the user is able to steer the personalisation to suit their changing needs, and help improve the accuracy of the user model. Developing an interface to support the scrutinisation of an adaptive hypertext is difficult. Users may not scrutinise often as it is a distraction from their main task. But when users need to scrutinise, perhaps to correct a system misconception, they need to easily find and access the scrutinisation tools. Ideally, the tools should not require any training and users should be able to use them effectively without prior experience or if have not used them for a long time, since this is how users are likely to scrutinise in practice. The contributions of thesis are: (1) SASY/ATML, a domain independent, reusable framework for creation and delivery of scrutable adaptive hypertext; (2)a toolkit of graphical tools that allow the user to scrutinise, or inspect and understand what personalisation occurred and control it; (3) evaluation of the scrutinisation tools and (4) a set of guidelines for providing support for the scrutinisation of an adaptive hypertext through the exploration of several forms of scrutinisation tools

Assessing the Effectiveness and Usability of Personalized Internet Search through a Longitudinal Evaluation

This paper discusses a longitudinal user evaluation of Prospector, a personalized Internet meta-search engine capable of personalized re-ranking of search results. Twenty-one participants used Prospector as their primary search engine for 12 days, agreed to have their interaction with the system logged, and completed three questionnaires. The data logs show that the personalization provided by Prospector is successful: participants preferred re-ranked results that appeared higher up. However, the questionnaire results indicated that people would prefer to use Google instead (their search engine of choice). Users would, nevertheless, consider employing a personalized search engine to perform searches with terms that require disambiguation and/or contextualization. We conclude the paper with a discussion on the merit of combining system- and user-centered evaluation for the case of personalized systems

Light-weight ontologies for scrutable user modelling

This thesis is concerned with the ways light-weight ontologies can support scrutability for large user models and the user modelling process. It explores the role that light-weight ontologies can play, and how they can be exploited, for the purpose of creating and maintaining large, scrutable user models consisting of hundreds of components. We address problems in four key areas: ontology creation, metadata annotation, creation and maintenance of large user models, and user model visualisation, with a goal to provide a simple and adaptable approach that maintains scrutability. Each of these key areas presents a number of challenges that we address. Our solution is the development of a toolkit, LOSUM, which consists of a number of tools to support the user modelling process. It incorporates light-weight ontologies to fulfill a number of roles: aiding in metadata creation, providing structure for large user model visualisation, and as a means to reason across granularities in the user model. In conjunction with this, LOSUM also features a novel visualisation tool, SIV, which performs a dual role of ontology and user model visualisation, supporting the process of ontology creation, metadata annotation, and user model visualisation. We evaluated our approach at each stage with small user studies, and conducted a large scale integrative evaluation of these approaches together in an authentic learning context with 114 students, of whom 77 had exposure to their learner models through SIV. The results showed that students could use the interface and understand the process of user model construction. The flexibility and adaptability of the toolkit has also been demonstrated in its deployment in several other application areas