A proposal for the evaluation of adaptive information retrieval systems using simulated interaction

The Centre for Next Generation Localisation (CNGL) is involved in building interactive adaptive systems which combine Information Retrieval (IR), Adaptive Hypermedia (AH) and adaptive web techniques and technologies. The complex functionality of these systems coupled with the variety of potential users means that the experiments necessary to evaluate such systems are difficult to plan, implement and execute. This evaluation requires both component-level scientific evaluation and user-based evaluation. Automated replication of experiments and simulation of user interaction would be hugely beneficial in the evaluation of adaptive information retrieval systems (AIRS). This paper proposes a methodology for the evaluation of AIRS which leverages simulated interaction. The hybrid approach detailed combines: (i) user-centred methods for simulating interaction and personalisation; (ii) evaluation metrics that combine Human Computer Interaction (HCI), AH and IR techniques; and (iii) the use of qualitative and quantitative evaluations. The benefits and limitations of evaluations based on user simulations are also discussed

A Methodology for Simulated Experiments in Interactive Search

Interactive information retrieval has received much attention in recent years, e.g. [7]. Furthermore, increased activity in developing interactive features in search systems used across existing popular Web search engines suggests that interactive systems are being recognised as a promising next step in assisting information search. One of the most challenging problems with interactive systems however remains evaluation. We describe the general specifications of a methodology for conducting controlled and reproducible experiments in the context of interactive search. It was developed in the AutoAdapt project1 focusing on search in intranets, but the methodology is more generic than that and can be applied to interactive Web search as well. The goal of this methodology is to evaluate the ability of different algorithms to produce domain models that provide accurate suggestions for query modifications. The AutoAdapt project investigates the application of automatically constructed adaptive domain models for providing suggestions for query modifications to the users of an intranet search engine. This goes beyond static models such as the one employed to guide users who search the Web site of the University of Essex which is based on a domain model that has been built in advance using the documents’ markup structure

Personalization via collaboration in web retrieval systems: a context based approach

World Wide Web is a source of information, and searches on the Web can be analyzed to detect patterns in Web users' search behaviors and information needs to effectively handle the users' subsequent needs. The rationale is that the information need of a user at a particular time point occurs in a particular context, and queries are derived from that need. In this paper, we discuss an extension of our personalization approach that was originally developed for a traditional bibliographic retrieval system but has been adapted and extended with a collaborative model for the Web retrieval environment. We start with a brief introduction of our personalization approach in a traditional information retrieval system. Then, based on the differences in the nature of documents, users and search tasks between traditional and Web retrieval environments, we describe our extensions of integrating collaboration in personalization in the Web retrieval environment. The architecture for the extension integrates machine learning techniques for the purpose of better modeling users' search tasks. Finally, a user-oriented evaluation of Web-based adaptive retrieval systems is presented as an important aspect of the overall strategy for personalization

A Voice-Enabled Framework for Recommender and Adaptation Systems in E-Learning

With the proliferation of learning resources on the Web, finding suitable content (using telephone) has become a rigorous task for voice-based online learners to achieve better performance. The problem with Finding Content Suitability (FCS) with voice E-Learning applications is more complex when the sight-impaired learner is involved. Existing voice-enabled applications in the domain of E-Learning lack the attributes of adaptive and reusable learning objects to be able to address the FCS problem. This study provides a Voice-enabled Framework for Recommender and Adaptation (VeFRA) Systems in E-learning and an implementation of a system based on the framework with dual user interfaces – voice and Web. A usability study was carried out in a visually impaired and non-visually impaired school using the International Standard Organization’s (ISO) 9241-11 specification to determine the level of effectiveness, efficiency and user satisfaction. The result of the usability evaluation reveals that the prototype application developed for the school has “Good Usability” rating of 4.13 out of 5 scale. This shows that the application will not only complement existing mobile and Web-based learning systems, but will be of immense benefit to users, based on the system’s capacity for taking autonomous decisions that are capable of adapting to the needs of both visually impaired and non-visually impaired learners

Generic adaptation framework for unifying adaptive web-based systems

The Generic Adaptation Framework (GAF) research project first and foremost creates a common formal framework for describing current and future adaptive hypermedia (AHS) and adaptive webbased systems in general. It provides a commonly agreed upon taxonomy and a reference model that encompasses the most general architectures of the present and future, including conventional AHS, and different types of personalization-enabling systems and applications such as recommender systems (RS) personalized web search, semantic web enabled applications used in personalized information delivery, adaptive e-Learning applications and many more. At the same time GAF is trying to bring together two (seemingly not intersecting) views on the adaptation: a classical pre-authored type, with conventional domain and overlay user models and data-driven adaptation which includes a set of data mining, machine learning and information retrieval tools. To bring these research fields together we conducted a number GAF compliance studies including RS, AHS, and other applications combining adaptation, recommendation and search. We also performed a number of real systems’ case-studies to prove the point and perform a detailed analysis and evaluation of the framework. Secondly it introduces a number of new ideas in the field of AH, such as the Generic Adaptation Process (GAP) which aligns with a layered (data-oriented) architecture and serves as a reference adaptation process. This also helps to understand the compliance features mentioned earlier. Besides that GAF deals with important and novel aspects of adaptation enabling and leveraging technologies such as provenance and versioning. The existence of such a reference basis should stimulate AHS research and enable researchers to demonstrate ideas for new adaptation methods much more quickly than if they had to start from scratch. GAF will thus help bootstrap any adaptive web-based system research, design, analysis and evaluation

An Online Framework for Supporting the Evaluation of Personalised Information Retrieval Systems

Scope - Personalised Information Retrieval (PIR) has been gaining attention because it investigates intelligent ways for enhancing content delivery. Web users can have personalised services and more accurate information. Problem - Several PIR systems have been proposed in the literature; however, they have not been properly tested or evaluated. Proposal – The authors propose a generally applicable web-based interface, which provides PIR developers and evaluators with: i) implicit recommendations on how to evaluate a specific PIR system; ii) a repository containing studies on user-centred and layered evaluation studies; iii) recommendations on how to best combine different evaluation methods, metrics and measurement criteria in order to most effectively evaluate their system; iv) a UCE methodology which details how to apply existing UCE techniques; v) a taxonomy of evaluations of adaptive systems; and vi) interface translation support (49 languages supported)

A Hybrid Recommender Strategy on an Expanded Content Manager in Formal Learning

The main topic of this paper is to find ways to improve learning in a formal Higher Education Area. In this environment, the teacher publishes or suggests contents that support learners in a given course, as supplement of classroom training. Generally, these materials are pre-stored and not changeable. These contents are typically published in learning management systems (the Moodle platform emerges as one of the main choices) or in sites created and maintained on the web by teachers themselves. These scenarios typically include a specific group of students (class) and a given period of time (semester or school year). Contents reutilization often needs replication and its update requires new edition and new submission by teachers. Normally, these systems do not allow learners to add new materials, or to edit existing ones. The paper presents our motivations, and some related concepts and works. We describe the concepts of sequencing and navigation in adaptive learning systems, followed by a short presentation of some of these systems. We then discuss the effects of social interaction on the learners’ choices. Finally, we refer some more related recommender systems and their applicability in supporting learning. One central idea from our proposal is that we believe that students with the same goals and with similar formal study time can benefit from contents' assessments made by learners that already have completed the same courses and have studied the same contents. We present a model for personalized recommendation of learning activities to learners in a formal learning context that considers two systems. In the extended content management system, learners can add new materials, select materials from teachers and from other learners, evaluate and define the time spent studying them. Based on learner profiles and a hybrid recommendation strategy, combining conditional and collaborative filtering, our second system will predict learning activities scores and offers adaptive and suitable sequencing learning contents to learners. We propose that similarities between learners can be based on their evaluation interests and their recent learning history. The recommender support subsystem aims to assist learners at each step suggesting one suitable ordered list of LOs, by decreasing order of relevance. The proposed model has been implemented in the Moodle Learning Management System (LMS), and we present the system’s architecture and design. We will evaluate it in a real higher education formal course and we intend to present experimental results in the near future

Principles in Patterns (PiP) : Institutional Approaches to Curriculum Design Institutional Story

The principal outputs of the PiP Project surround the Course and Class Approval (C-CAP) system. This web-based system built on Microsoft SharePoint addresses and resolves many of the issues identified by the project. Generally well received by both academic and support staff, the system provides personalised views, adaptive forms and contextualised support for all phases of the approval process. Although the system deliberately encapsulates and facilitates existing approval processes thus achieving buy-in, it is already achieving significant improvements over the previous processes, not only in reducing the administrative overheads but also in supporting curriculum design and academic quality. The system is now embedded across three faculties and is now considered by the University of Strathclyde to be a "core institutional service". Alongside the C-CAP system the PiP Project also cultivated a suite of approaches: an incremental systems development methodology; a structured and replicable evaluation approach, and; Strathclyde's Lean Approach to Efficiencies in Education Kit (SLEEK) business process improvement methodology Each is based on recognised formal techniques, providing the basis for a rigorous approach. This is contextualised within and adapted to the HE institutional context thus building the foundation not only for the project but ultimately for institution wide process improvement. This "institutional story" report summarises the principal outcomes of the Project

An Illustrated Methodology for Evaluating ASR Systems

Proceeding of: 9th International Workshop on Adaptive Multimedia Retrieval (AMR 2011) Took place 2011, July, 18-19, in Barcelona, Spain. The event Web site is http://stel.ub.edu/amr2011/Automatic speech recognition technology can be integrated in an information retrieval process to allow searching on multimedia contents. But, in order to assure an adequate retrieval performance is necessary to state the quality of the recognition phase, especially in speaker-independent and domainindependent environments. This paper introduces a methodology to accomplish the evaluation of different speech recognition systems in several scenarios considering also the creation of new corpora of different types (broadcast news, interviews, etc.), especially in other languages apart from English that are not widely addressed in speech community.This work has been partially supported by the Spanish Center for Industry Technological Development (CDTI, Ministry of Industry, Tourism and Trade), through the BUSCAMEDIA Project (CEN-20091026). And also by MA2VICMR: Improving the access, analysis and visibility of the multilingual and multimedia information in web for the Region of Madrid (S2009/TIC-1542).Publicad

Designer - Supporting Teachers Experience in Learning Management Systems

Our paper entitled “Designer - Supporting Teachers Experience in Learning Management Systems” has been accepted for presentation at the International Conference on Web-based Learning in Sinaia, Romania. This conference is one of the top conferences in the area of technology enhanced learning and the acceptance of our paper at this conference provided us with the chance to present our work to leading researchers in our area. In our paper, we introduced Designer, an approach for teachers to help them in designing courses via a semi-automatic design process based on dynamic user modeling and adaptive learning design generation. In a virtual learning environment supporting the competence development process as well as providing students with personalized/adaptive courses ends up being an elusive and time-consuming task for teachers and/or instructional designers. Our proposed tool, Designer, aims at addressing this issue by supporting teachers to create adaptive courses based on students’ competencies and learning styles. A qualitative and quantitative evaluation demonstrated the effectiveness of Designer in supporting teachers to create such adaptive courses. The presentation of our paper was received very well and leaded to many questions after the presentation, further discussions during the conference, many ideas for future research, and a potential collaboration with an international research group.In the lifelong learning context, the efficiency of learning is measured according to the users’ achievement of the target competences. However, in a virtual learning environment supporting the competence development process ends up being an elusive and time-consuming task for teachers or instructional designers. Furthermore, tailoring courses to the individual learner’s needs and preferences has high potential to improve the learning process of learners. However, again, this is a time-consuming and complex task for teachers and instructional designers. In this paper, we introduce Designer, an approach for teachers to help them in designing courses via a semi-automatic design process based on dynamic user modeling and adaptive learning design generation. A qualitative and quantitative evaluation demonstrated the effectiveness of Designer in supporting teachers to create adaptive courses