Building cost-benefit models of information interactions

Modeling how people interact with search interfaces has been of particular interest and importance to the field of Interactive Information Retrieval. Recently, there has been a move to developing formal models of the interaction between the user and the system, whether it be to: (i) run a simulation, (ii) conduct an economic analysis, (iii) measure system performance, or (iv) simply to better understand user interactions and hypothesise about user behaviours. In such models, they consider the costs and the benefits that arise through the interaction with the interface/system and the information surfaced during the course of interaction. In this half day tutorial, we will focus on describing a series of cost-benefit models that have been proposed in the literature and how they have been applied in various scenarios. The tutorial will be structured into two parts. First, we will provide an overview of Decision Theory and Cost-Benefit Analysis techniques, and how they can and have be applied to a variety of Interactive Information Retrieval scenarios. For example, when do facets helps?, under what conditions are query suggestions useful? and is it better to bookmark or re-find? The second part of the tutorial will be dedicated to building cost-benefit models where we will discuss different techniques to build and develop such models. In the practical session, we will also discuss how costs and benefits can be estimated, and how the models can help inform and guide experimentation. During the tutorial participants will be challenged to build cost models for a number of problems (or even bring their own problems to solve)

Principles in Patterns (PiP) : User Acceptance Testing of Course and Class Approval Online Pilot (C-CAP)

The PiP Evaluation Plan documents four distinct evaluative strands, the first of which entails an evaluation of the PiP system pilot (WP7:37 – Systems & tool evaluation). Phase 1 of this evaluative strand focused on the heuristic evaluation of the PiP Course and Class Approval Online Pilot system (C-CAP) and was completed in December 2011. Phase 2 of the evaluation is broadly concerned with "user acceptance testing". This entails exploring the extent to which C-CAP functionality meets users' expectations within specific curriculum design tasks, as well as eliciting data on C-CAP's overall usability and its ability to support academics in improving the quality of curricula. The general evaluative approach adopted therefore employs a combination of standard Human-Computer Interaction (HCI) approaches and specially designed data collection instruments, including protocol analysis, stimulated recall and pre- and post-test questionnaire instruments. This brief report summarises the methodology deployed, presents the results of the evaluation and discusses their implications for the further development of C-CAP

Affective Experiences of International and Home Students during the Information Search Process

An increasing number of students are studying abroad requiring that they interact with information in languages other than their mother tongue. The UK in particular has seen a large growth in international students within Higher Education. These non-native English speaking students present a distinct user group for university information services, such as university libraries. This article presents the findings from an in-depth study to understand differences between the search processes of home and international students. Data were collected using an online survey and diary-interview to capture thoughts and feelings in a more naturalistic way. International students are found to have similar information search processes to those of home students, but sometimes face additional difficulties in assessing search results such as confusion when dealing with differing cultural perspectives. The potential implications for information service providers, particularly university libraries, are discussed, such as providing assistance to students for identifying appropriate English sources

Behavioral Task Modeling for Entity Recommendation

Our everyday tasks involve interactions with a wide range of information. The information that we manage is often associated with a task context. However, current computer systems do not organize information in this way, do not help the user find information in task context, but require explicit user actions such as searching and information seeking. We explore the use of task context to guide the delivery of information to the user proactively, that is, to have the right information easily available at the right time. In this thesis, we used two types of novel contextual information: 24/7 behavioral recordings and spoken conversations for task modeling. The task context is created by monitoring the user's information behavior from temporal, social, and topical aspects; that can be contextualized by several entities such as applications, documents, people, time, and various keywords determining the task. By tracking the association amongst the entities, we can infer the user's task context, predict future information access, and proactively retrieve relevant information for the task at hand. The approach is validated with a series of field studies, in which altogether 47 participants voluntarily installed a screen monitoring system on their laptops 24/7 to collect available digital activities, and their spoken conversations were recorded. Different aspects of the data were considered to train the models. In the evaluation, we treated information sourced from several applications, spoken conversations, and various aspects of the data as different kinds of influence on the prediction performance. The combined influences of multiple data sources and aspects were also considered in the models. Our findings revealed that task information could be found in a variety of applications and spoken conversations. In addition, we found that task context models that consider behavioral information captured from the computer screen and spoken conversations could yield a promising improvement in recommendation quality compared to the conventional modeling approach that considered only pre-determined interaction logs, such as query logs or Web browsing history. We also showed how a task context model could support the users' work performance, reducing their effort in searching by ranking and suggesting relevant information. Our results and findings have direct implications for information personalization and recommendation systems that leverage contextual information to predict and proactively present personalized information to the user to improve the interaction experience with the computer systems.Jokapäiväisiin tehtäviimme kuuluu vuorovaikutusta monenlaisten tietojen kanssa. Hallitsemamme tiedot liittyvät usein johonkin tehtäväkontekstiin. Nykyiset tietokonejärjestelmät eivät kuitenkaan järjestä tietoja tällä tavalla tai auta käyttäjää löytämään tietoja tehtäväkontekstista, vaan vaativat käyttäjältä eksplisiittisiä toimia, kuten tietojen hakua ja etsimistä. Tutkimme, kuinka tehtäväkontekstia voidaan käyttää ohjaamaan tietojen toimittamista käyttäjälle ennakoivasti, eli siten, että oikeat tiedot olisivat helposti saatavilla oikeaan aikaan. Tässä väitöskirjassa käytimme kahdenlaisia uusia kontekstuaalisia tietoja: 24/7-käyttäytymistallenteita ja tehtävän mallintamiseen liittyviä puhuttuja keskusteluja. Tehtäväkonteksti luodaan seuraamalla käyttäjän tietokäyttäytymistä ajallisista, sosiaalisista ja ajankohtaisista näkökulmista katsoen; sitä voidaan kuvata useilla entiteeteillä, kuten sovelluksilla, asiakirjoilla, henkilöillä, ajalla ja erilaisilla tehtävää määrittävillä avainsanoilla. Tarkastelemalla näiden entiteettien välisiä yhteyksiä voimme päätellä käyttäjän tehtäväkontekstin, ennustaa tulevaa tiedon käyttöä ja hakea ennakoivasti käsillä olevaan tehtävään liittyviä asiaankuuluvia tietoja. Tätä lähestymistapaa arvioitiin kenttätutkimuksilla, joissa yhteensä 47 osallistujaa asensi vapaaehtoisesti kannettaviin tietokoneisiinsa näytönvalvontajärjestelmän, jolla voitiin 24/7 kerätä heidän saatavilla oleva digitaalinen toimintansa, ja joissa tallennettiin myös heidän puhutut keskustelunsa. Mallien kouluttamisessa otettiin huomioon datan eri piirteet. Arvioinnissa käsittelimme useista sovelluksista, puhutuista keskusteluista ja datan eri piirteistä saatuja tietoja erilaisina vaikutuksina ennusteiden toimivuuteen. Malleissa otettiin huomioon myös useiden tietolähteiden ja näkökohtien yhteisvaikutukset. Havaintomme paljastivat, että tehtävätietoja löytyi useista sovelluksista ja puhutuista keskusteluista. Lisäksi havaitsimme, että tehtäväkontekstimallit, joissa otetaan huomioon tietokoneen näytöltä ja puhutuista keskusteluista saadut käyttäytymistiedot, voivat parantaa suositusten laatua verrattuna tavanomaiseen mallinnustapaan, jossa tarkastellaan vain ennalta määritettyjä vuorovaikutuslokeja, kuten kyselylokeja tai verkonselaushistoriaa. Osoitimme myös, miten tehtäväkontekstimalli pystyi tukemaan käyttäjien suoritusta ja vähentämään heidän hakuihin tarvitsemaansa työpanosta järjestämällä hakutuloksia ja ehdottamalla heille asiaankuuluvia tietoja. Tuloksillamme ja havainnoillamme on suoria vaikutuksia tietojen personointi- ja suositusjärjestelmiin, jotka hyödyntävät kontekstuaalista tietoa ennustaakseen ja esittääkseen ennakoivasti personoituja tietoja käyttäjälle ja näin parantaakseen vuorovaikutuskokemusta tietokonejärjestelmien kanssa

Support for Information-Seeking Strategies

Längere Such-Episoden umfassen mehrere Such-Aktionen. Diese Such-Aktionen können in verschiedene Klassen unterteilt werden. Die Klassifikation, die in dieser Arbeit verwendet wird, ist die ISS-Klassifikation von Belkin, Marchetti und Cool, die vier Facetten verwendet (method, goal, mode, resource used), von denen jede zwei Werte hat. Unter der Annahme, dass Support-Mechanismen für jede Klasse bekannt sind, war die Forschungsfrage, ob man jede dieser Klassen durch ein anderes, spezialisiertes Such-Interface unterstützen muss, um eine optimale Unterstützung über unterschiedliche Situationen hinweg zu erreichen, oder ob es reicht, wenn ein einziges Interface Support-Mechanismen für alle denkbaren Such-Aktionen anbietet. Die Forschungsfrage wurde in insgesamt drei Experimenten untersucht. Die ISS-Klassifikation besteht aus 16 Klassen. Da die Untersuchung der Forschungsfrage für jede dieser 16 Klassen zu aufwändig gewesen wäre, wurden zwei Facetten, goal und resource used, ausgeschlossen. Dadurch blieben zwei Facetten, method und mode, mit insgesamt vier Klassen übrig. Support-Mechanismen für die vier Facetten-Werte, scanning, searching, recognition und specification, wurden gesammelt unter der Annahme, dass diese Mechanismen ebenso unabhängig voneinander sind wie die zugrunde liegenden Facetten. Der Facetten-Wert recognition wurde in zwei Experimenten untersucht. Das erste Experiment untersuchte eine Tabellen-basierte Ergebnislisten-Darstellung mit einer Listen-basierten Darstellung mit Highlighting bezüglich ihrer Auswirkung auf den Erfolg bei visueller Suche. Versuchsteilnehmer wurden gebeten, Such-Ziele in vorgefertigten Ergebnislisten mit beiden Darstellungs-Varianten, aber nur unter Verwendung visueller Suche, zu finden (Messwiederholung). Ihr Erfolg wurde gemessen anhand der gefundenen Such-Ziele pro Zeit. Weder Liste noch Tabelle zeigten statistisch signifikante Vorteile gegenüber der jeweils anderen Variante. Das zweite Experiment führte eine Baseline-Variante ein, die aus einer herkömmlichen Listen-basierten Darstellung ohne Highlighting bestand. Von dieser Änderung abgesehen, war das Experiment dem ersten recht ähnlich. Auch in diesem Experiment wurde kein statistisch signifikanter Unterschied zwischen den Darstellungs-Varianten gefunden. Für die anderen Facetten-Werte wurden Support-Mechanismen anhand einer Literatur-Suche identifiziert und im letzten Experiment verwendet. Die Haupt-Forschungsfrage wurde untersucht anhand von drei verschiedenen Such-Systemen, die sich einander ähnelten und auf dem ezDL-System basierten. Die erste Variante (baseline) war eine sehr vereinfachte Variante des ezDL-Systems, das außer einer Übersetzungs-Einrichtung keine Support-Mechanismen enthielt. Das zweite System war ein adaptives System, das Support-Mechanismen passend zur aktuellen Such-Aktion des Teilnehmers anbot. Das dritte System enthielt alle Support-Mechanismen des zweiten Systems für alle ISS-Klassen auf einmal. Versuchsteilnehmer wurden gebeten, Suchaufgaben mit einem der drei Systeme zu bearbeiten (ohne Messwiederholung). Ihr Erfolg wurde gemessen durch die Anzahl der gefundenen Dokumente pro Zeit. Kein statistisch signifikanter Unterschied zwischen den Systemen wurde gefunden.Longer search episodes comprise multiple search actions. These search actions can be grouped into several classes. The classification used in this work is the ISS classification by Belkin, Marchetti and Cool, which uses four facets (method, goal, mode and resource used), each of which has to values. Assuming that support features for each class are known, the research question was whether it is necessary to support each ISS class by a different search user interface in order to optimally help the user across many situations, or if a single interface can offer support mechanisms for any search action the user is being involved in. The research question was examined in three experiments. The ISS classification consists of 16 classes. Since studying the research question for all of these classes would have been too difficult, two facets, resource used and learning, were omitted, leaving the two facets method and mode with a total of four remaining classes for examination. Support mechanisms for each value of the two facets, scanning, searching, recognition, and specification, were gathered, assuming that the support mechanisms are as independent of each other as the underlying facets. Support features for the facet value recognition was examined in two experiments. The first experiment compared a table-based result list presentation with a list-based one using highlighting in terms of their support for visual search. Participants were asked to locate search targets in manufactured result lists using each of the result list variants solely by means of visual search (within-subjects design). Their success was measured by how many search targets they found per time. Neither list nor table provided a statistically significant advantage. The second experiment added a baseline result list without any support for visual search; apart of this, the experiment was very similar to the first one. Again, none of the studied result list variants showed statistically significant differences to any other. For the other facet values, the support mechanisms were gathered in a literature search, which identified some promising mechanisms which were then used in the last experiment. The main research question was examined using three search systems that were similar to each other. The first one (baseline) was a very basic variant of the ezDL system and provided no advanced support features other than a translation feature. The second system was an adaptive interface that provided support features only for the ISS class the user was being engaged in. The third system provided all support features of the second system for all ISS classes at once. Participants were asked to complete search tasks with one of the systems (between-subjects design). Their success was measured by how many of the required documents they could locate per time. None of the systems studied provided any statistically significant benefit over any of the other systems

Event and map content personalisation in a mobile and context-aware environment.

Effective methods for information access are of the greatest importance for our modern lives “ particularly with respect to handheld devices. Personalisation is one such method which models a users characteristics to deliver content more focused to the users needs. The emerging area of sophisticated mobile computing devices has started to inspire new forms of personalised systems that include aspects of the persons contextual environment. This thesis seeks to understand the role of personalisation and context, to evaluate the effectiveness of context for content personalisation and to investigate the event and map content domain for mobile usage. The work presented in this thesis has three parts: The first part is a user experiment on context that investigated the contextual attributes of time, location and interest, with respect to participants perception of their usefulness. Results show highly dynamic and interconnected effects of context on participants usefulness ratings. In the second part, these results were applied to create a predictive model of context that was related to attribution theory and then combined with an information retrieval score to create a weighted personalisation model. In the third part of this work, the personalisation model was applied in a mobile experiment. Participants solved situational search tasks using a (i) non-personalized and a (ii) personalized mobile information system, and rating entertainment events based on usefulness. Results showed that the personalised system delivered about 20% more useful content to the mobile user than the non-personalised system, with some indication for reduced search effort in terms of time and the amount of queries per task. The work presented provides evidence for the promising potential of context to facilitate personalised information delivery to users of mobile devices. Overall, it serves as an example of an investigation into the effectiveness of context from multiple angles and provides a potential link to some of the aspects of psychology as a potential source for a deeper understanding of contextual processes in humans