A strategy for evaluating search of “Real” personal information archives

Personal information archives (PIAs) can include materials from many sources, e.g. desktop and laptop computers, mobile phones, etc. Evaluation of personal search over these collections is problematic for reasons relating to the personal and private nature of the data and associated information needs and measuring system response effectiveness. Conventional information retrieval (IR) evaluation involving use of Cranfield type test collections to establish retrieval effectiveness and laboratory testing of interactive search behaviour have to be re-thought in this situation. One key issue is that personal data and information needs are very different to search of more public third party datasets used in most existing evaluations. Related to this, understanding the issues of how users interact with a search system for their personal data is important in developing search in this area on a well grounded basis. In this proposal we suggest an alternative IR evaluation strategy which preserves privacy of user data and enables evaluation of both the accuracy of search and exploration of interactive search behaviour. The general strategy is that instead of a common search dataset being distributed to participants, we suggest distributing standard expandable personal data collection, indexing and search tools to non-intrusively collect data from participants conducting search tasks over their own data collections on their own machines, and then performing local evaluation of individual results before central agregation

Enhanced visualisation techniques to support access to personal information across multiple devices

The increasing number of devices owned by a single user makes it increasingly difficult to access, organise and visualise personal information (PI), i.e. documents and media, across these devices. The primary method that is currently used to organise and visualise PI is the hierarchical folder structure, which is a familiar and widely used means to manage PI. However, this hierarchy does not effectively support personal information management (PIM) across multiple devices. Current solutions, such as the Personal Information Dashboard and Stuff I’ve Seen, do not support PIM across multiple devices. Alternative PIM tools, such as Dropbox and TeamViewer, attempt to provide a means of accessing PI across multiple devices, but these solutions also suffer from several limitations. The aim of this research was to investigate to what extent enhanced information visualisation (IV) techniques could be used to support accessing PI across multiple devices. An interview study was conducted to identify how PI is currently managed across multiple devices. This interview study further motivated the need for a tool to support visualising PI across multiple devices and identified requirements for such an IV tool. Several suitable IV techniques were selected and enhanced to support PIM across multiple devices. These techniques comprised an Overview using a nested circles layout, a Tag Cloud and a Partition Layout, which used a novel set-based technique. A prototype, called MyPSI, was designed and implemented incorporating these enhanced IV techniques. The requirements and design of the MyPSI prototype were validated using a conceptual walkthrough. The design of the MyPSI prototype was initially implemented for a desktop or laptop device with mouse-based interaction. A sample personal space of information (PSI) was used to evaluate the prototype in a controlled user study. The user study was used to identify any usability problems with the MyPSI prototype. The results were highly positive and the participants agreed that such a tool could be useful in future. No major problems were identified with the prototype. The MyPSI prototype was then implemented on a mobile device, specifically an Android tablet device, using a similar design, but supporting touch-based interaction. Users were allowed to upload their own PSI using Dropbox, which was visualised by the MyPSI prototype. A field study was conducted following the Multi-dimensional In-depth Long-term Case Studies approach specifically designed for IV evaluation. The field study was conducted over a two-week period, evaluating both the desktop and mobile versions of the MyPSI prototype. Both versions received positive results, but the desktop version was slightly preferred over the mobile version, mainly due to familiarity and problems experienced with the mobile implementation. Design recommendations were derived to inform future designs of IV tools to support accessing PI across multiple devices. This research has shown that IV techniques can be enhanced to effectively support accessing PI across multiple devices. Future work will involve customising the MyPSI prototype for mobile phones and supporting additional platforms

Enhanced visualisation techniques to support access to personal information across multiple devices

The increasing number of devices owned by a single user makes it increasingly difficult to access, organise and visualise personal information (PI), i.e. documents and media, across these devices. The primary method that is currently used to organise and visualise PI is the hierarchical folder structure, which is a familiar and widely used means to manage PI. However, this hierarchy does not effectively support personal information management (PIM) across multiple devices. Current solutions, such as the Personal Information Dashboard and Stuff I’ve Seen, do not support PIM across multiple devices. Alternative PIM tools, such as Dropbox and TeamViewer, attempt to provide a means of accessing PI across multiple devices, but these solutions also suffer from several limitations. The aim of this research was to investigate to what extent enhanced information visualisation (IV) techniques could be used to support accessing PI across multiple devices. An interview study was conducted to identify how PI is currently managed across multiple devices. This interview study further motivated the need for a tool to support visualising PI across multiple devices and identified requirements for such an IV tool. Several suitable IV techniques were selected and enhanced to support PIM across multiple devices. These techniques comprised an Overview using a nested circles layout, a Tag Cloud and a Partition Layout, which used a novel set-based technique. A prototype, called MyPSI, was designed and implemented incorporating these enhanced IV techniques. The requirements and design of the MyPSI prototype were validated using a conceptual walkthrough. The design of the MyPSI prototype was initially implemented for a desktop or laptop device with mouse-based interaction. A sample personal space of information (PSI) was used to evaluate the prototype in a controlled user study. The user study was used to identify any usability problems with the MyPSI prototype. The results were highly positive and the participants agreed that such a tool could be useful in future. No major problems were identified with the prototype. The MyPSI prototype was then implemented on a mobile device, specifically an Android tablet device, using a similar design, but supporting touch-based interaction. Users were allowed to upload their own PSI using Dropbox, which was visualised by the MyPSI prototype. A field study was conducted following the Multi-dimensional In-depth Long-term Case Studies approach specifically designed for IV evaluation. The field study was conducted over a two-week period, evaluating both the desktop and mobile versions of the MyPSI prototype. Both versions received positive results, but the desktop version was slightly preferred over the mobile version, mainly due to familiarity and problems experienced with the mobile implementation. Design recommendations were derived to inform future designs of IV tools to support accessing PI across multiple devices. This research has shown that IV techniques can be enhanced to effectively support accessing PI across multiple devices. Future work will involve customising the MyPSI prototype for mobile phones and supporting additional platforms

Identification of re-finding tasks and search difficulty

We address the problem of identifying if users are attempting to re-find information and estimating the level of difficulty of the re-finding task. Identifying re-finding tasks and detecting search difficulties will enable search engines to respond dynamically to the search task being undertaken. To this aim, we conduct user studies and query log analysis to make a better understanding of re-finding tasks and search difficulties. Computing features particularly gathered in our user studies, we generate training sets from query log data, which is used for constructing automatic identification (prediction) models. Using machine learning techniques, our built re-finding identification model, which is the first model at the task level, could significantly outperform the existing query-based identifications. While past research assumes that previous search history of the user is available to the prediction model, we examine if re-finding detection is possible without access to this information. Our evaluation indicates that such detection is possible, but more challenging. We further describe the first predictive model in detecting re-finding difficulty, showing it to be significantly better than existing approaches for detecting general search difficulty. We also analyze important features for both identifications of re-finding and difficulties. Next, we investigate detailed identification of re-finding tasks and difficulties in terms of the type of the vertical document to be re-found. The accuracy of constructed predictive models indicates that re-finding tasks are indeed distinguishable across verticals and in comparison to general search tasks. This illustrates the requirement of adapting existing general search techniques for the re-finding context in terms of presenting vertical-specific results. Despite the overall reduction of accuracy in predictions independent of the original search of the user, it appears that identifying “image re-finding” is less dependent on such past information. Investigating the real-time prediction effectiveness of the models show that predicting ``image'' document re-finding obtains the highest accuracy early in the search. Early predictions would benefit search engines with adaptation of search results during re-finding activities. Furthermore, we study the difficulties in re-finding across verticals given some of the established indications of difficulties in the general web search context. In terms of user effort, re-finding “image” vertical appears to take more effort in terms of number of queries and clicks than other investigated verticals, while re-finding “reference” documents seems to be more time consuming when there is a longer time gap between the re-finding and corresponding original search. Exploring other features suggests that there could be particular difficulty indications for the re-finding context and specific to each vertical. To sum up, this research investigates the issue of effectively supporting users with re-finding search tasks. To this end, we have identified features that allow for more accurate distinction between re-finding and general tasks. This will enable search engines to better adapt search results for the re-finding context and improve the search experience of the users. Moreover, features indicative of similar/different and easy/difficult re-finding tasks can be employed for building balanced test environments, which could address one of the main gaps in the re-finding context