Learning object metadata surrogates in search result interfaces: user evaluation, design and content

The purpose of this research was to evaluate user interaction with learning object metadata surrogates both in terms of content and presentation. The main objectives of this study were: (1) to review the literature on learning object metadata and user-centred evaluation of metadata surrogates in the context of cognitive information retrieval (including user-centred relevance and usability research); (2) to develop a framework for the evaluation of user interaction with learning object metadata surrogates in search result interfaces; (3) to investigate the usability of metadata surrogates in search result interfaces of learning object repositories (LORs) in terms of various presentation aspects (such as amount of information, structure and highlighting of query terms) as a means for facilitating the user relevance judgment process; (4) to investigate in-depth the type of content that should be included in learning object metadata surrogates in order to facilitate the process of relevance judgment; (5) to provide a set of recommendations—guidelines for the design of learning object metadata surrogates in search result interfaces both in terms of content and presentation. [Continues.

Information retrieval from civil engineering repositories: the importance of context and granularity

Information about the design and construction of buildings can be structured in a particular way. This is especially correct given the increasing complexity of building product models and the emergence of building information models with project documents linked to them. In addition, engineers usually have distinct information needs. Research shows that engineers working with building information models place particular importance on the understanding of retrieved content before using it or applying it and that exploration of context is essential for this understanding. Both these factors (the nature of engineering content and the information needs of engineers) make general information retrieval techniques for computing relevance and visualizing search results less applicable in civil engineering information retrieval systems. This paper argues that granularity is a fundamental concept that needs to be considered when measuring relevance and visualizing search results in information retrieval systems for repositories of building design and construction content. It is hypothesized that the design of systems with careful regard for granularity would improve engineers’ relevance judgment behavior. To test this hypothesis, a prototype system, called CoMem-XML, was developed and evaluated in terms of the time needed for users to find relevant information, the accuracy of their relevance judgment, and their subjective satisfaction with the prototype. A user study was conducted in which test subjects were asked to complete tasks by using various forms of the prototype, to complete a satisfaction questionnaire, and to be interviewed. The findings show that users perform better and are more satisfied when the search result interface of the CoMem-XML system presents only relevant information in context. On the other hand, interfaces that present the retrieved information out of context (i.e., without highlighting its position in the parts hierarchy) are less effective for participants to judge relevance

A study of students’ information searching strategies

Concerns have been expressed with respect to students’ ability to search for information using electronic search engines and databases. This research adopted a structured method comprising a combination of questionnaire surveys, an observational study and a ‘sense making’ interview to assess the information searching skills of a group of 14 students undertaking their final year dissertation studies on undergraduate programmes within the Department of Civil and Building Engineering at Loughborough University. The findings reveal that the participants encountered problems with each type of search engine used (Google, Metalib, the Library OPAC system, and individual databases) and lacked knowledge of how to use advanced search strategies. All the participants formulated queries using simple words or free text and there was no evidence of consideration of structured word searching using systematically selected keywords. The results indicate priority areas for additional tuition in information literacy

Presentation of laboratory test results in patient portals: influence of interface design on risk interpretation and visual search behaviour

Abstract Background Patient portals are considered valuable instruments for self-management of long term conditions, however, there are concerns over how patients might interpret and act on the clinical information they access. We hypothesized that visual cues improve patients’ abilities to correctly interpret laboratory test results presented through patient portals. We also assessed, by applying eye-tracking methods, the relationship between risk interpretation and visual search behaviour. Methods We conducted a controlled study with 20 kidney transplant patients. Participants viewed three different graphical presentations in each of low, medium, and high risk clinical scenarios composed of results for 28 laboratory tests. After viewing each clinical scenario, patients were asked how they would have acted in real life if the results were their own, as a proxy of their risk interpretation. They could choose between: 1) Calling their doctor immediately (high interpreted risk); 2) Trying to arrange an appointment within the next 4 weeks (medium interpreted risk); 3) Waiting for the next appointment in 3 months (low interpreted risk). For each presentation, we assessed accuracy of patients’ risk interpretation, and employed eye tracking to assess and compare visual search behaviour. Results Misinterpretation of risk was common, with 65% of participants underestimating the need for action across all presentations at least once. Participants found it particularly difficult to interpret medium risk clinical scenarios. Participants who consistently understood when action was needed showed a higher visual search efficiency, suggesting a better strategy to cope with information overload that helped them to focus on the laboratory tests most relevant to their condition. Conclusions This study confirms patients’ difficulties in interpreting laboratories test results, with many patients underestimating the need for action, even when abnormal values were highlighted or grouped together. Our findings raise patient safety concerns and may limit the potential of patient portals to actively involve patients in their own healthcare

Learning object metadata surrogates in search result interfaces : User evaluation, design and content

EThOS - Electronic Theses Online ServiceGBUnited Kingdo

An eye-tracking approach to the analysis of relevance judgments on the web : the case of Google search engine

Eye movement data can provide an in-depth view of human reasoning and the decision-making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk-aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research

Design factors affecting relevance judgment behaviour in the context of metadata surrogates

This paper reports on the findings of a user study which investigated the effects of structure, highlighting and quantity of information on university students’ interaction with metadata surrogates of learning object repositories (LORs) during the relevance judgement process. The literature review revealed a lack of studies investigating how metadata surrogates should be designed to meet users’ needs and improve the effectiveness of relevance judgement. In order to address this issue, different versions of a prototype system, called METALOR 2, were developed and evaluated in terms of the time needed for users to find relevant information, the accuracy of their relevance judgements and their satisfaction. To evaluate the prototype, a user study was conducted where participants were asked to complete a set of tasks and fill in satisfaction questionnaires. The findings showed that participants performed better and were more satisfied with those versions of the prototype which highlighted metadata elements that included the query terms, used clusters or categories to organize metadata elements in the surrogates and included metadata elements which were relevant to the query or task at hand. The paper concludes with some recommendations for improving the design of metadata surrogates in search result interfaces