Evaluating Digital Libraries: A Longitudinal and Multifaceted View

published or submitted for publicatio

Resources for Evaluation of Summarization Techniques

We report on two corpora to be used in the evaluation of component systems for the tasks of (1) linear segmentation of text and (2) summary-directed sentence extraction. We present characteristics of the corpora, methods used in the collection of user judgments, and an overview of the application of the corpora to evaluating the component system. Finally, we discuss the problems and issues with construction of the test set which apply broadly to the construction of evaluation resources for language technologies.Comment: LaTeX source, 5 pages, US Letter, uses lrec98.st

The Status of Health Information Delivery in the United States: The Role of Libraries in the Complex Health Care Environment

published or submitted for publicatio

Making informed choices in social care: the importance of accessible information

The current policy trend is to encourage greater choice in the use of welfare services. To make informed choices, people need information. The process of finding and using information has costs for individuals in terms of effort, time and material resources. These costs are different for different people and impact on their use of information in different ways. Thus, the accessibility of information is important in ensuring those people who need to make choices can do so in an informed way. This paper discusses the importance of information in making informed choices about social support by drawing on the findings of a scoping review of government research and development activity on the accessibility of information about adult social care services. The scoping review was carried out in spring 2006. Details of recent, current and planned projects were obtained through discussions with staff in government departments, government agencies and other related organisations identified using a snowballing technique. Forty-two contacts were made. Eleven research and 36 development projects were identified that aimed to investigate or improve the accessibility of information about social care services. A limited literature search was undertaken on information needs in areas not already under investigation by government. Eighteen articles were identified. Information and helpline staff from six voluntary organisations gave their views on the accessibility of information about social care services. Our findings show that there is no government-related or other recent research evidence on the specific information access needs for some user groups and services, for example, people from ethnic minority groups. For other user groups, such as people with chaotic lifestyles, there is evidence on information needs but no current or planned development projects to address these needs. The implications for the costs of finding and processing information to aid informed choices are discussed

An investigation into the application of Claims Analysis to evaluate usability of a digital library interface

There is a need for tools that help developers evaluate the usability of digital library interfaces. The potential for using Claims Analysis to help developers in this way has been investigated in three linked case studies. The first explored the design rationale of an existing design with its developers. This showed that they had considered positive consequences for novice uses but that they found it difficult to identify negative effects. The second study explored the detailed design of an add-on feature. A scenario and sample claims were introduced to evaluate exploratory use within an action cycle of planning, execution and evaluation. This framework provided an effective stimulus to enable the developers to evaluate the design and explore opportunities for redesign. Finally, some novice users explored the digital library and the findings from this were used to validate a user scenario and claims

Evaluating implicit feedback models using searcher simulations

In this article we describe an evaluation of relevance feedback (RF) algorithms using searcher simulations. Since these algorithms select additional terms for query modification based on inferences made from searcher interaction, not on relevance information searchers explicitly provide (as in traditional RF), we refer to them as implicit feedback models. We introduce six different models that base their decisions on the interactions of searchers and use different approaches to rank query modification terms. The aim of this article is to determine which of these models should be used to assist searchers in the systems we develop. To evaluate these models we used searcher simulations that afforded us more control over the experimental conditions than experiments with human subjects and allowed complex interaction to be modeled without the need for costly human experimentation. The simulation-based evaluation methodology measures how well the models learn the distribution of terms across relevant documents (i.e., learn what information is relevant) and how well they improve search effectiveness (i.e., create effective search queries). Our findings show that an implicit feedback model based on Jeffrey's rule of conditioning outperformed other models under investigation

A Survey of Symbolic Execution Techniques

Many security and software testing applications require checking whether certain properties of a program hold for any possible usage scenario. For instance, a tool for identifying software vulnerabilities may need to rule out the existence of any backdoor to bypass a program's authentication. One approach would be to test the program using different, possibly random inputs. As the backdoor may only be hit for very specific program workloads, automated exploration of the space of possible inputs is of the essence. Symbolic execution provides an elegant solution to the problem, by systematically exploring many possible execution paths at the same time without necessarily requiring concrete inputs. Rather than taking on fully specified input values, the technique abstractly represents them as symbols, resorting to constraint solvers to construct actual instances that would cause property violations. Symbolic execution has been incubated in dozens of tools developed over the last four decades, leading to major practical breakthroughs in a number of prominent software reliability applications. The goal of this survey is to provide an overview of the main ideas, challenges, and solutions developed in the area, distilling them for a broad audience. The present survey has been accepted for publication at ACM Computing Surveys. If you are considering citing this survey, we would appreciate if you could use the following BibTeX entry: http://goo.gl/Hf5FvcComment: This is the authors pre-print copy. If you are considering citing this survey, we would appreciate if you could use the following BibTeX entry: http://goo.gl/Hf5Fv