An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applications

Both humans and machines learn the meaning of unknown words through contextual information in a sentence, but not all contexts are equally helpful for learning. We introduce an effective method for capturing the level of contextual informativeness with respect to a given target word. Our study makes three main contributions. First, we develop models for estimating contextual informativeness, focusing on the instructional aspect of sentences. Our attention-based approach using pre-trained embeddings demonstrates state-of-the-art performance on our single-context dataset and an existing multi-sentence context dataset. Second, we show how our model identifies key contextual elements in a sentence that are likely to contribute most to a reader's understanding of the target word. Third, we examine how our contextual informativeness model, originally developed for vocabulary learning applications for students, can be used for developing better training curricula for word embedding models in batch learning and few-shot machine learning settings. We believe our results open new possibilities for applications that support language learning for both human and machine learner

An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric

Many evaluation metrics have been defined to evaluate the effectiveness ad-hoc retrieval and search result diversification systems. However, it is often unclear which evaluation metric should be used to analyze the performance of retrieval systems given a specific task. Axiomatic analysis is an informative mechanism to understand the fundamentals of metrics and their suitability for particular scenarios. In this paper, we define a constraint-based axiomatic framework to study the suitability of existing metrics in search result diversification scenarios. The analysis informed the definition of Rank-Biased Utility (RBU) -- an adaptation of the well-known Rank-Biased Precision metric -- that takes into account redundancy and the user effort associated to the inspection of documents in the ranking. Our experiments over standard diversity evaluation campaigns show that the proposed metric captures quality criteria reflected by different metrics, being suitable in the absence of knowledge about particular features of the scenario under study.Comment: Original version: 10 pages. Preprint of full paper to appear at SIGIR'18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, July 8-12, 2018, Ann Arbor, MI, USA. ACM, New York, NY, US

Assessing learning outcomes in web searching: A comparison of tasks and query strategies

Users make frequent use of Web search for learning-related tasks, but little is known about how different Web search interaction strategies affect outcomes for learning-oriented tasks, or what implicit or explicit indicators could reliably be used to assess search-related learning on the Web. We describe a lab-based user study in which we investigated potential indicators of learning in web searching, effective query strategies for learning, and the relationship between search behavior and learning outcomes. Using questionnaires, analysis of written responses to knowledge prompts, and search log data, we found that searchers’ perceived learning outcomes closely matched their actual learning outcomes; that the amount searchers wrote in post-search questionnaire responses was highly correlated with their cognitive learning scores; and that the time searchers spent per document while searching was also highly and consistently correlated with higher-level cognitive learning scores. We also found that of the three query interaction conditions we applied, an intrinsically diverse presentation of results was associated with the highest percentage of users achieving combined factual and conceptual knowledge gains. Our study provides deeper insight into which aspects of search interaction are most effective for supporting superior learning outcomes, and the difficult problem of how learning may be assessed effectively during Web search.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/145733/1/Collins-Thompson Rieh CHIIR 2016.pd

Searching as learning: Novel measures for information interaction research

There is growing recognition of the importance of learning as a search outcome and of the need to provide support for it. Yet, before we can consider learning as a part of search, we need to know how to assess it. This panel will focus on methods and measures for assessing learning in the context of search tasks and their outcomes. The panel will be interactive as the audience will be encouraged to engage in contributing their own experiences and ideas related to measures and methods to study learning as a part of search processes. Ideas and experiences with explicit and implicit indicators of learning and with evaluating learning outcomes will be shared during a dialogue between the audience and panelists. Outcomes from the panel discussions will contribute to formulating a research agenda for “search as learning.” The outcomes will be shared with the audience (and the wider ASIST community).Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/111136/1/meet14505101021.pd

Towards searching as a learning process: A review of current perspectives and future directions

We critically review literature on the association between searching and learning and contribute to the formulation of a research agenda for searching as learning. The paper begins by reviewing current literature that tends to characterize search systems as tools for learning. We then present a perspective on searching as learning that focuses on the learning that occurs during the search pro-cess, as well as search outputs and learning outcomes. The concept of ‘comprehensive search’ is proposed to describe iterative, reflec-tive and integrative search sessions that facilitate critical and creative learning beyond receptive learning. We also discuss how search interaction data can provide a rich source of implicit and explicit features through which to assess search-related learning. In conclu-sion, we summarize opportunities and challenges for future research with respect to four agendas: developing a search system that supports sense-making and enhances learning; supporting effective user interaction for searching as learning; providing an inquiry-based literacy tool within a search system; and assessing learning from online searching behaviour.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/145734/1/Rieh et al Towards searching as a learning process JIS2016.pd

Improved string matching under noisy channel conditions

Many document-based applications, including popular Web browsers, email viewers, and word processors, have a ‘Find on this Page ’ feature that allows a user to find every occurrence of a given string in the document. If the document text being searched is derived from a noisy process such as optical character recognition (OCR), the effectiveness of typical string matching can be greatly reduced. This paper describes an enhanced string-matching algorithm for degraded text that improves recall, while keeping precision at acceptable levels. The algorithm is more general than most approximate matching algorithms and allows string-to-string edits with arbitrary costs. We develop a method for evaluating our technique and use it to examine the relative effectiveness of each sub-component of the algorithm. Of the components we varied, we find that using confidence information from the recognition process lead to the largest improvements in matching accuracy