Theory-based user modeling for personalized interactive information retrieval

In an effort to improve users’ search experiences during their information seeking process, providing a personalized information retrieval system is proposed to be one of the effective approaches. To personalize the search systems requires a good understanding of the users. User modeling has been approved to be a good method for learning and representing users. Therefore many user modeling studies have been carried out and some user models have been developed. The majority of the user modeling studies applies inductive approach, and only small number of studies employs deductive approach. In this paper, an EISE (Extended Information goal, Search strategy and Evaluation threshold) user model is proposed, which uses the deductive approach based on psychology theories and an existing user model. Ten users’ interactive search log obtained from the real search engine is applied to validate the proposed user model. The preliminary validation results show that the EISE model can be applied to identify different types of users. The search preferences of the different user types can be applied to inform interactive search system design and development

A Diary Study-Based Evaluation Framework for Mobile Information Retrieval

International audienceIn this poster, we propose an evaluation framework that investigates the integration of the user context (interests, location and time) into the evaluation process of mobile IR. Our approach is based on a diary study where users are asked to log their queries annotated by their location and time. Users' interests are explicitly acquired or implicitly learned based on users' relevance judgments for the retrieved documents answering their queries. We propose two evaluation protocols namely training/test in chronological order and k-fold cross validation. We exploit this framework in order to evaluate the performance of our context-based personalized mobile search approach. Experimental results show the stability performance of our approach according to the proposed evaluation protocols and demonstrate the viability of the diary approach as a means to capture context in evaluation

Using a Research Log and Reflective Writing to Improve EBP and Information Literacy Skills of BSN Students

Background Baccalaureate nursing educators must prepare nurses to implement evidence-based practice (EBP). BSN nurses must be able to effectively identify, analyze, and synthesize evidence (AACN, 2008). In a nursing research course, students conducted group projects which required searching for the best evidence. Project evaluations revealed that students were not searching systematically. To facilitate EBP and information literacy skill development, a collaboration between the College of Nursing and Library Services emerged. Targeted Learning Outcomes 1. Formulate a strategic search using databases and Internet resources 2. Evaluate and select the ‘best available’ evidence 3. Document systematic search (keywords, subject headings, limiters, and results) 4. Describe why evidence was selected 5. Reflect on search process, difficulties, and potential revisions for next search. Teaching Learning Activities In spring of 2012, research logs were added to an EBP group project requiring students to identify the best evidence. Groups documented their search using a research log worksheet and narrative which included reflection of the search process, evidence appraisal, and strengths and weakness. Although the research log worksheet provided structure, specific problems including uncoordinated group searches, inadequate articulation of evidence selection, and limited reflection about strengths and weaknesses were still identified. Additional sessions reinforcing information literacy skills were integrated in the course. The information literacy skills sessions and research logs were implemented for two subsequent semesters. Evaluation of Approach Student research log and EBP group project scores will be compared over three semesters using ANOVA to determine differences in group performance. During initial data analysis, an independent t test reflected a significant difference between scores for Spring 2012 and Fall 2012 groups. The Fall 2012 groups who used research logs and experienced additional information literacy skills sessions scored significantly higher on their assignments than Spring 2012 groups. Data collection for Spring 2013 is in progress. Final study results as well as implications for nursing education will be articulated

Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion

Large Language Models (LLMs) excel at tackling various natural language tasks. However, due to the significant costs involved in re-training or fine-tuning them, they remain largely static and difficult to personalize. Nevertheless, a variety of applications could benefit from generations that are tailored to users' preferences, goals, and knowledge. Among them is web search, where knowing what a user is trying to accomplish, what they care about, and what they know can lead to improved search experiences. In this work, we propose a novel and general approach that augments an LLM with relevant context from users' interaction histories with a search engine in order to personalize its outputs. Specifically, we construct an entity-centric knowledge store for each user based on their search and browsing activities on the web, which is then leveraged to provide contextually relevant LLM prompt augmentations. This knowledge store is light-weight, since it only produces user-specific aggregate projections of interests and knowledge onto public knowledge graphs, and leverages existing search log infrastructure, thereby mitigating the privacy, compliance, and scalability concerns associated with building deep user profiles for personalization. We then validate our approach on the task of contextual query suggestion, which requires understanding not only the user's current search context but also what they historically know and care about. Through a number of experiments based on human evaluation, we show that our approach is significantly better than several other LLM-powered baselines, generating query suggestions that are contextually more relevant, personalized, and useful

Binomial Confidence Intervals and Contingency Tests: Mathematical Fundamentals and the Evaluation of Alternative Methods

Many statistical methods rely on an underlying mathematical model of probability based on a simple approximation, one that is simultaneously well-known and yet frequently misunderstood. The Normal approximation to the Binomial distribution underpins a range of statistical tests and methods, including the calculation of accurate confidence intervals, performing goodness of fit and contingency tests, line- and model-fitting, and computational methods based upon these. A common mistake is in assuming that, since the probable distribution of error about the “true value” in the population is approximately Normally distributed, the same can be said for the error about an observation. This paper is divided into two parts: fundamentals and evaluation. First, we examine the estimation of confidence intervals using three initial approaches: the “Wald” (Normal) interval, the Wilson score interval and the “exact” Clopper-Pearson Binomial interval. Whereas the first two can be calculated directly from formulae, the Binomial interval must be approximated towards by computational search, and is computationally expensive. However this interval provides the most precise significance test, and therefore will form the baseline for our later evaluations. We also consider two further refinements: employing log-likelihood in intervals (also requiring search) and the effect of adding a continuity correction. Second, we evaluate each approach in three test paradigms. These are the single proportion interval or 2 × 1 goodness of fit test, and two variations on the common 2 × 2 contingency test. We evaluate the performance of each approach by a “practitioner strategy”. Since standard advice is to fall back to “exact” Binomial tests in conditions when approximations are expected to fail, we report the proportion of instances where one test obtains a significant result when the equivalent exact test does not, and vice versa, across an exhaustive set of possible values. We demonstrate that optimal methods are based on continuity-corrected versions of the Wilson interval or Yates’ test, and that commonly-held beliefs about weaknesses of tests are misleading. Log-likelihood, often proposed as an improvement on , performs disappointingly. Finally we note that at this level of precision we may distinguish two types of 2 2 test according to whether the independent variable partitions data into independent populations, and we make practical recommendations for their use

Analysis of Search on Clinical Narrative within the EHR

Electronic Health Records (EHRs) are used increasingly in the hospital and outpatient set- tings, and patients are amassing digitized clinical information. On one hand, aggregating all the patient's clinical information can greatly assist health care workers in making sound decisions. On the other hand, it can result in information overload, making it difficult to browse for information within the health record. Considering the time constraints clinicians face, one way to reduce information overload is through a search utility. However, traditional, free-text search engines within the EHR can potentially miss documents that do not contain the query but that are relevant to the clinical user's search. This dissertation aims at addressing this gap by analyzing within-patient search of the EHR and examining various semantic search approaches on clinical narrative. Our work consists of three studies where clinical users' search needs are examined, traditional string-matching is analyzed, and semantic search approaches on clinical narrative are evaluated. The first study applied a mixed method approach in order to provide a better understanding of clinical users' search needs within the EHR. It is comprised of a retrospective log analysis of search log files and a survey that was administered to clinical professionals within our institution. The log analysis attempts to categorize how users of a search system query for information, and the survey tries to understand users' search preferences. This study showed that clinical users were very interested in search functionality within the EHR and that various types of users utilize a search utility differently. Overall, most users searched for specific laboratory tests and diseases within the health record. The last two studies rely on a gold standard, which was developed specifically for this dissertation. The gold standard contained a document collection, a set of queries, and for each document/query pair, a relevance judgment. This gold standard was used to evaluate and compare different search models on clinical narrative. The second study conducted was an error analysis of the traditional, vector-space model search approach. The study examined the false positives and false negatives of this approach and categorized the errors in order to identify gaps that semantic approaches may fill. The last study was a systematic evaluation of five different semantic search approaches. These search methods consisted of distributional semantic approaches and an ontology-based approach. The study identified that a mixed topic modeling and vector-space model approach was the best performing search algorithm on our gold standard. All of these studies lay the foundation for us to gain a deeper understanding of information retrieval methods within the electronic health record. Ultimately, this will allow health care professionals to easily access pertinent patient information, which could result in better health care delivery