Pretrained Language Model based Web Search Ranking: From Relevance to Satisfaction

Search engine plays a crucial role in satisfying users' diverse information needs. Recently, Pretrained Language Models (PLMs) based text ranking models have achieved huge success in web search. However, many state-of-the-art text ranking approaches only focus on core relevance while ignoring other dimensions that contribute to user satisfaction, e.g., document quality, recency, authority, etc. In this work, we focus on ranking user satisfaction rather than relevance in web search, and propose a PLM-based framework, namely SAT-Ranker, which comprehensively models different dimensions of user satisfaction in a unified manner. In particular, we leverage the capacities of PLMs on both textual and numerical inputs, and apply a multi-field input that modularizes each dimension of user satisfaction as an input field. Overall, SAT-Ranker is an effective, extensible, and data-centric framework that has huge potential for industrial applications. On rigorous offline and online experiments, SAT-Ranker obtains remarkable gains on various evaluation sets targeting different dimensions of user satisfaction. It is now fully deployed online to improve the usability of our search engine

Navigating Online Information Spaces with Lateral Reading: Lessons Learned from Two Librarians Working with Students and Educators

As online content’s credibility has gotten harder and harder to evaluate, librarians and other educators have been growing their strategies for teaching online source evaluation. One of those strategies is “lateral reading,” the practice of quickly evaluating a web source by seeing what others on the web say about that source. On the surface, lateral reading is quite simple. However, effective lateral reading often requires complex thinking. How will you search for information about a source? Which search results will you click on and how will you evaluate those sources? How will you decide what you trust and to what degree you trust it? In this session, two academic librarians will share about their experiences and lessons learned over their past four years of teaching lateral reading to undergraduate students and fellow educators. Participants will have the opportunity to practice lateral reading and to reflect on their related experiences. Presented at LibraryLink NJ Literacy Conferenc

Doctor of Philosophy

dissertationWith the steady increase in online shopping, more and more consumers are resorting to Product Search Engines and shopping sites such as Yahoo! Shopping, Google Product Search, and Bing Shopping as their first stop for purchasing goods online. These sites act as intermediaries between shoppers and merchants to drive user experience by enabling faceted search, comparison of products based on their specifications, and ranking of products based on their attributes. The success of these systems heavily relies on the variety and quality of the products that they present to users. In that sense, product catalogs are to online shopping what the Web index is to Web search. Therefore, comprehensive product catalogs are fundamental to the success of Product Search Engines. Given the large number of products and categories, and the speed at which they are released to the market, constructing and keeping catalogs up-to-date becomes a challenging task, calling for the need of automated techniques that do not rely on human intervention. The main goal of this dissertation is to automatically construct catalogs for product search engines. To achieve this goal, the following problems must be addressed by these search engines: (i) product synthesis-creation of product instances that conform with the catalog schema; (ii) product discovery- derivation of product instances for products whose schemata are not present in the catalog; (iii) schema synthesis- construction of schemata for new product categories. We propose an end-to-end framework that automates, to a great extent, these tasks. We present a detailed experimental evaluation using real data sets which shows that our framework is effective, scaling to a large number of products and categories, and resilient to noise that is inherent in Web data

Facilitating access to evidence: Primary Health Care Search Filter

Background: The evidence base developed by, and relevant to, primary health care (PHC) is rapidly increasing. With the wealth of literature available, searchers trying to find PHC-specific citations can feel overwhelmed. Objectives: Flinders Filters and the Primary Health Care Research & Information Service collaborated to develop a search filter enabling efficient and effective retrieval of relevant PHC literature. Methods: Stage 1 involved developing a PHC Search Filter in the OvidSP Medline platform using a rigor-ous experimental methodology. The search filter was then translated for Web-based ‘one-click searching’ in PubMed during Stage 2. Stage 3 involved planning and implementing a mixed-methods evaluation. Results: The search filter sensitivity was 77.0% with a post hoc relevance assessment of 78 .3%. Four months after its launch, a mixed-methods study evaluated the PHC Search Filter. With 90 respondents, analysis of data from the online survey demonstrated overarching benefits, a positive response to the tool and directions for further refinement of the PHC Search Filter. Discussion: Designing the PHC Search Filter follow ed an established method that ensures the tool offers a validated search strategy. Evaluation results suggest that the PHC Search Filter is a useful tool that is easy to navigate. Challenges for the Filter relate to access to full text articles, while challenges for the evaluation relate to the small sample size. Conclusions: The PHC Search Filter reduces the burden associated with literature searching, increases the value of the results that are received and provides a useful resource to improve the likelihood of incorporating relevant evidence into policy and practice

Knowing How Good Our Searches Are: An Approach Derived from Search Filter Development Methodology

This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐ Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.Objective – Effective literature searching is of paramount importance in supporting evidence based practice, research, and policy. Missed references can have adverse effects on outcomes. This paper reports on the development and evaluation of an online learning resource, designed for librarians and other interested searchers, presenting an evidence based approach to enhancing and testing literature searches. Methods – We developed and evaluated the set of free online learning modules for librarians called Smart Searching, suggesting the use of techniques derived from search filter development undertaken by the CareSearch Palliative Care Knowledge Network and its associated project Flinders Filters. The searching module content has been informed by the processes and principles used in search filter development. The self-paced modules are intended to help librarians and other interested searchers test the effectiveness of their literature searches, provide evidence of search performance that can be used to improve searches, as well as to evaluate and promote searching expertise. Each module covers one of four techniques, or core principles, employed in search filter development: (1) collaboration with subject experts; (2) use of a reference sample set; (3) term identification through frequency analysis; and (4) iterative testing. Evaluation of the resource comprised ongoing monitoring of web analytics to determine factors such as numbers of users and geographic origin; a user survey conducted online elicited qualitative information about the usefulness of the resource. Results – The resource was launched in May 2014. Web analytics show over 6,000 unique users from 101 countries (at 9 August 2015). Responses to the survey (n=50) indicated that 80% would recommend the resource to a colleague. Conclusions – An evidence based approach to searching, derived from search filter development methodology, has been shown to have value as an online learning resource. More information is needed about the reasons why people are using the resource beyond what could be ascertained by the survey results

Diving Below the Surface: A Layered Approach to Teaching Online Source Evaluation through Lateral and Critical Reading

As online environments have in many ways changed how information (including misinformation) is created and distributed, many educators have recognized a need for teaching new strategies for evaluating online sources for credibility and potential bias. Educators like Mike Caulfield and research groups like the Stanford History Education Group (SHEG) have stressed the need for “lateral reading,” a habit of fact-checking when initially evaluating a source. When reading laterally, a person doesn’t spend extensive time initially examining what a source says about itself; instead, they quickly move off of the site in question to look at what others have said about the source and determine if that source is worth a closer read. Lateral reading is a vital first step to checking a source’s credibility, particularly when the credibility or motivations of a source’s creators are unclear. It’s also an important part of critical reading in everyday life. The value of lateral reading was made evident in SHEG’s 2019 study, in which professional fact-checkers, who regularly practice “lateral reading,” far outperformed history professors and undergraduate college students in identifying misleading information. SHEG’s Civic Online Reasoning curriculum helps many middle and high school teachers integrate lateral reading strategies into curricula. In higher education, Mike Caulfield’s work on web literacy, in particular his book Web Literacy for Student Fact-Checkers and his SIFT strategy, offers helpful models for lateral reading. Despite lateral reading’s seeming simplicity (moving off a webpage to find other coverage), if done formulaically, it’s not nearly as effective as if done with critical thought. For example, once you’ve left the webpage in question, what do you search for and what do you click on? How do you read the sources you use to investigate the source in question? Personal beliefs and biases can also powerfully influence how accurately a person evaluates a source. As research on confirmation bias illustrates, people tend to give less scrutiny to sources that fit with their pre-existing beliefs than they do to sources that challenge those beliefs. Done well, lateral reading requires a range of complex analytical and metacognitive reading skills and strategies. If you dig more deeply into SHEG and Mike Caulfield’s extensive work on lateral reading, this becomes evident. However, overviews of lateral reading tend to neglect how multi-layered critical lateral reading really is. Research on students’ web evaluation skills suggests that the range of critical reading strategies that are part of lateral reading are usually not taught explicitly. In contrast, some studies indicate that students are taught checklist approaches to evaluating online sources that focus on superficial features of a website (e.g., CRAAP, RADCAB). A checklist mentality can prevent students from engaging in more critical reading and evaluation of sources. The complexities of web evaluation point to the need for teaching lateral reading as a kind of critical reading that involves analysis of a source’s larger context and purpose as well as metacognitive reflection. In this chapter, we—two academic public services librarians who liaise with different disciplinary areas (the humanities/social sciences and life sciences)—discuss a scaffolded approach to teaching web source evaluation that brings together lateral and critical reading strategies. (Though lateral and critical are not mutually exclusive categories, critical thinking that ideally occurs during lateral reading is often not taught explicitly.) More specifically, we share our experience with developing an online tutorial on lateral reading and evaluating students’ analytical reading practices while they completed the tutorial. Finally, we discuss pedagogical takeaways useful for teaching lateral reading and critical source evaluation in a range of contexts

A Review of Selected E-Recruiting Websites: Disability Accessibility Considerations

Ten job boards1 and 31 corporate E-recruiting websites were evaluated for accessibility for people with disabilities. The examination was performed using both an automated accessibility testing software (Bobby v3.2) and an examination of a sub-sample of the sites through a “simulated” application process. The simulated application process was performed utilizing only the information available to a screen reader and navigating the site using only keyboard commands, duplicating how a blind individual would typically navigate the web. The purpose of this second method was to see if it would be possible to successfully proceed through the entire multi-step job search and application process. None of the job board pages (home, job search, signup, or resumé submittal pages) evaluated by Bobby were found to be accessible. The vast majority of corporate E-recruiting sites also failed Bobby’s tests. The simulated application process evaluation was slightly more promising, but still only three of the nine job boards and three of the twelve corporate sites evaluated were accessible enough to work through the entire process of registration, job searching, resumé submittal, and application for a position. Many of the issues encountered could easily be corrected through the consistent use of alternative text for essential submit image buttons (i.e. “apply,” “post resumé”)

Lessons Learned: Recommendations for Establishing Critical Periodic Scientific Benchmarking

The dependence of life scientists on software has steadily grown in recent years. For many tasks, researchers have to decide which of the available bioinformatics software are more suitable for their specific needs. Additionally researchers should be able to objectively select the software that provides the highest accuracy, the best efficiency and the highest level of reproducibility when integrated in their research projects. Critical benchmarking of bioinformatics methods, tools and web services is therefore an essential community service, as well as a critical component of reproducibility efforts. Unbiased and objective evaluations are challenging to set up and can only be effective when built and implemented around community driven efforts, as demonstrated by the many ongoing community challenges in bioinformatics that followed the success of CASP. Community challenges bring the combined benefits of intense collaboration, transparency and standard harmonization. Only open systems for the continuous evaluation of methods offer a perfect complement to community challenges, offering to larger communities of users that could extend far beyond the community of developers, a window to the developments status that they can use for their specific projects. We understand by continuous evaluation systems as those services which are always available and periodically update their data and/or metrics according to a predefined schedule keeping in mind that the performance has to be always seen in terms of each research domain. We argue here that technology is now mature to bring community driven benchmarking efforts to a higher level that should allow effective interoperability of benchmarks across related methods. New technological developments allow overcoming the limitations of the first experiences on online benchmarking e.g. EVA. We therefore describe OpenEBench, a novel infra-structure designed to establish a continuous automated benchmarking system for bioinformatics methods, tools and web services. OpenEBench is being developed so as to cater for the needs of the bioinformatics community, especially software developers who need an objective and quantitative way to inform their decisions as well as the larger community of end-users, in their search for unbiased and up-to-date evaluation of bioinformatics methods. As such OpenEBench should soon become a central place for bioinformatics software developers, community-driven benchmarking initiatives, researchers using bioinformatics methods, and funders interested in the result of methods evaluation.Preprin

Developing Critical Thinking in online search

Digital skills especially those related to Information Literacy, are today considered fundamental to the education of students, both at school and at university. Searching and evaluating information found on the Internet is surely an important competency. An effective way to develop this competency is to educate students about the development of critical thinking. The article presents a qualitative-quantitative survey conducted during a course in Educational Technologies within a five year Degree program. The outcomes of the survey reveal some interesting behaviors and perceptions of students when they are faced with the Web search process and the characteristics of their critical thinking processes: some aspects of critical thinking are generally well supported, but others are acquired only after specific training. Experience shows that if properly motivated by metacognitive reflections and a clear method, students can actually critically evaluate the information presented online, the sources, and the sustainability of the arguments found. Positive results also occurred when the evaluation process was done in a collaborative modality