7,621 research outputs found
Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models
Is neural IR mostly hype? In a recent SIGIR Forum article, Lin expressed
skepticism that neural ranking models were actually improving ad hoc retrieval
effectiveness in limited data scenarios. He provided anecdotal evidence that
authors of neural IR papers demonstrate "wins" by comparing against weak
baselines. This paper provides a rigorous evaluation of those claims in two
ways: First, we conducted a meta-analysis of papers that have reported
experimental results on the TREC Robust04 test collection. We do not find
evidence of an upward trend in effectiveness over time. In fact, the best
reported results are from a decade ago and no recent neural approach comes
close. Second, we applied five recent neural models to rerank the strong
baselines that Lin used to make his arguments. A significant improvement was
observed for one of the models, demonstrating additivity in gains. While there
appears to be merit to neural IR approaches, at least some of the gains
reported in the literature appear illusory.Comment: Published in the Proceedings of the 42nd Annual International ACM
SIGIR Conference on Research and Development in Information Retrieval (SIGIR
2019
Report on the Information Retrieval Festival (IRFest2017)
The Information Retrieval Festival took place in April 2017 in Glasgow. The focus of the workshop was to bring together IR researchers from the various Scottish universities and beyond in order to facilitate more awareness, increased interaction and reflection on the status of the field and its future. The program included an industry session, research talks, demos and posters as well as two keynotes. The first keynote was delivered by Prof. Jaana Kekalenien, who provided a historical, critical reflection of realism in Interactive Information Retrieval Experimentation, while the second keynote was delivered by Prof. Maarten de Rijke, who argued for more Artificial Intelligence usage in IR solutions and deployments. The workshop was followed by a "Tour de Scotland" where delegates were taken from Glasgow to Aberdeen for the European Conference in Information Retrieval (ECIR 2017
The Lucene for Information Access and Retrieval Research (LIARR) Workshop at SIGIR 2017
As an empirical discipline, information access and retrieval research requires substantial software infrastructure to index and search large collections. This workshop is motivated by the desire to better align information retrieval research with the practice of building search applications from the perspective of open-source information retrieval systems. Our goal is to promote the use of Lucene for information access and retrieval research
Towards Query Logs for Privacy Studies: On Deriving Search Queries from Questions
Translating verbose information needs into crisp search queries is a
phenomenon that is ubiquitous but hardly understood. Insights into this process
could be valuable in several applications, including synthesizing large
privacy-friendly query logs from public Web sources which are readily available
to the academic research community. In this work, we take a step towards
understanding query formulation by tapping into the rich potential of community
question answering (CQA) forums. Specifically, we sample natural language (NL)
questions spanning diverse themes from the Stack Exchange platform, and conduct
a large-scale conversion experiment where crowdworkers submit search queries
they would use when looking for equivalent information. We provide a careful
analysis of this data, accounting for possible sources of bias during
conversion, along with insights into user-specific linguistic patterns and
search behaviors. We release a dataset of 7,000 question-query pairs from this
study to facilitate further research on query understanding.Comment: ECIR 2020 Short Pape
Searching Spontaneous Conversational Speech
The ACM SIGIR Workshop on Searching Spontaneous Conversational Speech was held as part of the 2007 ACM SIGIR Conference in Amsterdam.\ud
The workshop program was a mix of elements, including a keynote speech, paper presentations and panel discussions. This brief report describes the organization of this workshop and summarizes the discussions
SIGIR: scholar vs. scholars' interpretation
Google Scholar allows researchers to search through a free and extensive source of information on scientific publications. In this paper we show that within the limited context of SIGIR proceedings, the rankings created by Google Scholar are both significantly different and very negatively correlated with those of domain experts
Fourth International Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse
© ACM, 2011. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM SIGIR Forum (2011) http://doi.acm.org/10.1145/1988852.1988860[EN] The Fourth International Workshop on Uncovering Plagiarism, Authorship, and Social
Software Misuse (PAN 10) was held in conjunction with the 2010 Conference on Multilingual
and Multimodal Information Access Evaluation (CLEF-10) in Padua, Italy. The workshop
was organized as a competition covering two tasks: plagiarism detection and Wikipedia
vandalism detection. This report gives a short overview of the plagiarism detection task.
Detailed analyses of both tasks have been published as CLEF Notebook Papers [3, 6], which
can be downloaded at www.webis.de/publications.Our special thanks go to the participants of the competition for their devoted work. We also
thank Yahoo! Research for their sponsorship. This work is partially funded by CONACYTMexico
and the MICINN project TEXT-ENTERPRISE 2.0 TIN2009-13391-C04-03 (Plan
I+D+i).Stein, B.; Rosso, P.; Stamatatos, E.; Potthast, M.; Barrón Cedeño, LA.; Koppel, M. (2011). Fourth International Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse. ACM SIGIR Forum. 45(1):45-48. https://doi.org/10.1145/1988852.1988860S454845
Benchmarking news recommendations: the CLEF NewsREEL use case
The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms. The goal is to create an algorithm that is able to generate news items that users would click, respecting a strict time constraint. The lab challenges participants to compete in either a "living lab" (Task 1) or perform an evaluation that replays recorded streams (Task 2). In this report, we discuss the objectives and challenges of the NewsREEL lab, summarize last year's campaign and outline the main research challenges that can be addressed by participating in NewsREEL 2016
Online Forum Thread Retrieval using Pseudo Cluster Selection and Voting Techniques
Online forums facilitate knowledge seeking and sharing on the Web. However,
the shared knowledge is not fully utilized due to information overload. Thread
retrieval is one method to overcome information overload. In this paper, we
propose a model that combines two existing approaches: the Pseudo Cluster
Selection and the Voting Techniques. In both, a retrieval system first scores a
list of messages and then ranks threads by aggregating their scored messages.
They differ on what and how to aggregate. The pseudo cluster selection focuses
on input, while voting techniques focus on the aggregation method. Our combined
models focus on the input and the aggregation methods. The result shows that
some combined models are statistically superior to baseline methods.Comment: The original publication is available at
http://www.springerlink.com/. arXiv admin note: substantial text overlap with
arXiv:1212.533
Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2017)
The large scale of scholarly publications poses a challenge for scholars in
information seeking and sensemaking. Bibliometrics, information retrieval (IR),
text mining and NLP techniques could help in these search and look-up
activities, but are not yet widely used. This workshop is intended to stimulate
IR researchers and digital library professionals to elaborate on new approaches
in natural language processing, information retrieval, scientometrics, text
mining and recommendation techniques that can advance the state-of-the-art in
scholarly document understanding, analysis, and retrieval at scale. The BIRNDL
workshop at SIGIR 2017 will incorporate an invited talk, paper sessions and the
third edition of the Computational Linguistics (CL) Scientific Summarization
Shared Task.Comment: 2 pages, workshop paper accepted at the SIGIR 201
- …