7 research outputs found
Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search
Despite substantial interest in applications of neural networks to
information retrieval, neural ranking models have only been applied to standard
ad hoc retrieval tasks over web pages and newswire documents. This paper
proposes MP-HCNN (Multi-Perspective Hierarchical Convolutional Neural Network)
a novel neural ranking model specifically designed for ranking short social
media posts. We identify document length, informal language, and heterogeneous
relevance signals as features that distinguish documents in our domain, and
present a model specifically designed with these characteristics in mind. Our
model uses hierarchical convolutional layers to learn latent semantic
soft-match relevance signals at the character, word, and phrase levels. A
pooling-based similarity measurement layer integrates evidence from multiple
types of matches between the query, the social media post, as well as URLs
contained in the post. Extensive experiments using Twitter data from the TREC
Microblog Tracks 2011--2014 show that our model significantly outperforms prior
feature-based as well and existing neural ranking models. To our best
knowledge, this paper presents the first substantial work tackling search over
social media posts using neural ranking models.Comment: AAAI 2019, 10 page
Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models
Is neural IR mostly hype? In a recent SIGIR Forum article, Lin expressed
skepticism that neural ranking models were actually improving ad hoc retrieval
effectiveness in limited data scenarios. He provided anecdotal evidence that
authors of neural IR papers demonstrate "wins" by comparing against weak
baselines. This paper provides a rigorous evaluation of those claims in two
ways: First, we conducted a meta-analysis of papers that have reported
experimental results on the TREC Robust04 test collection. We do not find
evidence of an upward trend in effectiveness over time. In fact, the best
reported results are from a decade ago and no recent neural approach comes
close. Second, we applied five recent neural models to rerank the strong
baselines that Lin used to make his arguments. A significant improvement was
observed for one of the models, demonstrating additivity in gains. While there
appears to be merit to neural IR approaches, at least some of the gains
reported in the literature appear illusory.Comment: Published in the Proceedings of the 42nd Annual International ACM
SIGIR Conference on Research and Development in Information Retrieval (SIGIR
2019
End-to-end Neural Information Retrieval
In recent years we have witnessed many successes of neural networks in the information
retrieval community with lots of labeled data. Yet it remains unknown whether the same
techniques can be easily adapted to search social media posts where the text is much
shorter. In addition, we find that most neural information retrieval models are compared
against weak baselines. In this thesis, we build an end-to-end neural information retrieval
system using two toolkits: Anserini and MatchZoo. In addition, we also propose a novel
neural model to capture the relevance of short and varied tweet text, named MP-HCNN.
With the information retrieval toolkit Anserini, we build a reranking architecture based
on various traditional information retrieval models (QL, QL+RM3, BM25, BM25+RM3),
including a strong pseudo-relevance feedback baseline: RM3. With the neural network
toolkit MatchZoo, we offer an empirical study of a number of popular neural network
ranking models (DSSM, CDSSM, KNRM, DUET, DRMM). Experiments on datasets from
the TREC Microblog Tracks and the TREC Robust Retrieval Track show that most
existing neural network models cannot beat a simple language model baseline. How-
ever, DRMM provides a significant improvement over the pseudo-relevance feedback baseline
(BM25+RM3) on the Robust04 dataset and DUET, DRMM and MP-HCNN can provide
significant improvements over the baseline (QL+RM3) on the microblog datasets. Further
detailed analyses suggest that searching social media and searching news articles exhibit
several different characteristics that require customized model design, shedding light on
future directions
Cross-Domain Sentence Modeling for Relevance Transfer with BERT
Standard bag-of-words term-matching techniques in document retrieval fail to exploit rich semantic information embedded in the document texts. One promising recent trend in facilitating context-aware semantic matching has been the development of massively pretrained deep transformer models, culminating in BERT as their most popular example today. In this work, we propose adapting BERT as a neural re-ranker for document retrieval to achieve large improvements on news articles. Two fundamental issues arise in applying BERT to ``ad hoc'' document retrieval on newswire collections: relevance judgments in existing test collections are provided only at the document level, and documents often exceed the length that BERT was designed to handle. To overcome these challenges, we compute and aggregate sentence-level evidence to rank documents. The lack of appropriate relevance judgments in test collections is addressed by leveraging sentence-level and passage-level relevance judgments fortuitously available in collections from other domains to capture cross-domain notions of relevance. Our experiments demonstrate that models of relevance can be transferred across domains. By leveraging semantic cues learned across various domains, we propose a model that achieves state-of-the-art results on three standard TREC newswire collections. We explore the effects of cross-domain relevance transfer, and trade-offs between using document and sentence scores for document ranking. We also present an end-to-end document retrieval system that integrates the open-source Anserini information retrieval toolkit, discussing the related technical challenges and design decisions