Context-Based Quotation Recommendation

While composing a new document, anything from a news article to an email or essay, authors often utilize direct quotes from a variety of sources. Although an author may know what point they would like to make, selecting an appropriate quote for the specific context may be time-consuming and difficult. We therefore propose a novel context-aware quote recommendation system which utilizes the content an author has already written to generate a ranked list of quotable paragraphs and spans of tokens from a given source document. We approach quote recommendation as a variant of open-domain question answering and adapt the state-of-the-art BERT-based methods from open-QA to our task. We conduct experiments on a collection of speech transcripts and associated news articles, evaluating models' paragraph ranking and span prediction performances. Our experiments confirm the strong performance of BERT-based methods on this task, which outperform bag-of-words and neural ranking baselines by more than 30% relative across all ranking metrics. Qualitative analyses show the difficulty of the paragraph and span recommendation tasks and confirm the quotability of the best BERT model's predictions, even if they are not the true selected quotes from the original news articles.Comment: 12 pages, 3 figure

Enhancing Prediction Reliability Of Deep Learning By Data Confidence For Recommendation Systems: A Case Study On Named Entity Recognition

Reliability is crucial for industrial recommendation systems. Recent advancement in deep neural networks has greatly improved the performance of modern recommendation systems. However, there is a lack of research on estimating how reliable such recommendation systems are in practical scenarios. Due to the blackbox nature of the deep learning-based systems, many times additional labor has to be involved to examine the prediction accuracy manually, which is costly and time-consuming. To address the problem, we propose a novel approach to estimate the model confidence for a deep learning-based recommendation system. Our approach utilized data statistics to improve the traditional model confidence estimation and maintain the model’s high performance. We further proposed a new evaluation metric to properly compare different prediction confidence estimation approaches. Experimental results showed that the external data statistics could effectively improve the prediction reliability by increasing confidence score, which will lead to significant reduction of the time and labors on the system’s prediction result examination. Index Terms—Prediction Reliability, Recommendation Systems, Deep Learning, Data Confidence, Named Entity Recognitio