Fast and effective automated indexing is critical for search and personalized
services. Key phrases that consist of one or more words and represent the main
concepts of the document are often used for the purpose of indexing. In this
paper, we investigate the use of additional semantic features and
pre-processing steps to improve automatic key phrase extraction. These features
include the use of signal words and freebase categories. Some of these features
lead to significant improvements in the accuracy of the results. We also
experimented with 2 forms of document pre-processing that we call light
filtering and co-reference normalization. Light filtering removes sentences
from the document, which are judged peripheral to its main content.
Co-reference normalization unifies several written forms of the same named
entity into a unique form. We also needed a "Gold Standard" - a set of labeled
documents for training and evaluation. While the subjective nature of key
phrase selection precludes a true "Gold Standard", we used Amazon's Mechanical
Turk service to obtain a useful approximation. Our data indicates that the
biggest improvements in performance were due to shallow semantic features, news
categories, and rhetorical signals (nDCG 78.47% vs. 68.93%). The inclusion of
deeper semantic features such as Freebase sub-categories was not beneficial by
itself, but in combination with pre-processing, did cause slight improvements
in the nDCG scores.Comment: In 8th International Conference on Language Resources and Evaluation
(LREC 2012