Recognizing Textual Entailment (RTE) was proposed as a unified evaluation
framework to compare semantic understanding of different NLP systems. In this
survey paper, we provide an overview of different approaches for evaluating and
understanding the reasoning capabilities of NLP systems. We then focus our
discussion on RTE by highlighting prominent RTE datasets as well as advances in
RTE dataset that focus on specific linguistic phenomena that can be used to
evaluate NLP systems on a fine-grained level. We conclude by arguing that when
evaluating NLP systems, the community should utilize newly introduced RTE
datasets that focus on specific linguistic phenomena.Comment: 1st Workshop on Evaluation and Comparison for NLP systems (Eval4NLP)
at EMNLP 2020; 18 page