Advocates for Neuro-Symbolic Artificial Intelligence (NeSy) assert that
combining deep learning with symbolic reasoning will lead to stronger AI than
either paradigm on its own. As successful as deep learning has been, it is
generally accepted that even our best deep learning systems are not very good
at abstract reasoning. And since reasoning is inextricably linked to language,
it makes intuitive sense that Natural Language Processing (NLP), would be a
particularly well-suited candidate for NeSy. We conduct a structured review of
studies implementing NeSy for NLP, with the aim of answering the question of
whether NeSy is indeed meeting its promises: reasoning, out-of-distribution
generalization, interpretability, learning and reasoning from small data, and
transferability to new domains. We examine the impact of knowledge
representation, such as rules and semantic networks, language structure and
relational structure, and whether implicit or explicit reasoning contributes to
higher promise scores. We find that systems where logic is compiled into the
neural network lead to the most NeSy goals being satisfied, while other factors
such as knowledge representation, or type of neural architecture do not exhibit
a clear correlation with goals being met. We find many discrepancies in how
reasoning is defined, specifically in relation to human level reasoning, which
impact decisions about model architectures and drive conclusions which are not
always consistent across studies. Hence we advocate for a more methodical
approach to the application of theories of human reasoning as well as the
development of appropriate benchmarks, which we hope can lead to a better
understanding of progress in the field. We make our data and code available on
github for further analysis.Comment: Surve