Benchmarking Joint Lexical and Syntactic Analysis on Multiword-Rich Data

International audienceThis article evaluates the extension of a dependency parser that performs joint syntactic analysis and multiword expression identification. We show that, given sufficient training data, the parser benefits from explicit multiword information and improves overall labeled accuracy score in eight of the ten evaluation cases

Benchmarking Joint Lexical and Syntactic Analysis on Multiword-Rich Data

International audienceThis article evaluates the extension of a dependency parser that performs joint syntactic analysis and multiword expression identification. We show that, given sufficient training data, the parser benefits from explicit multiword information and improves overall labeled accuracy score in eight of the ten evaluation cases

Indirectly Named Entity Recognition

[EN] We define here indirectly named entities, as a term to denote multiword expressions referring to known named entities by means of periphrasis.  While named entity recognition is a classical task in natural language processing, little attention has been paid to indirectly named entities and their treatment. In this paper, we try to address this gap, describing issues related to the detection and understanding of indirectly named entities in texts. We introduce a proof of concept for retrieving both lexicalised and non-lexicalised indirectly named entities in French texts. We also show example cases where this proof of concept is applied, and discuss future perspectives. We have initiated the creation of a first lexicon of 712 indirectly named entity entries that is available for future research.This research has been funded by the FEDER (Fonds européen de développement régional) and selected by the French-Swiss programme Interreg V. We would like to thank Claire Wuillemin for her preliminary work in the DecRIPT project about the State-of-the-Art in NER and SER in 2020. We would also like to thank for their advice Gilles Falquet, Luka Nerima, Eric Wehrli and Jean-Philippe Goldman at the University of Geneva.Kauffmann, A.; Rey, F.; Atanassova, I.; Gaudinat, A.; Greenfield, P.; Madinier, H.; Cardey, S. (2021). Indirectly Named Entity Recognition. Journal of Computer-Assisted Linguistic Research. 5(1):27-46. https://doi.org/10.4995/jclr.2021.15922OJS274651Abney, Steven. 1987. "The English Noun Phrase in its Sentential Aspect." PhD diss., Massachusetts Institute of Technology.Alsharaf, H., S. Cardey, P. Greenfield, D. Limame, and I. Skouratov. 2003. "Fixedness, the complexity and fragility of the phenomenon: some solutions for natural language processing." In Proceedings of ICL17. Prague, Czech Republic: Matfyzpress.Ananthanarayanan, Rema, Vijil Chenthamarakshan, Prasad M Deshpande, and Raghuram Krishnapuram. 2008. "Rule Based Synonyms for Entity Extraction from Noisy Text." In Proceedings of the Second Workshop on Analytics for Noisy Unstructured Text Data AND '08, 31-38. Singapore: Association for Computing Machinery. https://doi.org/10.1145/1390749.1390756Bachellier, Jean-Louis. 1972. "Sur-Nom." Le texte: de la théorie à la recherche, no. 19: 69-92. doi :10.3406/comm.1972.1283. https://doi.org/10.3406/comm.1972.1283Baldwin, Timothy, and Su Nam Kim. 2013. "Multiword Expressions." In Handbook of Natural Language Processing, Second Edition, edited by Nitin Indurkhya and Fred J. Damerau, 267-292. Boca Raton, USA: CRCPress.Bohn, C., and Kjeti Nørvag. 2010. "Extracting Named Entities and Synonyms from Wikipedia." In Proceedings of the 24th IEEE International Conference on Advanced Information Networking and Applications, 1300-1307. https://doi.org/10.1109/AINA.2010.50Cai, Desheng, and Gongqing Wu. 2019. "Content-aware attributed entity embedding for synonymous named entity discovery." Neurocomputing 329: 237-247. https://doi.org/10.1016/j.neucom.2018.10.055Chakrabarti, K., S. Chaudhuri, T. Cheng, and Dong Xin. 2012. "A framework for robust discovery of entity synonyms." In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1384-1392, Beijing, China: Association for Computing Machinery. https://doi.org/10.1145/2339530.2339743Charton, Eric, Michel Gagnon, and Benoit Ozell. 2011. "Génération automatique de motifs de détection d'entités nommées en utilisant des contenus encyclopédiques (Automatic generation of named entity detection patterns using encyclopedic contents)" [in French]. In Actes de la 18e conférence sur le Traitement Automatique des Langues Naturelles. Articles longs, 13-24. Montpellier, France: ATALA.Cho, Hyejin, Wonjun Choi, and Hyunju Lee. 2017. "A method for named entity normalization in biomedical articles: application to diseases and plants." BMC bioinformatics 18, no. 1 ( 1-12. https://doi.org/10.1186/s12859-017-1857-8Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171-4186. Minneapolis, Minnesota: Association for Computational Linguistics.Friburger, Nathalie. 2006. "Linguistique et reconnaissance automatique des noms propres." Meta 51, no. 4: 637-650. doi:10.7202/014331ar. https://doi.org/10.7202/014331arGuenoune, Hani, Kevin Cousot, Mathieu Lafourcade, Melissa Mekaoui, and Cédric Lopez. 2020. "A Dataset for Anaphora Analysis in French Emails." In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference, 165-175. Barcelona, Spain (online): Association for Computational Linguistics.Honnibal, Matthew, and Ines Montani. 2017. "spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing."Kampeera, Wannachai, and Sylviane Cardey-Greenfield. 2012. "Building a Lexically and Semantically-Rich Resource for Paraphrase Processing." In Advances in Natural Language Processing, edited by Hitoshi Isahara and Kyoko Kanzaki, 138-143. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-33983-7_14Kauffmann, Alexis. 2013. "Structural Asymmetries in Machine Translation: The case of English-Japanese". PhD diss., Université de Genève. https://doi.org/10.13097/archive-ouverte/unige:34540.Lample, Guillaume, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. "Neural Architectures for Named Entity Recognition." In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 260-270. San Diego, California: Association for Computational Linguistics. https://doi.org/10.18653/v1/N16-1030Lin, Bill Yuchen, Dong-Ho Lee, M. Shen, Ryan Rene Moreno, X. Huang, Prashant Shiralkar, and X. Ren. 2020. "TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8503-8511. Online: Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.752Lopez, C., Melissa Mekaoui, K. Aubry, Jean Bort, and Philippe Garnier. 2019. "Reconnaissance d'entités nommées itérative sur une structure en dépendances syntaxiques avec l'ontologie NERD." Revue des Nouvelles Technologies de l'Information, Extraction et Gestion des connaissances, RNTI-E-35, 81-92.Ma, Jie, Jun Liu, Y. Li, X. Hu, Yudai Pan, S. Sun, and Qika Lin. 2020. "Jointly Optimized Neural Coreference Resolution with Mutual Attention." In Proceedings of the 13th International Conference on Web Search and Data Mining. Houston, Texas, USA: Association for Computing Machinery. https://doi.org/10.1145/3336191.3371787Manning, Christopher D., Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55-60. Baltimore, Maryland: Association for Computational Linguistics. https://doi.org/10.3115/v1/P14-5010Martin, Louis, Benjamin Muller, Pedro Javier Ortiz Suarez, Yoann Dupont, Laurent Romary, Eric Villemonte de la Clergerie, Benoıt Sagot, and Djamé Seddah. 2020. "Les modèles de langue contextuels CamemBERT pour le français: impact de la taille et de l'hétérogénéité des données d'entrainement (CamemBERT Contextual Language Models for French: Impact of Training Data Size and Heterogeneity)" [in French]. In Actes de la 6e conférence conjointe Journées d'Etudes sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Etudiants Chercheurs en Informatique pour le' Traitement Automatique des Langues (RECITAL, 22e édition). Volume 2: Traitement Automatique des Langues Naturelles, 54-65. Nancy, France: ATALA et AFCP.Mitkov, Ruslan. 2014. Anaphora resolution. Routledge. https://doi.org/10.4324/9781315840086Mohamed, Muhidin A., and Mourad Chabane Oussalah. 2020. "A hybrid approach for paraphrase identification based on knowledge-enriched semantic heuristics." Language Resources and Evaluation 54 : 457-485. https://doi.org/10.1007/s10579-019-09466-4Nadeau, David, and Satoshi Sekine. 2007. "A survey of named entity recognition and classification." Lingvisticae Investigationes 30: 3-26. https://doi.org/10.1075/li.30.1.03nadNayel, Hamada A., H. L. Shashirekha, Hiroyuki Shindo, and Yuji Matsumoto. 2019. "Improving Multi-Word Entity Recognition for Biomedical Texts." CoRRabs/1908.05691. arXiv:1908.05691.Nebhi, Kamel. 2013. "Named Entity Disambiguation using Freebase and Syntactic Parsing." In [email protected], Damien, Maud Ehrmann, and Sophie Rosset. 2016. "Evaluating Named Entity Recognition." Chap. 6 in Named Entities for Computational Linguistics, 111-129. John Wiley & Sons, Ltd. https://doi.org/10.1002/9781119268567.ch6Ortiz Suarez, Pedro Javier, Yoann Dupont, Benjamin Muller, Laurent Romary, and Benoıt Sagot. 2020. "Establishing a New State-of-the-Art for French Named Entity Recognition" [in English]. In Proceedings of the 12th Language Resources and Evaluation Conference, 4631-4638. Marseille, France: European Language Resources Association.Petit, Gérard. 2006. "Le nom de marque déposée : nom propre, nom commun et terme." Meta 51, no. 4: 690-705. doi:10.7202/014335ar. https://doi.org/10.7202/014335arQu, Meng, Xiang Ren, and Jiawei Han. 2017. "Automatic Synonym Discovery with Knowledge Bases." In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 997-1005. KDD '17. Halifax, NS, Canada: Association for Computing Machinery. https://doi.org/10.1145/3097983.3098185Racicot, André. 2009. "Traduire le monde: Venise du Nord et autres surnoms." L'Actualité langagière, vol. 6, n° 2, 23. Travaux publics et Services gouvernementaux Canada.Rey, François-Claude, and Kauffmann Alexis. 2021. "French indirectly named entities (version 1.3) [Data set]." Zenodo. https://doi.org/10.5281/zenodo.5158253.Rosales-Méndez, Henry, Aidan Hogan, and Barbara Poblete. 2019. "Fine-Grained Evaluation for Entity Linking." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 718-727. Hong Kong, China: Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1066Sales, Juliano Efson, André Freitas, Brian Davis, and Siegfried Handschuh. 2016. "A Compositional-Distributional Semantic Model for Searching Complex Entity Categories." In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, 199-208. Berlin, Germany: Association for Computational Linguistics. https://doi.org/10.18653/v1/S16-2025Schmitt, X., S. Kubler, J. Robert, M. Papadakis, and Y. LeTraon. 2019. "A Replicable Comparison Study of NER Software: StanfordNLP, NLTK, OpenNLP, SpaCy, Gate." In Proceedings of the Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS), 338-343. https://doi.org/10.1109/SNAMS.2019.8931850Shang, Jingbo, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. "Learning Named Entity Tagger using Domain-Specific Dictionary." In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2054-2064. Brussels, Belgium: Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1230Shen, Jiaming, Ruiliang Lyu, Xiang Ren, Michelle Vanni, Brian Sadler, and Jiawei Han. 2019. "Mining entity synonyms with efficient neural set generation." In Proceedings of the AAAI Conference on Artificial Intelligence, 33:249-256. doi:10.1609/aaai.v33i01.3301249. https://doi.org/10.1609/aaai.v33i01.3301249Shinyama, Yusuke, Satoshi Sekine, and Kiyoshi Sudo. 2002. "Automatic Paraphrase Acquisition from News Articles." In Proceedings of the Second International Conference on Human Language Technology Research, 313-318. HLT '02. San Diego, California: Morgan Kaufmann Publishers Inc. https://doi.org/10.3115/1289189.1289218Sjöblom, Paula. 2016. "Commercial names." Chap. V.31 in The Oxford Handbook of Names and Naming, edited by Carole Hough, 453-464. Oxford, UK: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199656431.013.56Tenney, Ian, Dipanjan Das, and Ellie Pavlick. 2019. "BERT Rediscovers the Classical NLP Pipeline." In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4593-4601. Florence, Italy: Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1452Treps, Marie. 2012. La rançon de la gloire - Les surnoms de nos politiques. Paris, France: Editions du Seuil.Watanabe, Taiki, Akihiro Tamura, Takashi Ninomiya, Takuya Makino, and Tomoya Iwakura. 2019. "Multi-Task Learning for Chemical Named Entity Recognition with Chemical Compound Paraphrasing." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 6244-6249. Hong Kong, China: Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1648Wehrli, Eric, and Luka Nerima. 2018. "Anaphora resolution, collocations and translation." In Multiword units in machine translation and translation technology, edited by Johanna Monti, Violeta Seretan, Gloria Corpas Pastor, and Ruslan Mitkov, 244-256. John Benjamins. https://doi.org/10.1075/cilt.341.12wehWehrli, Eric, Violeta Seretan, and Luka Nerima. 2010. "Sentence Analysis and Collocation Identification." In Proceedings of the 2010 Workshop on Multiword Expressions: from Theory to Applications, 28-36. Beijing, China: Coling 2010 Organizing Committee.Weston, L., V. Tshitoyan, J. Dagdelen, O. Kononova, A. Trewartha, K. A. Persson, G. Ceder, and A. Jain. 2019. "Named Entity Recognition and Normalization Applied to Large-Scale Information Extraction from the Materials Science Literature." Journal of Chemical Information and Modeling 59, no. 9: 3692-3702. doi: 10.1021/acs.jcim.9b00470. https://doi.org/10.1021/acs.jcim.9b00470Wu, G., Y. He, and X. Hu. 2018. "Entity Linking: An Issue to Extract Corresponding Entity With Knowledge Base." IEEE Access 6: 6220-6231. doi:10.1109/ACCESS.2017.2787787. https://doi.org/10.1109/ACCESS.2017.2787787Yang, Yiying, Xi Yin, Haiqin Yang, Xingjian Fei, Hao Peng, Kaijie Zhou, Kunfeng Lai, and Jianping Shen. 2021. "KGSynNet: A Novel Entity Synonyms Discovery Framework with Knowledge Graph." In Database Systems for Advanced Applications, edited by Christian S. Jensen, Ee-Peng Lim, De-Nian Yang, Wang-Chien Lee, Vincent S. Tseng, Vana Kalogeraki, Jen-Wei Huang, and Chih-Ya Shen, 174-190. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-73194-6_13Zhang, Ruoyu, Wenpeng Lu, Shoujin Wang, Xueping Peng, Rui Yu, and Yuan Gao. 2021. "Chinese clinical named entity recognition based on stacked neural network." Concurrency and Computation: Practice and Experience : 33:e5775. doi:10.1002/cpe.5775. https://doi.org/10.1002/cpe.577

Multiword expression processing: A survey

Multiword expressions (MWEs) are a class of linguistic forms spanning conventional word boundaries that are both idiosyncratic and pervasive across different languages. The structure of linguistic processing that depends on the clear distinction between words and phrases has to be re-thought to accommodate MWEs. The issue of MWE handling is crucial for NLP applications, where it raises a number of challenges. The emergence of solutions in the absence of guiding principles motivates this survey, whose aim is not only to provide a focused review of MWE processing, but also to clarify the nature of interactions between MWE processing and downstream applications. We propose a conceptual framework within which challenges and research contributions can be positioned. It offers a shared understanding of what is meant by "MWE processing," distinguishing the subtasks of MWE discovery and identification. It also elucidates the interactions between MWE processing and two use cases: Parsing and machine translation. Many of the approaches in the literature can be differentiated according to how MWE processing is timed with respect to underlying use cases. We discuss how such orchestration choices affect the scope of MWE-aware systems. For each of the two MWE processing subtasks and for each of the two use cases, we conclude on open issues and research perspectives

Current trends

Deep parsing is the fundamental process aiming at the representation of the syntactic structure of phrases and sentences. In the traditional methodology this process is based on lexicons and grammars representing roughly properties of words and interactions of words and structures in sentences. Several linguistic frameworks, such as Headdriven Phrase Structure Grammar (HPSG), Lexical Functional Grammar (LFG), Tree Adjoining Grammar (TAG), Combinatory Categorial Grammar (CCG), etc., offer different structures and combining operations for building grammar rules. These already contain mechanisms for expressing properties of Multiword Expressions (MWE), which, however, need improvement in how they account for idiosyncrasies of MWEs on the one hand and their similarities to regular structures on the other hand. This collaborative book constitutes a survey on various attempts at representing and parsing MWEs in the context of linguistic theories and applications

Representation and parsing of multiword expressions

This book consists of contributions related to the definition, representation and parsing of MWEs. These reflect current trends in the representation and processing of MWEs. They cover various categories of MWEs such as verbal, adverbial and nominal MWEs, various linguistic frameworks (e.g. tree-based and unification-based grammars), various languages including English, French, Modern Greek, Hebrew, Norwegian), and various applications (namely MWE detection, parsing, automatic translation) using both symbolic and statistical approaches

Semantic Representation and Inference for NLP

Semantic representation and inference is essential for Natural Language Processing (NLP). The state of the art for semantic representation and inference is deep learning, and particularly Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and transformer Self-Attention models. This thesis investigates the use of deep learning for novel semantic representation and inference, and makes contributions in the following three areas: creating training data, improving semantic representations and extending inference learning. In terms of creating training data, we contribute the largest publicly available dataset of real-life factual claims for the purpose of automatic claim verification (MultiFC), and we present a novel inference model composed of multi-scale CNNs with different kernel sizes that learn from external sources to infer fact checking labels. In terms of improving semantic representations, we contribute a novel model that captures non-compositional semantic indicators. By definition, the meaning of a non-compositional phrase cannot be inferred from the individual meanings of its composing words (e.g., hot dog). Motivated by this, we operationalize the compositionality of a phrase contextually by enriching the phrase representation with external word embeddings and knowledge graphs. Finally, in terms of inference learning, we propose a series of novel deep learning architectures that improve inference by using syntactic dependencies, by ensembling role guided attention heads, incorporating gating layers, and concatenating multiple heads in novel and effective ways. This thesis consists of seven publications (five published and two under review).Comment: PhD thesis, the University of Copenhage

R&I smart specialisation strategies: classification of EU regions’ priorities. Results from automatic text analysis

Building on automatic text analysis, this paper proposes an original categorization of Research and Innovation Smart Specialisation Strategy (RIS3) priorities and provides a common language (with detailed dictionaries) to classify priorities and then to associate EU regions to multiclass categories of priorities. This result is a powerful tool to interpret the current state of diversification across regions, with its potential of complementarities and synergies that might support territorial integrated development paths. It would also support regions in their future strategic programmes on RIS3. A case study on the Alpine macro-region shows innovation development paths to outline macroregion strategic planning

Contextual compositionality detection with external knowledge bases and word embeddings

When the meaning of a phrase cannot be inferred from the individual meanings of its words (e.g., hot dog), that phrase is said to be non-compositional. Automatic compositionality detection in multiword phrases is critical in any application of semantic processing, such as search engines [9]; failing to detect non-compositional phrases can hurt system effectiveness notably. Existing research treats phrases as either compositional or non-compositional in a deterministic manner. In this paper, we operationalize the viewpoint that compositionality is contextual rather than deterministic, i.e., that whether a phrase is compositional or non-compositional depends on its context. For example, the phrase \ufffdgreen card\ufffd is compositional when referring to a green colored card, whereas it is non-compositional when meaning permanent residence authorization. We address the challenge of detecting this type of contextual compositionality as follows: given a multi-word phrase, we enrich the word embedding representing its semantics with evidence about its global context (terms it often collocates with) as well as its local context (narratives where that phrase is used, which we call usage scenarios). We further extend this representation with information extracted from external knowledge bases. The resulting representation incorporates both localized context and more general usage of the phrase and allows to detect its compositionality in a non-deterministic and contextual way. Empirical evaluation of our model on a dataset of phrase compositionality1, manually collected by crowdsourcing contextual compositionality assessments, shows that our model outperforms state-of-the-art baselines notably on detecting phrase compositionality