40 research outputs found
HuSpaCy : an industrial-strength Hungarian natural language processing toolkit
Although there are a couple of open-source language processing pipelines available for Hungarian, none of them satisfies the requirements of today’s NLP applications. A language processing pipeline should consist of close to state-of-the-art lemmatization, morphosyntactic analysis, entity recognition and word embeddings. Industrial text processing applications have to satisfy non-functional software quality requirements, what is more, frameworks supporting multiple languages are more and more favored. This paper introduces HuSpaCy, an industryready Hungarian language processing toolkit. The presented tool provides components for the most important basic linguistic analysis tasks. It is open-source and is available under a permissive license. Our system is built upon spaCy’s NLP components resulting in an easily usable, fast yet accurate application. Experiments confirm that HuSpaCy has high accuracy while maintaining resource-efficient prediction capabilities
Advancing Hungarian Text Processing with HuSpaCy: Efficient and Accurate NLP Pipelines
This paper presents a set of industrial-grade text processing models for
Hungarian that achieve near state-of-the-art performance while balancing
resource efficiency and accuracy. Models have been implemented in the spaCy
framework, extending the HuSpaCy toolkit with several improvements to its
architecture. Compared to existing NLP tools for Hungarian, all of our
pipelines feature all basic text processing steps including tokenization,
sentence-boundary detection, part-of-speech tagging, morphological feature
tagging, lemmatization, dependency parsing and named entity recognition with
high accuracy and throughput. We thoroughly evaluated the proposed
enhancements, compared the pipelines with state-of-the-art tools and
demonstrated the competitive performance of the new models in all text
preprocessing steps. All experiments are reproducible and the pipelines are
freely available under a permissive license.Comment: Submitted to TSD 2023 Conferenc
Hybrid lemmatization in HuSpaCy
Lemmatization is still not a trivial task for morphologically rich languages.
Previous studies showed that hybrid architectures usually work better for these
languages and can yield great results. This paper presents a hybrid lemmatizer
utilizing both a neural model, dictionaries and hand-crafted rules. We
introduce a hybrid architecture along with empirical results on a widely used
Hungarian dataset. The presented methods are published as three HuSpaCy models.Comment: published at the conference XIX. Magyar Sz\'am\'it\'og\'epes
Nyelv\'eszeti Konferencia (XIX. Hungarian Computational Linguistics
Conference
Hybrid lemmatization in HuSpaCy
Lemmatization is still not a trivial task for morphologically rich languages. Previous studies showed that hybrid architectures usually work better for these languages and can yield great results. This paper presents a hybrid lemmatizer utilizing both a neural model, dictionaries and hand-crafted rules. We introduce a hybrid architecture along with empirical results on a widely used Hungarian dataset. The presented methods are published as three HuSpaCy models