Langutils: A natural language toolkit for common lisp

Abstract

In recent years, Natural Language Processing (NLP) has emerged as an important capability in many applications and areas of research. Natural language can be both the domain of application and an important component in the human-computer interface. This paper describes the design and implementation of "langutils, ” a highperformance natural language toolkit for Common Lisp. We introduce the techniques of real-world NLP and explore tradeoffs in the representation and implementation of tokenization, part-of-speech tagging, and parsing. The paper concludes with a discussion of the use of the toolkit in two natural language applications