Integrating Dictionary and Web N-grams for Chinese Spell Checking

Abstract Chinese spell checking is an important component of many NLP applications, including word processors, search engines, and automatic essay rating. Nevertheless, compared to spell checkers for alphabetical languages (e.g., English or French), Chinese spell checkers are more difficult to develop because there are no word boundaries in the Chinese writing system and errors may be caused by various Chinese input methods. In this paper, we propose a novel method for detecting and correcting Chinese typographical errors. Our approach involves word segmentation, detection rules, and phrase-based machine translation. The error detection module detects errors by segmenting words and checking word and phrase frequency based on compiled and Web corpora. The phonological or morphological typographical errors found then are corrected by running a decoder based on the statistical machine translation model (SMT). The results show that the proposed system achieves significantly better accuracy in error detection and more satisfactory performance in error correction than the state-of-the-art systems

Text segmentation and error detection for Chinese spell checking.

Ng Mau Kit Michael.Thesis (M.Phil.)--Chinese University of Hong Kong, 1999.Includes bibliographical references (leaves 117-120).Abstract and appendix in English and Chinese.Abstract --- p.iAcknowledgments --- p.ivChapter 1 --- Introduction --- p.1Chapter 2 --- Background Knowledge and Basic Concepts --- p.7Chapter 2.1 --- Classification of Natural Languages --- p.7Chapter 2.2 --- Chinese Spell Checking --- p.9Chapter 2.3 --- Characteristics of Chinese --- p.12Chapter 2.3.1 --- Word Frequency and Statistical Information of Chinese Words --- p.12Chapter 2.3.2 --- Chinese Grammar --- p.15Chapter 2.3.2.1 --- Word Class --- p.15Chapter 2.3.2.2 --- Grammar Rules --- p.17Chapter 3 --- Problems with Chinese Spell Checking and Related Work --- p.18Chapter 3.1 --- Ambiguities --- p.19Chapter 3.2 --- Unknown Words --- p.20Chapter 3.3 --- Text Errors --- p.21Chapter 3.4 --- Combinatory Explosion --- p.23Chapter 3.5 --- Related Work --- p.26Chapter 4 --- The Chinese Spell Checking System --- p.33Chapter 4.1 --- Architecutre of the Chinese Spell Checking System (CSCS) --- p.35Chapter 4.2 --- The Segmenter and the Error Detector --- p.39Chapter 5 --- The Block-of-Combinations Segmentation Algorithm and Error Detection --- p.42Chapter 5.1 --- Single-character-word Function --- p.43Chapter 5.2 --- Segmentation Strategy --- p.46Chapter 5.3 --- Maximum Number of Combinations of the BOC --- p.51Chapter 5.4 --- A Case Study of the BOC --- p.54Chapter 5.5 --- Evaluation of the BOC --- p.59Chapter 5.5.1 --- Accuracy --- p.59Chapter 5.5.2 --- Speed --- p.61Chapter 5.5.3 --- Discussion --- p.62Chapter 5.6 --- Experiments on Error Detection for the BOC --- p.63Chapter 5.6.1 --- Experimental Results of the Error Detection for the BOC --- p.65Chapter 6 --- The Genetic Algorithm Segmentation Method --- p.69Chapter 6.1 --- Basic Concepts of Genetic Algorithm --- p.69Chapter 6.2 --- Genetic Algorithm Model --- p.73Chapter 6.2.1 --- Chromosome Representation --- p.75Chapter 6.2.2 --- The Flow of the GAS --- p.76Chapter 6.2.2.1 --- Crossover --- p.77Chapter 6.2.2.2 --- Replacement --- p.78Chapter 6.2.2.3 --- Mutation --- p.80Chapter 6.2.2.4 --- Termination Criteria --- p.80Chapter 6.2.3 --- Fitness Function --- p.81Chapter 6.2.3.1 --- Single-character-word Function --- p.82Chapter 6.2.3.2 --- Known-word Function and Unknown-word Function --- p.83Chapter 6.2.3.3 --- Grammar Rules Scoring Function --- p.83Chapter 6.3 --- Maximum Number of Combinations of the GAS --- p.86Chapter 6.4 --- Evaluation of the GAS --- p.86Chapter 6.5 --- Discussion --- p.88Chapter 7 --- The Improved-BOC Algorithm for Handling Unknown Words and Errors --- p.90Chapter 7.1 --- Segmentation Principle of the Improved-BOC Method --- p.91Chapter 7.2 --- Improvement of the Scoring Function --- p.93Chapter 7.2.1 --- The Choice of Grammar Rules --- p.93Chapter 7.2.2 --- Phrase-structure Style --- p.96Chapter 7.2.3 --- Computer Model of Grammar Rules for Handling Unknown Words --- p.98Chapter 7.3 --- Evaluation of Segmentation --- p.102Chapter 7.4 --- Error Detection --- p.104Chapter 7.4.1 --- Evaluation of Error Detection --- p.106Chapter 7.5 --- Discussion --- p.108Chapter 7.6 --- "Comparison between the MM, BOC, GA and Improved-BOC" --- p.109Chapter 8 --- Conclusion --- p.114Bibliography --- p.117Appendix A: Sample Result of the Genetic Algorithm Segmentation Method --- p.121Appendix B: Set of Grammar Rules --- p.12