2 research outputs found
Onto Word Segmentation of the Complete Tang Poems
We aim at segmenting words in the Complete Tang Poems (CTP). Although it is
possible to do some research about CTP without doing full-scale word
segmentation, we must move forward to word-level analysis of CTP for conducting
advanced research topics. In November 2018 when we submitted the manuscript for
DH 2019 (ADHO), we collected only 2433 poems that were segmented by trained
experts, and used the segmented poems to evaluate the segmenter that considered
domain knowledge of Chinese poetry. We trained pointwise mutual information
(PMI) between Chinese characters based on the CTP poems (excluding the 2433
poems, which were used exclusively only for testing) and the domain knowledge.
The segmenter relied on the PMI information to the recover 85.7% of words in
the test poems. We could segment a poem completely correct only 17.8% of the
time, however. When we presented our work at DH 2019, we have annotated more
than 20000 poems. With a much larger amount of data, we were able to apply
biLSTM models for this word segmentation task, and we segmented a poem
completely correct above 20% of the time. In contrast, human annotators
completely agreed on their annotations about 40% of the time.Comment: 5 pages, 2 tables, presented at the 2019 International Conference on
Digital Humanities (ADHO
Corpus of Chinese Dynastic Histories: Gender Analysis over Two Millennia
Chinese dynastic histories form a large continuous linguistic space of
approximately 2000 years, from the 3rd century BCE to the 18th century CE. The
histories are documented in Classical (Literary) Chinese in a corpus of over 20
million characters, suitable for the computational analysis of historical
lexicon and semantic change. However, there is no freely available open-source
corpus of these histories, making Classical Chinese low-resource. This project
introduces a new open-source corpus of twenty-four dynastic histories covered
by Creative Commons license. An original list of Classical Chinese
gender-specific terms was developed as a case study for analyzing the
historical linguistic use of male and female terms. The study demonstrates
considerable stability in the usage of these terms, with dominance of male
terms. Exploration of word meanings uses keyword analysis of focus corpora
created for genderspecific terms. This method yields meaningful semantic
representations that can be used for future studies of diachronic semantics.Comment: 12th Conference on Language Resources and Evaluation (LREC 2020), 9
pages, 7 table