Text readability assessment has gained significant attention from researchers
in various domains. However, the lack of exploration into corpus compatibility
poses a challenge as different research groups utilize different corpora. In
this study, we propose a novel evaluation framework, Cross-corpus text
Readability Compatibility Assessment (CRCA), to address this issue. The
framework encompasses three key components: (1) Corpus: CEFR, CLEC, CLOTH, NES,
OSP, and RACE. Linguistic features, GloVe word vector representations, and
their fusion features were extracted. (2) Classification models: Machine
learning methods (XGBoost, SVM) and deep learning methods (BiLSTM,
Attention-BiLSTM) were employed. (3) Compatibility metrics: RJSD, RRNSS, and
NDCG metrics. Our findings revealed: (1) Validated corpus compatibility, with
OSP standing out as significantly different from other datasets. (2) An
adaptation effect among corpora, feature representations, and classification
methods. (3) Consistent outcomes across the three metrics, validating the
robustness of the compatibility assessment framework. The outcomes of this
study offer valuable insights into corpus selection, feature representation,
and classification methods, and it can also serve as a beginning effort for
cross-corpus transfer learning.Comment: 14 pages,17 figure