AI generated content (AIGC) presents considerable challenge to educators
around the world. Instructors need to be able to detect such text generated by
large language models, either with the naked eye or with the help of some
tools. There is also growing need to understand the lexical, syntactic and
stylistic features of AIGC. To address these challenges in English language
teaching, we first present ArguGPT, a balanced corpus of 4,038 argumentative
essays generated by 7 GPT models in response to essay prompts from three
sources: (1) in-class or homework exercises, (2) TOEFL and (3) GRE writing
tasks. Machine-generated texts are paired with roughly equal number of
human-written essays with three score levels matched in essay prompts. We then
hire English instructors to distinguish machine essays from human ones. Results
show that when first exposed to machine-generated essays, the instructors only
have an accuracy of 61% in detecting them. But the number rises to 67% after
one round of minimal self-training. Next, we perform linguistic analyses of
these essays, which show that machines produce sentences with more complex
syntactic structures while human essays tend to be lexically more complex.
Finally, we test existing AIGC detectors and build our own detectors using SVMs
and RoBERTa. Results suggest that a RoBERTa fine-tuned with the training set of
ArguGPT achieves above 90% accuracy in both essay- and sentence-level
classification. To the best of our knowledge, this is the first comprehensive
analysis of argumentative essays produced by generative large language models.
Machine-authored essays in ArguGPT and our models will be made publicly
available at https://github.com/huhailinguist/ArguGP