The COMPS project employs computer chat for students working in small groups solving classroom problems. This summer’s project aims to build computer classifiers that could effectively “look over the shoulders” of the students while working, to approximately recognize whether the students are engaging in productive discussion. Research questions are: can we write machine classifiers that can recognize reasoning, agreement, and disagreement in student discussions? Can we achieve this using only a common English vocabulary?
Several thousand lines of COMPS transcripts were manually annotated. A topic modelling program was used to determine 10 main topics which appeared in the transcripts and the words in those topics. A Linear Classifier and a Support Vector Machine Classifier used the topic model to predict the annotation of each line of dialogue.
To address the common English vocabulary research question, an intersection of many transcripts from different sources was combined with Google word lists and modified to accommodate text-chat conventions.
In the normal vocabulary, we found f1 scores of 0.7 and above for reasoning. Using only common vocabulary, the scores were slightly lower.
The next step is to train our topic model on a combination of transcripts and apply it to other transcripts from different student discussions
