Coding Text Answers to Open-ended Questions: Human Coders and Statistical Learning Algorithms Make Similar Mistakes

Text answers to open-ended questions are often manually coded into one of several predefined categories or classes. More recently, researchers have begun to employ statistical models to automatically classify such text responses. It is unclear whether such automated coders and human coders find the same type of observations difficult to code or whether humans and models might be able to compensate for each other’s weaknesses. We analyze correlations between estimated error probabilities of human and automated coders and find: 1) Statistical models have higher error rates than human coders 2) Automated coders (models) and human coders tend to make similar coding mistakes. Specifically, the correlation between the estimated coding error of a statistical model and that of a human is comparable to that of two humans. 3) Two very different statistical models give highly correlated estimated coding errors. Therefore, a) the choice of statistical model does not matter, and b) having a second automated coder would be redundant

On the Automatic Coding of Text Answers to Open-ended Questions in Surveys

Open-ended questions allow participants to answer survey questions without any constraint. Responses to open-ended questions, however, are more difficult to analyze quantitatively than close-ended questions. In this thesis, I focus on analyzing text responses to open-ended questions in surveys. The thesis includes three parts: double coding of open-ended questions, predictions of potential coding errors in manual coding, and comparison between manual coding and automatic coding. Double coding refers to two coders coding the same observations independently. It is often used to assess coders' reliability. I investigate the usage of double coding to improve the performance of automatic coding. I find that, when the budget for manual coding is fixed, double coding which involves a more experienced expert coder results in a smaller but cleaner training set than single coding, and improves the prediction of statistical learning models when the coding error rate of coders exceeds a threshold. When data have already been double coded, double coding always outperforms single coding. In many research projects, only a subset of data can be double coded due to limited funding. My idea is that researchers can make use of the double-coded subset to improve the coding quality of the remaining single-coded observations. Therefore, I propose a model-assisted coding process that predicts the risk of coding errors. High risk text answers are then double-coded. The proposed coding process reduces coding error while keeping the ability to assess inter-coder reliability. Manual coding and automatic coding are two main approaches to code responses to open-ended questions, yet the similarity or difference in terms of coding error has not been well studied. I compare the coding error of human coders and automated coders. I find, despite a different error rate, human coders and automated coders make similar mistakes

Evaluating the Effectiveness of the Computer-Based Education Platform, Pharmacy5in5, on Pharmacists’ Knowledge of Anticholinergic Toxicity Using a Randomized Controlled Trial

Background: Computer-based education has been widely implemented in healthcare professional development education. However, there has been little examination of the potential for computer-based education to enhance pharmacists’ knowledge. This study aims to assess the effectiveness of computer-based education on improving pharmacists’ knowledge compared to printed education material. Methods: This study was a web-based randomized controlled trial. Participants were randomly allocated to either an intervention group where they had access to the computer-based education module on Pharmacy5in5.ca or to a control group where they had access to printed educational material. Knowledge gain was assessed using a pre- and post-knowledge test. Results: A total of 120 pharmacists were recruited and 101 completed the post-knowledge test (50/60 in the intervention group; 51/60 in the control group). Both groups showed a significant increase in knowledge gain (intervention group: pre-test mean score 19.35 ± 3.56, post-test mean score 22.42 ± 3.812, p value p value p value = 0.333). Conclusions: In this study, a computer-based education module enhanced pharmacists’ knowledge to a similar degree to printed education material. Efforts should be made to provide computer-based education as an option to support pharmacists’ professional development