Humans vs. Machines: Comparing Coding of Interviewer Question-Asking Behaviors Using Recurrent Neural Networks to Human Coders

Standardized survey interviewing techniques are intended to reduce interviewers’ effects on survey data. A common method to assess whether or not interviewers read survey questions exactly as worded is behavior coding. However, manually behavior coding an entire survey is expensive and time-consuming. Machine learning techniques such as Recurrent Neural Networks (RNNs) may offer a way to partially automate this process, saving time and money. RNNs learn to categorize sequential data (e.g., conversational speech) based on patterns learned from previously categorized examples. Yet the feasibility of an automated RNN-based behavior coding approach and how accurately this approach codes behaviors compared to human behavior coders are unknown. In this poster, we compare coding of interviewer question-asking behaviors by human undergraduate coders to the coding of transcripts performed by RNNs. Humans transcribe and manually behavior code each interview in the Work and Leisure Today II telephone survey (AAPOR RR3=7.8%) at the conversational turn level (n=47,900 question-asking turns) to identify when interviewers asked questions (1) exactly as worded, (2) with minor changes (i.e., changes not affecting question meaning), or (3) with major changes (i.e., changes affecting question meaning). With a random subset of interview transcripts as learning examples, we train RNNs to classify interviewer question-asking behaviors into these same categories. A random 10% subsample of transcripts (n=94) were also coded by master coders to evaluate inter-coder reliability. We compare the reliability of coding (versus the master coders) by the human coders with the reliability of the coding (versus the master coders) by the RNNs. Preliminary results indicate that the human coders and the RNNs have equal reliability (p\u3e.05) for questions with a large proportion of major and minor changes. We conclude with implications for behavior coding telephone interview surveys using machine learning in general, and RNNs in particular

Scalable Decision-Theoretic Planning in Open and Typed Multiagent Systems

In open agent systems, the set of agents that are cooperating or competing changes over time and in ways that are nontrivial to predict. For example, if collaborative robots were tasked with fighting wildfires, they may run out of suppressants and be temporarily unavailable to assist their peers. We consider the problem of planning in these contexts with the additional challenges that the agents are unable to communicate with each other and that there are many of them. Because an agent's optimal action depends on the actions of others, each agent must not only predict the actions of its peers, but, before that, reason whether they are even present to perform an action. Addressing openness thus requires agents to model each other's presence, which becomes computationally intractable with high numbers of agents. We present a novel, principled, and scalable method in this context that enables an agent to reason about others' presence in its shared environment and their actions. Our method extrapolates models of a few peers to the overall behavior of the many-agent system, and combines it with a generalization of Monte Carlo tree search to perform individual agent reasoning in many-agent open environments. Theoretical analyses establish the number of agents to model in order to achieve acceptable worst case bounds on extrapolation error, as well as regret bounds on the agent's utility from modeling only some neighbors. Simulations of multiagent wildfire suppression problems demonstrate our approach's efficacy compared with alternative baselines.Comment: Pre-print with appendices for AAAI 202

ConferenceXP-Powered I-MINDS: A Multiagent System for Intelligently Supporting Online Collaboration

In this paper, we describe a multiagent system designed for intelligently supporting online human collaboration, built on top of the ConferenceXP platform developed by Microsoft Research. Many current collaborative systems are passive in nature and do not provide active, intelligent support to users. A multiagent system can be used to track user behavior, perform automated tasks for humans, find optimal collaborative groups, and create and present helpful processed information based on data mining without detracting from the rest of the collaborative experience. Our ConferenceXP-powered I-MINDS application currently offers five different components for enhancing collaboration and sup-porting moderator decision making by giving each user a personal agent that works with other agents to further sup-port the entire system. These capabilities include two modes for group-based discussions, one for question/answer pairs between users and moderators, a search engine for retrieving tracked data, and a centralized classroom/team management system for quickly accessing user performance. CXP+I-MINDS has been successfully deployed to support an interactive business course where its intelligent activities assisted the professor in teaching, and we are working on delivering it to support a wireless classroom

What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming

Metacognition and self-regulation are popular areas of interest in programming education, and they have been extensively researched outside of computing. While computing education researchers should draw upon this prior work, programming education is unique enough that we should explore the extent to which prior work applies to our context. The goal of this systematic review is to support research on metacognition and self-regulation in programming education by synthesizing relevant theories, measurements, and prior work on these topics. By reviewing papers that mention metacognition or self-regulation in the context of programming, we aim to provide a benchmark of our current progress towards understanding these topics and recommendations for future research. In our results, we discuss eight common theories that are widely used outside of computing education research, half of which are commonly used in computing education research. We also highlight 11 theories on related constructs (e.g., self-efficacy) that have been used successfully to understand programming education. Towards measuring metacognition and self-regulation in learners, we discuss seven instruments and protocols that have been used and highlight their strengths and weaknesses. To benchmark the current state of research, we examined papers that primarily studied metacognition and self-regulation in programming education and synthesize the reported interventions used and results from that research. While the primary intended contribution of this paper is to support research, readers will also learn about developing and supporting metacognition and self-regulation of students in programming courses