Multimodal Learning For Classroom Activity Detection

Classroom activity detection (CAD) focuses on accurately classifying whether the teacher or student is speaking and recording both the length of individual utterances during a class. A CAD solution helps teachers get instant feedback on their pedagogical instructions. This greatly improves educators' teaching skills and hence leads to students' achievement. However, CAD is very challenging because (1) the CAD model needs to be generalized well enough for different teachers and students; (2) data from both vocal and language modalities has to be wisely fused so that they can be complementary; and (3) the solution shouldn't heavily rely on additional recording device. In this paper, we address the above challenges by using a novel attention based neural framework. Our framework not only extracts both speech and language information, but utilizes attention mechanism to capture long-term semantic dependence. Our framework is device-free and is able to take any classroom recording as input. The proposed CAD learning framework is evaluated in two real-world education applications. The experimental results demonstrate the benefits of our approach on learning attention based neural network from classroom data with different modalities, and show our approach is able to outperform state-of-the-art baselines in terms of various evaluation metrics.Comment: The 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020

A framework for realistic 3D tele-immersion

Meeting, socializing and conversing online with a group of people using teleconferencing systems is still quite differ- ent from the experience of meeting face to face. We are abruptly aware that we are online and that the people we are engaging with are not in close proximity. Analogous to how talking on the telephone does not replicate the experi- ence of talking in person. Several causes for these differences have been identified and we propose inspiring and innova- tive solutions to these hurdles in attempt to provide a more realistic, believable and engaging online conversational expe- rience. We present the distributed and scalable framework REVERIE that provides a balanced mix of these solutions. Applications build on top of the REVERIE framework will be able to provide interactive, immersive, photo-realistic ex- periences to a multitude of users that for them will feel much more similar to having face to face meetings than the expe- rience offered by conventional teleconferencing systems

The Practitioner’s Panacea for Measuring Learner-Centeredness?

The Decibel Analysis for Research in Teaching (DART; Owens et al., 2017), a sound-based metric of learner-centeredness, is highly accessible, requires no training, and can be conducted with minimal classroom observations; yet, DART has not been evaluated in comparison with other validated metrics or in consideration of potentially confounding classroom characteristics (e.g. enrollment, classroom size, number of doors). We analyzed recordings from 42 class sessions of an undergraduate biology course with DART, the Reformed Teaching Observation Protocol (RTOP), and nine classroom characteristics. We found that enrollment was the best single predictor of the DART output of learner-centeredness, percent Multiple Voice

The effect of teacher scaffolding and student comprehension monitoring on a multimedia/interactive videodisc science lesson for second graders

Imagery based computer instruction is predicted to have a major impact on educational curriculum in the next century. Yet research on the effectiveness of imagery technology for early elementary-age children is a relatively unexplored area. The purpose of this study was to examine age-appropriate uses of a multimedia/interactive videodisc (IVD) science lesson for second graders in two areas. First, the unique properties that these media offer as a stand-alone teaching tool were assessed. Second, the non-technological strategies of teacher scaffolding and comprehension monitoring as supplements to IVD programs were investigated. A learner controlled multimedia/IVD instructional program was specifically designed for this study. The learning objectives were to teach the scientific processes of classification and problem solving through observing, comparing, and contrasting two species of primates: apes and monkeys

Training Noise-Robust Spoken Phrase Detectors with Scarce and Private Data: An Application to Classroom Observation Videos

We explore how to automatically detect specific phrases in audio from noisy, multi-speaker videos using deep neural networks. Specifically, we focus on classroom observation videos that contain a few adult teachers and several small children (\u3c 5 years old). At any point in these videos, multiple people may be talking, shouting, crying, or singing simultaneously. Our goal is to recognize polite speech phrases such as Good job , Thank you , Please , and You\u27re welcome , as the occurrence of such speech is one of the behavioral markers used in classroom observation coding via the Classroom Assessment Scoring System (CLASS) protocol. Commercial speech recognition services such as Google Cloud Speech are impractical because of data privacy concerns. Therefore, we train and test our own custom models using a combination of publicly available classroom videos from YouTube, as well as a private dataset of real classroom observation videos collected by our colleagues at the University of Virginia. We also crowdsource an additional 1152 recordings of polite speech phrases to augment our training dataset. Our contributions are the following: (1) we design a crowdsourcing task for efficiently labeling speech events in classroom videos, (2) we develop a neural network-based architecture for speech recognition, robust to noise and overlapping speech, and (3) we explore methods to synthesize new and authentic audio data, both to increase the training set size and reduce the class imbalance. Finally, using our trained polite speech detector, (4) we investigate the relationship between polite speech and CLASS scores and enable teachers to visualize their use of polite language

Reading Fluency Instruction of Students with Cognitive Disabilities Using a Multiple Probe Methodology

Despite making up less than one percent of the student population, students with significant cognitive disabilities have the right to receive the best education possible. There is currently a paucity of research regarding effective reading instruction within a comprehensive approach, especially in the area of fluency. The purpose of this investigation was to determine if there was a functional relation between repeated reading and choral reading and the word correct per minute oral reading of six high school students with significant cognitive disabilities. Additionally, the extent to which fluency impacts reading comprehension was also examined. Five of six participants demonstrated an increase of words correct per minute from baseline to treatment. Non-parametric measures of effect indicate no effect as a whole and weak to medium effect for each participant. Four of six participants improved their mean reading comprehension score during treatment