Captions versus transcripts for online video content

ABSTRACT Captions provide deaf and hard of hearing (DHH) users ac cess to the audio component of web videos and television. While hearing consumers can watch and listen simultane ously, the transformation of audio to text requires deaf view ers to watch two simultaneous visual streams: the video and the textual representation of the audio. This can be a prob lem when the video has a lot of text or the content is dense, e.g., in Massively Open Online Courses. We explore the ef fect of providing caption history on users' ability to follow captions and be more engaged. We compare traditional onvideo captions that display a few words at a time to off-video transcripts that can display many more words at once, and investigate the trade off of requiring more effort to switch be tween the transcript and visuals versus being able to review more content history. We find significant difference in users' preferences for viewing video with on-screen captions over off-screen transcripts in terms of readability, but no signifi cant difference in users' preferences in following and under standing the video and narration content. We attribute this to viewers' perceived understanding significantly improving when using transcripts over captions, even if they were less easy to track. We then discuss the implications of these re sults for on-line education, and conclude with an overview of potential methods for combining the benefits of both onscreen captions and transcripts

Creating Accessible Videos: Captions and Transcripts

The rapid shift to online teaching due to the coronavirus disease of 2019 (COVID-19) exponentially increased the extent to which faculty use videoconferencing/virtual classroom tools such as Zoom, Google Meet, and Microsoft Teams. It also exposed the challenge of ensuring that all students could access all video content. Faculty may need to implement captions or transcripts in videos to not only address certain student accommodations but also fulfill an institution’s legal responsibilities, under accessibility laws, to conform to the W3C’s Web Content Accessibility Guidelines (WCAG). Faculty can create captions and transcripts manually, use a third-party service to create them, or generate them with some recording tools. As faculty have quickly needed to adapt course materials from in-person to online instruction, they need to learn best practices for technology use in order to help all students succeed. In this paper, I share some personal experiences and lessons I have learned about creating and using accessible video content for online courses

What's Cookin'? Interpreting Cooking Videos using Text, Speech and Vision

We present a novel method for aligning a sequence of instructions to a video of someone carrying out a task. In particular, we focus on the cooking domain, where the instructions correspond to the recipe. Our technique relies on an HMM to align the recipe steps to the (automatically generated) speech transcript. We then refine this alignment using a state-of-the-art visual food detector, based on a deep convolutional neural network. We show that our technique outperforms simpler techniques based on keyword spotting. It also enables interesting applications, such as automatically illustrating recipes with keyframes, and searching within a video for events of interest.Comment: To appear in NAACL 201

Creating Oral History Collections in Digital Commons

Furman University has seen a boom in the creation of oral histories over the past several years. This presentation will look at two of their oral history collections and discuss how they are built and presented within Digital Commons. The Furman University Oral Histories project contains historical histories digitized nearly a decade ago with limited metadata and poor audio/video quality. The Oral Histories of Columbian Textile Workers in Greenville, South Carolina is a new collection of recent histories which contain videos, Spanish/English translations, transcripts, and robust metadata. The presenters will discuss the benefits and challenges of each collection as they fit into the Digital Commons framework. Their presentation will address: using the book gallery structure to build the digital collections, options for hosting streaming video, working with audio-only files, captioning, transcriptions, metadata customization, and design. At the end of the presentation, attendees should have a good understanding of one method to successfully build and manage an oral history collection within Digital Commons

Online comments as input enhancement

[EN] Chapelle (2003) proposed three general types of input enhancement that help L2 learners “acquire features of the linguistic input that they are exposed to during the course reading or listening for meaning” (p. 40): input salience, input modification, and input elaboration. In 2010, Cárdenas-Claros and Gruba argued that Chapelle’s different types of input enhancement “can be and have been operationalized through help options” primarily utilized in the teaching of reading, listening, writing, grammar, and vocabulary such as glossed words, video/audio control features, captions, subtitles, and grammar explanations (p. 79). As understood from Cárdenas-Claros and Gruba’s classification of help options, input enhancement can only be accomplished through one process: salience, modification, or elaboration. In this article, we argue that YouTube comments have the potential to be (1) a help option that facilitate both listening comprehension of the videos and vocabulary learning and that (2) input enhancement accomplished by comments can be achieved by a combination of different types of input enhancement. Put another way, the aural input of a YouTube video can be salient, modified, and elaborated, thanks to the various types of comments YouTube videos often receive.Aldukhayel, D. (2021). Online comments as input enhancement. The EuroCALL Review. 29(2):44-54. https://doi.org/10.4995/eurocall.2021.14212OJS4454292Barton, D., and Lee, C. (2013). Language online: Investigating digital texts and practices. Routledge.Benson, P. (2015). Commenting to learn: Evidence of language and intercultural learning in comments on YouTube videos. Language Learning & Technology, 19(3), 88-105. https://doi.org/10125/44435Bou-Franch, P., Lorenzo-Dus, N., and Blitvich, P. (2012). Social interaction in YouTube text-based polylogues: A study of coherence. Journal of Computer-Mediated Communication, 17(4), 501-521. https://doi.org/10.1111/j.1083-6101.2012.01579.xCárdenas -Claros, M. S. (2005). Field dependence/field independence: How do students perform in CALL-based listening activities? Iowa State University, unpublished MA thesis. https://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=8941&context=rtdCárdenas-Claros, M. S., and Gruba, P. A. (2010). Help options in CALL: A systematic review. CALICO Journal, 27(1), 69-90. http://www.jstor.org/stable/calicojournal.27.1.69 https://doi.org/10.11139/cj.27.1.69-90Cárdenas-Claros, M. S., and Gruba, P. A. (2013). Decoding the "CoDe": A framework for conceptualizing and designing help options in computer-based second language listening. ReCALL, 25(2), 250-271. https://doi.org/10.1017/S0958344013000049Cárdenas-Claros, M. S., and Gruba, P. A. (2014). Listeners' interactions with help options in CALL. Computer Assisted Language Learning, 27(3), 228-245. https://doi.org/10.1080/09588221.2012.724425Chapelle, C. A. (2003). English language learning and technology: Lectures on applied linguistics in the age of information and communication technology. John Benjamins Publishing. https://doi.org/10.1075/lllt.7Chen, C. W. Y. (2020). Analyzing online comments: A language-awareness approach to cultivating digital literacies. Computer Assisted Language Learning, 33(4), 435-454. https://doi.org/10.1080/09588221.2019.1569068Cross, J. (2017). Help options for L2 listening in CALL: A research agenda. Language Teaching, 50(4), 544-560. https://doi.org/10.1017/S0261444817000209Dworman, G., and Rosenbaum, S. (2004). Helping users to use help: Improving interaction with help systems. In Proceedings of the ACM CHI '04: CHI '04 Extended Abstracts on Human Factors in Computing Systems (pp. 1717-1718). New York: ACM Press. https://doi.org/10.1145/985921.986198Ellis, R. (1999). Learning a second language through interaction. John Benjamins Publishing. https://doi.org/10.1075/sibil.17Ellison, M. (2007). Embedded user assistance: The future for software help? Interactions, 14(1), 30-31. https://doi.org/10.1145/1189976.1189997Grgurovic', M., and Hegelheimer, V. (2007). Help options and multimedia listening: Student's use of subtitles and transcripts. Language Learning & Technology, 11(1), 45-66. https://doi.org/10125/44088Hegelheimer, V. (1998). Effects of textual glosses and sentence level audio glosses on online reading comprehension and vocabulary recall. University of Illinois, unpublished PhD.Hegelheimer, V., and Tower, D. (2004). Using CALL in the classroom: Analyzing student interactions in an authentic classroom. System, 32, 185-205. https://doi.org/10.1016/j.system.2003.11.007Hsu, J. F. J. (1994). Computer assisted language learning (CALL): The effect of ESL students' use of interactional modifications on listening comprehension. Iowa State University unpublished PhD.Jones, L. (2007). Effects of collaboration and multimedia annotations on vocabulary learning and listening comprehension. CALICO Journal, 24(1), 33-58. https://doi.org/10.1558/cj.v24i1.33-58Jones, L., and J. Plass (2002). Supporting listening comprehension and vocabulary acquisition in French with multimedia annotations. The Modern Language Journal, 86(4), 546-561. http://www.jstor.org/stable/1192724 https://doi.org/10.1111/1540-4781.00160Jones, G. M., and Schieffelin, B. B. (2009). Talking text and talking back: "My BFF Jill" from boob tube to YouTube. Journal of Computer-Mediated Communication, 14(4), 1050-1079. https://doi.org/10.1111/j.1083-6101.2009.01481.xKelleher, C., and Paucsh, R. (2005). Stencil-based tutorials: Design and evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 541-550. https://doi.org/10.1145/1054972.1055047Khan, M. L. (2017). Social media engagement: What motivates user participation and consumption on YouTube? Computers in Human Behavior, 66, 236-247. https://doi.org/10.1016/j.chb.2016.09.024Larsen-Freeman, D., and Long, M. (1991). An introduction to second language acquisition research. Longman.Liou, H. C. (1997). Research of on-line help as learner strategies for multimedia CALL evaluation. CALICO Journal, 14, 81-96. https://doi.org/10.1558/cj.v14i2-4.81-96Madden, A., Ruthven, I., and McMenemy, D. (2013). A classification scheme for content analyses of YouTube video comments. Journal of Documentation, 69 (5), 693-714. https://doi.org/10.1108/JD-06-2012-0078Oh, S. Y. (2001). Two types of input modification and EFL reading comprehension: Simplification versus elaboration. TESOL Quarterly, 35(1), 69-96. https://doi.org/10.2307/3587860Prensky, M. (2001). Digital natives, digital immigrants. On the Horizon, 9(5), 1-6. https://doi.org/10.1108/10748120110424816Plass, J. L., Chun, D. M., Mayer, R. E., and Leutner, D. (1998). Supporting visual and verbal learning preferences in a second-language multimedia learning environment. Journal of Educational Psychology, 90(1), 25-36. https://psycnet.apa.org/doi/10.1037/0022-0663.90.1.25 https://doi.org/10.1037/0022-0663.90.1.25Pujolà, J. T. (2002). CALLing for help: Researching language learning strategies using help facilities in a web-based multimedia program. ReCALL, 14, 235-262. https://doi.org/10.1017/S0958344002000423Rivens Mompean, A., and Guichon, N. (2009). Assessing the use of aids for computer-mediated tasks: taking notes while listening. JALTCALL Journal, 5, 45-60. https://doi.org/10.29140/jaltcall.v5n2.79Robinson, P. (1995). Attention, memory and the "noticing" hypothesis. Language Learning, 45, 285-331. https://doi.org/10.1111/j.1467-1770.1995.tb00441.xSchmidt, R. (1990). The role of consciousness in second language learning. Applied Linguistics, 11(2), 129-158. https://doi.org/10.1093/applin/11.2.129Schultes, P., Dorner, V., and Lehner, F. (2013). Leave a Comment! An In-Depth Analysis of User Comments on YouTube. Wirtschaftsinformatik, 42, 659-673.Sharwood Smith, M. (1993). Input enhancement in instructed SLA: Theoretical bases. Studies in Second Language Acquisition, 15, 165-179. https://doi.org/10.1017/S0272263100011943Skehan, P. (1998). A cognitive approach to language learning. Oxford University Press. https://doi.org/10.1177/003368829802900209Syodorenko, T. (2010). Modality of input and vocabulary acquisition. Language Learning & Technology, 14(2), 50-73. https://doi.org/10125/44214Terantino, J. (2011). YouTube for foreign languages: You have to see this video. Language Learning and Technology, 15(1), 10-16. https://doi.org/10125/44231Tolson, A. (2010). A new authenticity? Communicative practices on YouTube. Critical Discourse Studies, 7(4), 277-289. https://doi.org/10.1080/17405904.2010.511834Watanabe, Y. (1997). Input, intake, and retention: Effects of increased processing on incidental learning of foreign language vocabulary. Studies in Second Language Acquisition, 19(3), 287-307. https://doi.org/10.1017/S027226319700301XWillis, M. (2006). Building effective help systems: Modeling human help-seeking behavior. In: Proceedings of the 20th conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: design: Activities, artefacts and environments (pp. 433-436). New York: ACM. https://doi.org/10.1145/1228175.1228266Winke, P., Gass, S., and Syodorenko, T. (2010). The effects of captioning videos used for foreign language listening activities. Language Learning & Technology, 14(1), 65-86. https://doi.org/10125/44203Yano, Y., Long, M. H., and Ross, S. (1994). The effects of simplified and elaborated texts on foreign language reading comprehension. Language Learning, 25, 297-308. https://doi.org/10.1111/j.1467-1770.1994.tb01100.xYouTube Official Blog (2018). Faster removals and tackling comments - an update on what we're doing to enforce YouTube's Community Guidelines. https://youtube.googleblog.com/2018/12/faster-removals-and-tackling-comments.htm

Multimedia information technology and the annotation of video

The state of the art in multimedia information technology has not progressed to the point where a single solution is available to meet all reasonable needs of documentalists and users of video archives. In general, we do not have an optimistic view of the usability of new technology in this domain, but digitization and digital power can be expected to cause a small revolution in the area of video archiving. The volume of data leads to two views of the future: on the pessimistic side, overload of data will cause lack of annotation capacity, and on the optimistic side, there will be enough data from which to learn selected concepts that can be deployed to support automatic annotation. At the threshold of this interesting era, we make an attempt to describe the state of the art in technology. We sample the progress in text, sound, and image processing, as well as in machine learning

Comparison of the impacts of different multimodalities on incidental L2 vocabulary learning

Multimodality of input in incidental L2 vocabulary learning has recently been a topic of interest among language acquisition researchers, yet the results have been somewhat contradictory. This study seeks to compare the impacts of two different multimodalities on incidental L2 vocabulary learning, namely, reading-plus-watching (experimental group I) vs. reading-plus-listening (experimental group II), as compared to the reading only condition, which is included as a control measure. Experimental group I watched and read the transcriptions of four news texts with electronic glosses for the target words, while experimental group II read and listened to the same news texts again with electronic glosses for the same 20 target words. Next, the two experimental groups swapped roles with a new set of four news texts glossed for another group of 20 target words. The control group only read the same eight news texts without glosses. The results suggest that reading-plus-listening can be a more conducive multimodal presentation for incidental vocabulary learning as compared to reading-plus-watching. The results also challenge the validity of some principles of the Cognitive Theory of Multimedia Learning in incidental L2 vocabulary learning, while providing supporting evidence for some other principles

Accuracy of Sign Interpreting and Real-Time Captioning ofScience Videos for the Delivery of Instruction to Deaf StudentsAccuracy of Sign Interpreting and Real-Time Captioning ofScience Videos for the Delivery of Instruction to Deaf StudentsAccuracy of Sign Interpreting and Real-Time Captioning of Science Videos for the Delivery of Instruction to Deaf Students

The purpose of this study was to quantitatively examine the impact of third-party support service providers on the quality of science information available to deaf students in regular science classrooms. Three different videotapes that were developed by NASA for high school science classrooms were selected for the study, allowing for different concepts and vocabulary to be examined. The focus was on the accuracy of translation as measured by the number of key science words included in the transcripts (captions) or videos (interpreted).Data were collected via transcripts completed by CART (computer assisted real-time captionists) or through videos of sign language interpreters. All participants were required to listen to and translate these NASA educational videos with no prior experience with this information so as not to influence their delivery. CART personnel using captions were found to be significantly more accurate in the delivery of science words as compared to the sign language interpreters in this study