A Video Library System Using Scene Detection and Automatic Tagging

We present a novel video browsing and retrieval system for edited videos, in which videos are automatically decomposed into meaningful and storytelling parts (i.e. scenes) and tagged according to their transcript. The system relies on a Triplet Deep Neural Network which exploits multimodal features, and has been implemented as a set of extensions to the eXo Platform Enterprise Content Management System (ECMS). This set of extensions enable the interactive visualization of a video, its automatic and semi-automatic annotation, as well as a keyword-based search inside the video collection. The platform also allows a natural integration with third-party add-ons, so that automatic annotations can be exploited outside the proposed platform

From Convergence to Connectivism: Teaching Journalism 2.0

Media Convergence leads to fundamental changes in the journalistic field. This is a challenge not only for the news industry but also for teaching journalism at universities. Improved skills and competences are needed in multi-modal editorial planning and cross-channel develop-ment of news stories. This paper describes the main drivers of change in the news industry, discusses consequences for journalism and teaching, and offers an innovative didactical ap-proach, which combines an interdisciplinary perspective with the concept of connectivism and cognitive apprenticeship. The consequences of convergence that affect journalism educa-tion are inspected under three sub-topics: the changes in the curricula, the changes in the courses syllabi, and the changes in the technological infrastructures of the academic institu-tions. It is pointed out, that recognizing connections and patterns to develop novel ideas and concepts is the core skill for individuals today. The paper shows, how that approach could be realized in an international 10-day-intensive program

A Multimodal Approach to Sarcasm Detection on Social Media

In recent times, a major share of human communication takes place online. The main reason being the ease of communication on social networking sites (SNSs). Due to the variety and large number of users, SNSs have drawn the attention of the computer science (CS) community, particularly the affective computing (also known as emotional AI), information retrieval, natural language processing, and data mining groups. Researchers are trying to make computers understand the nuances of human communication including sentiment and sarcasm. Emotion or sentiment detection requires more insights about the communication than it does for factual information retrieval. Sarcasm detection is particularly more difficult than categorizing sentiment. Because, in sarcasm, the intended meaning of the expression by the user is opposite to the literal meaning. Because of its complex nature, it is often difficult even for human to detect sarcasm without proper context. However, people on social media succeed in detecting sarcasm despite interacting with strangers across the world. That motivates us to investigate the human process of detecting sarcasm on social media where abundant context information is often unavailable and the group of users communicating with each other are rarely well-acquainted. We have conducted a qualitative study to examine the patterns of users conveying sarcasm on social media. Whereas most sarcasm detection systems deal in word-by-word basis to accomplish their goal, we focused on the holistic sentiment conveyed by the post. We argue that utilization of word-level information will limit the systems performance to the domain of the dataset used to train the system and might not perform well for non-English language. As an endeavor to make our system less dependent on text data, we proposed a multimodal approach for sarcasm detection. We showed the applicability of images and reaction emoticons as other sources of hints about the sentiment of the post. Our research showed the superior results from a multimodal approach when compared to a unimodal approach. Multimodal sarcasm detection systems, as the one presented in this research, with the inclusion of more modes or sources of data might lead to a better sarcasm detection model

Multiliteracies meaning-making: How four boys’ video gaming experiences influence their cultural knowledge—Two ethnographic cases

Scholars have acknowledged the potential contribution of video gaming to complex forms of learning, identifying links between gaming and engagement, experiential learning spaces, problem-solving, strategies, transliteracy reflectivity, critical literacy, and metacognitive thinking. Despite this movement toward the inclusion of video gaming in literacy teaching, concerns about certain risks raised by scholars have slowed the adoption of using video games to foster learning. Using a multiliteracies lens, this multi-case study examined the experiences of four boys engaged with video gaming in two different contexts: a community centre and an after-school video club. By drawing on Feminist Post-Structural Theory, Vygotskian, and video gaming technology, I have gained an understanding of the nature of boys’ behavior and learning in social settings while they engage in video game play. Studying the ways in which boys make meanings through multimodal ways of learning can offer insights into strategies that can potentially reinvent traditional literacy pedagogical boundaries and establish new ways and practices for building knowledge. These ethnographic cases, along with their naturalistic aspects, strengthened the authenticity of the social-contextual-cultural experiences of the four, adolescent-aged boys and allowed an understanding of their everyday experiences. Interpretations of the cultural meanings made by each of the boys, based on their individual unique experiences engaging with video games, can provide readers with insights into how to approach adolescent aged boys’ literacy development. This study describes how these four boys developed their multimodal ways of learning by engaging with visual perspectives of video games. My methodological approach documented what boys are saying, as much as possible, which is currently understudied in the literature surrounding boys and their video gaming practices. There were a number of findings emanating from this study, including the following: (i) boys use their video gaming practices for meaning-making and collaborative efforts in order to gain an understanding of several knowledge processes (such as decision-making, predicting, analyzing, strategizing, etc.), (ii) boys extend and apply their cultural knowledge as creative innovators, producing and publishing YouTube instructional videos for video game players and designing video games for a history project, (iii) boys demonstrate peer mentoring through storytelling, face-to-face interactions or in their online community of practice, (iv) boys make meanings using metacognitive literacy skills in a variety of ways, and (v) boys focus on cultural preservation and narrative storytelling. While acknowledging concerns related to video gaming, such as negative identity construction, violence, distraction, and time commitment for integration, this study seeks to contribute to the scholarly discussion about the use of video games in classrooms by explicitly considering the ways in which gaming may support boys’ meaning-making and cultural knowledge. Keywords Available designs, boys, community of practice, cultural meaning-making systems, literacy, multiliteracies, multimodal meanings, video gamin

Network Narrative: Prose Narrative Fiction and Participatory Cultural Production in Digital Information and Communication Networks

In this study of prose narrative created explicitly for participatory network communications environments I argue that network narratives constitute an important, born-networked form of literary and cultural expression. In the first half of the study I situate network narratives within a rich, dynamic process of reciprocity and codependence between the technological, material and formal properties of communication media on the one hand, and the uses of these media in cultural practices and forms of expression on the other. I point out how the medial and cultural flows that characterize contemporary network culture promote a codependent relation between narrative and information. This relation supports literary cultural expressions that invoke everyday communication practices increasingly shaped by mobile, networked computing devices. In the second half of this study, I extend theoretical work in the field of electronic literature and digital media to propose a set of four characteristics through which network narratives may be understood as distinct modes of networked, literary cultural expression. Network narratives, I suggest, are multimodal, distributed, participatory, and emergent. These attributes are present in distinct ways, within distinct topological layers of the narratives: in the story, discourse, and character networks of the narrative structure; in the formal and navigational structures; and in the participatory circuits of production, circulation and consumption. Attending to these topological layers and their interrelationships by using concepts derived from graph theory and network analysis offers a methodology that links the particular, closely read attributes and content of network narratives to a more distant understanding of changing patterns in broader, networked cultural production. Finally, I offer readings of five examples of network narratives. These include Kate Pullinger and Chris Joseph’s Flight Paths, Penguin Books and De Montfort University’s collaborative project A Million Penguins, the Apple iOS application The Silent History, Tim Burton’s collaboration with TIFF, BurtonStory, and a project by NFB Interactive, Out My Window. Each of these works incorporates user participation into its production circuits using different strategies, each with different implications for narrative and navigational structures. I conclude by describing these distinct strategies as additive participation – participation that becomes embedded within the work itself – and delineating different approaches that are employed independently or in combination by the authors and producers

Critical Literacy and Identities in World Language Education: Telling Reflective Stories of Digital Storytelling

This paper explores a digital storytelling project in world language education conducted as critical literacy (Janks, 1991; 2000). Digital storytelling here entails the analysis and production of short videos (called digital stories) that tell a storyteller\u27s personally significant experience by digitally combining a voice-over, images, and music. In other words, this study theorizes digital storytelling in a world language as pedagogical opportunities to examine the effects of language in use and to transform their relations to language through the production of and reflection on identity text (Cummins et al., 2005). Two areas of guiding questions were: the design process and the range of identity options that the storytelling and its reflection makes possible. This study took a narrative case study of seven students who participated in a digital storytelling project in a low intermediate college Japanese course in the U.S. where the author was the instructor. The two primary data sources are the participants\u27 digital stories and reflective narratives. Additional data sources include instructional materials and the participants\u27 in-process drafts and reflective writings. Two methods of data analysis were used: inductive content analysis for the recurrent themes and discourse positioning analysis for the interactional achievements (Davies & Harré, 1990; Wortham, 2001). The analysis indicated the purpose-driven use of various resources in different stages of digital storytelling, such as different aspects of the sample stories that matched their personal investment in storytelling. Academic literacy in other languages, meta-linguistic awareness, and media awareness also affected their design processes. They reported the use of iconic and symbolic images with different intents and effects respectively. These multimodal resources afforded multi-sensory engagement. The analysis also indicated the participants\u27 positionings of other characters to create identity positions for their old selves in the digital stories. The participants’ shifting relations to the digital storytelling project were observed in their interview tellings as a series of assigned tasks and an owned project revealing the sense of agency. However, participants had different range of positions, suggesting the need to further consider the multiple layers of discourses that participants engaged. This adds to our understanding of shifting identity affordances

Sporthesia: Augmenting Sports Videos Using Natural Language

Augmented sports videos, which combine visualizations and video effects to present data in actual scenes, can communicate insights engagingly and thus have been increasingly popular for sports enthusiasts around the world. Yet, creating augmented sports videos remains a challenging task, requiring considerable time and video editing skills. On the other hand, sports insights are often communicated using natural language, such as in commentaries, oral presentations, and articles, but usually lack visual cues. Thus, this work aims to facilitate the creation of augmented sports videos by enabling analysts to directly create visualizations embedded in videos using insights expressed in natural language. To achieve this goal, we propose a three-step approach - 1) detecting visualizable entities in the text, 2) mapping these entities into visualizations, and 3) scheduling these visualizations to play with the video - and analyzed 155 sports video clips and the accompanying commentaries for accomplishing these steps. Informed by our analysis, we have designed and implemented Sporthesia, a proof-of-concept system that takes racket-based sports videos and textual commentaries as the input and outputs augmented videos. We demonstrate Sporthesia's applicability in two exemplar scenarios, i.e., authoring augmented sports videos using text and augmenting historical sports videos based on auditory comments. A technical evaluation shows that Sporthesia achieves high accuracy (F1-score of 0.9) in detecting visualizable entities in the text. An expert evaluation with eight sports analysts suggests high utility, effectiveness, and satisfaction with our language-driven authoring method and provides insights for future improvement and opportunities.Comment: 10 pages, IEEE VIS conferenc