Narrative music: towards an understanding of musical narrative functions in multimedia

As the computer screen is replacing the book as the dominant medium for communication (Kress, 2003), questions about how meaning is constituted by the multimodal interaction of different media (including music) is becoming increasingly important in contemporary research of pedagogy, sociology and media studies. The overall aim with this licentiate thesis is to explore musical narrative functions as they appear in multimedia such as film and computer games. The thesis is based on three publications. Publication 1 proposes a classification of musical narrative functions, with 6 narrative classes(the Emotive, Informative, Descriptive, Guiding, Temporal and Rhetorical classes) and 11 categories. The relational interplay of music with contextual factors is emphasized. Publication 2 describes the design of a software tool, REMUPP (Relations Between Musical Parameters and Perceived Properties), to be used for experimental studies of musical expression. REMUPP is used for real time alteration of musical expression, by the manipulation of musical parameters such as tempo, harmony, rhythm, articulation, etc. Publication 3 describes a quasi-experiment using REMUPP, where a group of young participants (12-13 years old) were given the task of adapting musical expression – by manipulating 7 parameters – to make it fit 3 visual scenes shown on a computer screen. They also answered a questionnaire asking about their musical backgrounds and habits of listening to music, watching movies and playing computer games. Numerical data from the manipulations were analyzed statistically with regards to the preferred values of the musical parameters in relation to the different visual scenes. The results indicated awareness and knowledge about codes and conventions of musical narrative functions, and were to some degree affected by the participants’ gender, musical backgrounds and media habits

Detecting and removing visual distractors for video aesthetic enhancement

Personal videos often contain visual distractors, which are objects that are accidentally captured that can distract viewers from focusing on the main subjects. We propose a method to automatically detect and localize these distractors through learning from a manually labeled dataset. To achieve spatially and temporally coherent detection, we propose extracting features at the Temporal-Superpixel (TSP) level using a traditional SVM-based learning framework. We also experiment with end-to-end learning using Convolutional Neural Networks (CNNs), which achieves slightly higher performance than other methods. The classification result is further refined in a post-processing step based on graph-cut optimization. Experimental results show that our method achieves an accuracy of 81% and a recall of 86%. We demonstrate several ways of removing the detected distractors to improve the video quality, including video hole filling; video frame replacement; and camera path re-planning. The user study results show that our method can significantly improve the aesthetic quality of videos

Personalized Cinemagraphs using Semantic Understanding and Collaborative Learning

Cinemagraphs are a compelling way to convey dynamic aspects of a scene. In these media, dynamic and still elements are juxtaposed to create an artistic and narrative experience. Creating a high-quality, aesthetically pleasing cinemagraph requires isolating objects in a semantically meaningful way and then selecting good start times and looping periods for those objects to minimize visual artifacts (such a tearing). To achieve this, we present a new technique that uses object recognition and semantic segmentation as part of an optimization method to automatically create cinemagraphs from videos that are both visually appealing and semantically meaningful. Given a scene with multiple objects, there are many cinemagraphs one could create. Our method evaluates these multiple candidates and presents the best one, as determined by a model trained to predict human preferences in a collaborative way. We demonstrate the effectiveness of our approach with multiple results and a user study.Comment: To appear in ICCV 2017. Total 17 pages including the supplementary materia

The Need to Revise Copyright Law to Reflect the Changing Costs and Benefits of Modern Digital Reuse of Artistic Creations

Copyright law has always sought to maximize the quantity of valuable creative works available to society. While protecting the creative artists is essential, it is in some sense incidental; the reason to protect the artist is that them, there would be nothing to copy. As new digital technologies for transforming artistic works gain in capability, the ease of producing innovative and valuable works based on the reuse of prior work increases, meaning that society can now benefit from an increased supply of works based on the reuse of others. This suggests that restrictions on reuse that were considered optimal in the past should now be relaxed. We suggest changes to copyright law to achieve this new optimum. We suggest that artistic merit should once again be considered relevant to copyright law, in this case to determine when artistic works should be permitted to reuse works still subject to copyright protection. We retain the concept of originality in deciding when works based on reuse should themselves be granted copyright

"You Tube and I Find" - personalizing multimedia content access

Recent growth in broadband access and proliferation of small personal devices that capture images and videos has led to explosive growth of multimedia content available everywhereVfrom personal disks to the Web. While digital media capture and upload has become nearly universal with newer device technology, there is still a need for better tools and technologies to search large collections of multimedia data and to find and deliver the right content to a user according to her current needs and preferences. A renewed focus on the subjective dimension in the multimedia lifecycle, fromcreation, distribution, to delivery and consumption, is required to address this need beyond what is feasible today. Integration of the subjective aspects of the media itselfVits affective, perceptual, and physiological potential (both intended and achieved), together with those of the users themselves will allow for personalizing the content access, beyond today’s facility. This integration, transforming the traditional multimedia information retrieval (MIR) indexes to more effectively answer specific user needs, will allow a richer degree of personalization predicated on user intention and mode of interaction, relationship to the producer, content of the media, and their history and lifestyle. In this paper, we identify the challenges in achieving this integration, current approaches to interpreting content creation processes, to user modelling and profiling, and to personalized content selection, and we detail future directions. The structure of the paper is as follows: In Section I, we introduce the problem and present some definitions. In Section II, we present a review of the aspects of personalized content and current approaches for the same. Section III discusses the problem of obtaining metadata that is required for personalized media creation and present eMediate as a case study of an integrated media capture environment. Section IV presents the MAGIC system as a case study of capturing effective descriptive data and putting users first in distributed learning delivery. The aspects of modelling the user are presented as a case study in using user’s personality as a way to personalize summaries in Section V. Finally, Section VI concludes the paper with a discussion on the emerging challenges and the open problems

Who is the director of this movie? Automatic style recognition based on shot features

We show how low-level formal features, such as shot duration, meant as length of camera takes, and shot scale, i.e. the distance between the camera and the subject, are distinctive of a director's style in art movies. So far such features were thought of not having enough varieties to become distinctive of an author. However our investigation on the full filmographies of six different authors (Scorsese, Godard, Tarr, Fellini, Antonioni, and Bergman) for a total number of 120 movies analysed second by second, confirms that these shot-related features do not appear as random patterns in movies from the same director. For feature extraction we adopt methods based on both conventional and deep learning techniques. Our findings suggest that feature sequential patterns, i.e. how features evolve in time, are at least as important as the related feature distributions. To the best of our knowledge this is the first study dealing with automatic attribution of movie authorship, which opens up interesting lines of cross-disciplinary research on the impact of style on the aesthetic and emotional effects on the viewers

Quality Model for Massive Open Online Course (MOOC) Web Content

With the philosophy of providing open education to all, Massive Open Online Course (MOOC), which introduced in 2006, has been through its first decade. Despite its popularity and worldwide acceptance, MOOC faces a few criticisms about the weaknesses of its content such as lack of clarity, unstructured, poor design and lack of fundamental initial requirements. This caused by the paucity of understanding among content providers about the facet of qualities contributes to the content. There are some previous efforts to improve the quality of MOOCs, but none focused on the content from the content providers or experts' view. As a result, most of the vital internal quality factors are neglected. Besides, the operational definition for the MOOC content quality factors is still missing or not well-defined. Therefore, this research proposes a quality model for MOOC web content as a content provider’s reference to develop quality MOOC content. In order to achieve that, three basic elements were implemented which is the content’s provider perspective, MOOC content quality dimensions and MOOC content quality factors. Development of MOOC content quality dimension is based on 7C’s model and PDCA for continuity, while the determination of factors involves the process like a revision of the possible factors from literature, factors combination and categorization. This proposed hierarchical model tends to make MOOC's learning more optimistic and beneficial to the learners through the development of high-quality content

Reservoir hill and audiences for online interactive drama

This paper analyses the interactive experiences constructed for users of the New Zealand online interactive drama Reservoir Hill (2009, 2010), focusing both on the nature and levels of engagement which the series provided to users and the difficulties of audience research into this kind of media content. The series itself provided tightly prescribed forms of interactivity across multiple platforms, allowing forms of engagement that were greatly appreciated by its audience overall but actively explored only by a small proportion of users. The responses from members of the Reservoir Hill audience suggests that online users themselves are still learning the nature of, and constraints on, their engagements with various forms of online interactive media. This paper also engages with issue of how interactivity itself is defined, the difficulties of both connecting with audience members and securing timely access to online data, and the challenges of undertaking collaborative research with media producers in order to gain access to user data