Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies

Stories can have tremendous power -- not only useful for entertainment, they can activate our interests and mobilize our actions. The degree to which a story resonates with its audience may be in part reflected in the emotional journey it takes the audience upon. In this paper, we use machine learning methods to construct emotional arcs in movies, calculate families of arcs, and demonstrate the ability for certain arcs to predict audience engagement. The system is applied to Hollywood films and high quality shorts found on the web. We begin by using deep convolutional neural networks for audio and visual sentiment analysis. These models are trained on both new and existing large-scale datasets, after which they can be used to compute separate audio and visual emotional arcs. We then crowdsource annotations for 30-second video clips extracted from highs and lows in the arcs in order to assess the micro-level precision of the system, with precision measured in terms of agreement in polarity between the system's predictions and annotators' ratings. These annotations are also used to combine the audio and visual predictions. Next, we look at macro-level characterizations of movies by investigating whether there exist `universal shapes' of emotional arcs. In particular, we develop a clustering approach to discover distinct classes of emotional arcs. Finally, we show on a sample corpus of short web videos that certain emotional arcs are statistically significant predictors of the number of comments a video receives. These results suggest that the emotional arcs learned by our approach successfully represent macroscopic aspects of a video story that drive audience engagement. Such machine understanding could be used to predict audience reactions to video stories, ultimately improving our ability as storytellers to communicate with each other.Comment: Data Mining (ICDM), 2017 IEEE 17th International Conference o

Explorando de la obra espiritista de Amalia Domingo Soler desde la lectura distante: arcos emocionales mediante análisis de sentimiento

Esta presentación avanza los resultados publicados en un capítulo del libro Arias Doblas, R. Zaro Vera, J. (Eds.) 2022. Género, heterodoxia y traducción: difusión del ocultismo en España y el ámbito europeo. Kassel: Reichenberger.Amalia Domingo Soler (1835-1909) fue una escritora perteneciente al movimiento espiritista y una influyente librepensadora, feminista, editora y agitadora cultural, cuya prolífica obra literaria de cinco décadas (Correa Ramón, 2007). En esta ponencia proponemos una aproximación a su antología Cuentos espiritistas, reeditada en 2010, a través de las oportunidades que ofrece la lectura distante o distant Reading (Moretti, 2013) con un enfoque de análisis de sentimiento. Esta es una subdisciplina del procesamiento del lenguaje natural y el análisis de texto que permite identificar y cuantificar el sentimiento y la información subjetiva en los textos. El objetivo es clasificar estas obras según los seis arcos narrativos universales propuestos por Kurt Vonnegut (1995) mediante análisis de sentimiento e identificar las características del sentimiento relativos en cada una de las escenas. Hemos utilizado los potentes algoritmos de la herramienta Lingmotif (Moreno-Ortiz, 2022), desarrollada por la Universidad de Málaga, con el fin de profundizar en el estudio sistemático de los sentimientos y emociones en los textos literarios. Observamos coincidencias en el flujo de sentimientos de las narraciones y se identificaron las características del tono en cada una de las escenas y la importancia que tiene la polaridad para el desarrollo de la trama. Finalmente, se generó una serie de visualizaciones de datos (nubes de palabras, gráficos longitudinales de sentimiento e intensidad, etc.) que ponen sobre la mesa el valor adicional (Chu y Roy, 2017) que la lectura distante puede ofrecer a la investigación en estudios literarios ya que hasta el momento ha sido una metodología poco explorada (Kim y Klinger, 2021) en la literatura en lengua española. Esta presentación avanza los resultados publicados en un capítulo del libro Género, heterodoxia y traducción: difusión del ocultismo en España y el ámbito europeo.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech. Unión Europea - NextGenerationE

Live Sentiment Annotation of Movies via Arduino and a Slider

In this contribution, we present the first version of a novel approach and prototype to perform live sentiment annotation of movies while watching them. Our prototype consists of an Arduino microcontroller and a potentiometer, which is paired with a slider. We motivate the need for this approach by arguing that the presentation of multimedia content of movies as well as performing the annotation live during the viewing of the movie is beneficial for the annotation process and more intuitive for the viewer/annotator. After outlining the motivation and the technical setup of our system, we report on which studies we plan to validate the benefits of our system

Automated Composition of Picture-Synched Music Soundtracks for Movies

We describe the implementation of and early results from a system that automatically composes picture-synched musical soundtracks for videos and movies. We use the phrase "picture-synched" to mean that the structure of the automatically composed music is determined by visual events in the input movie, i.e. the final music is synchronised to visual events and features such as cut transitions or within-shot key-frame events. Our system combines automated video analysis and computer-generated music-composition techniques to create unique soundtracks in response to the video input, and can be thought of as an initial step in creating a computerised replacement for a human composer writing music to fit the picture-locked edit of a movie. Working only from the video information in the movie, key features are extracted from the input video, using video analysis techniques, which are then fed into a machine-learning-based music generation tool, to compose a piece of music from scratch. The resulting soundtrack is tied to video features, such as scene transition markers and scene-level energy values, and is unique to the input video. Although the system we describe here is only a preliminary proof-of-concept, user evaluations of the output of the system have been positive.Comment: To be presented at the 16th ACM SIGGRAPH European Conference on Visual Media Production. London, England: 17th-18th December 2019. 10 pages, 9 figure

Análisis de sentimientos en Twitter: Un estudio comparativo

Sentiment analysis helps to determine the perception of users in different aspects of daily life, such as product preferences in the market, level of user confidence in work environments, or political preferences. The idea is to predict trends or preferences based on feelings. In this article we evaluate the most common techniques used for this type of analysis, considering machine learning and deep machine learning techniques. Our main contribution is based on a proposal for a methodological strategy that covers the phases of data preprocessing, construction of predictive models and their evaluation. From the results, the best classical model was SVM, with 78% accuracy, and 79% F1 metric (F1 score). For the Deep Learning models, the classical models had the best results. The model with the best performance was the Deep Learning Long Short Term Memory (LSTM), reaching 88% accuracy and 89% F1 metric. The worst of the Deep Learning models was the CNN, with 77% accuracy as an F1 metric. Concluding that the Long Short Term Memory (LSTM) algorithm proved to be the best performance, reaching up to 89% accuracy.El análisis de sentimientos ayuda a determinar la percepción de usuarios en diferentes aspectos de la vida cotidiana, como preferencias de productos en el mercado, nivel de confianza de los usuarios en ambientes de trabajo, o preferencias políticas. La idea es predecir tendencias o preferencias basados en sentimientos. En este artículo evaluamos las técnicas más comunes usadas para este tipo de análisis, considerando técnicas de aprendizaje de máquina y aprendizaje de máquina profundo. Nuestra contribución principal se basa en una propuesta de una estrategia metodológica que abarca las fases de preprocesamiento de datos, construcción de modelos predictivos y su evaluación. De los resultados, el mejor modelo clásico fue SVM, con 78% de precisión, y 79% de métrica F1 (F1 score). Para los modelos de Deep Learning, con mejores resultados fueron los modelos clásicos. El modelo con mejor desempeño fue el de Deep Learning Long Short Term Memory (LSTM), alcanzando un 88% de precisión y 89% de métrica F1. El peor de los modelos de Deep Learning fue el CNN, con 77% de precisión como de métrica F1. Concluyendo que, el algoritmo Long Short Term Memory (LSTM) demostró ser el mejor rendimiento, alcanzando hasta un 89% de precisión.

Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies

© 2017 IEEE. Stories can have tremendous power - not only useful for entertainment, they can activate our interests and mobilize our actions. The degree to which a story resonates with its audience may be in part reflected in the emotional journey it takes the audience upon. In this paper, we use machine learning methods to construct emotional arcs in movies, calculate families of arcs, and demonstrate the ability for certain arcs to predict audience engagement. The system is applied to Hollywood films and high quality shorts found on the web. We begin by using deep convolutional neural networks for audio and visual sentiment analysis. These models are trained on both new and existing large-scale datasets, after which they can be used to compute separate audio and visual emotional arcs. We then crowdsource annotations for 30-second video clips extracted from highs and lows in the arcs in order to assess the micro-level precision of the system, with precision measured in terms of agreement in polarity between the system's predictions and annotators' ratings. These annotations are also used to combine the audio and visual predictions. Next, we look at macro-level characterizations of movies by investigating whether there exist 'universal shapes' of emotional arcs. In particular, we develop a clustering approach to discover distinct classes of emotional arcs. Finally, we show on a sample corpus of short web videos that certain emotional arcs are statistically significant predictors of the number of comments a video receives. These results suggest that the emotional arcs learned by our approach successfully represent macroscopic aspects of a video story that drive audience engagement. Such machine understanding could be used to predict audience reactions to video stories, ultimately improving our ability as storytellers to communicate with each other