EmoNets: Multimodal deep learning approaches for emotion recognition in video

The task of the emotion recognition in the wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies. The videos depict acted-out emotions under realistic conditions with a large degree of variation in attributes such as pose and illumination, making it worthwhile to explore approaches which consider combinations of features from multiple modalities for label assignment. In this paper we present our approach to learning several specialist models using deep learning techniques, each focusing on one modality. Among these are a convolutional neural network, focusing on capturing visual information in detected faces, a deep belief net focusing on the representation of the audio stream, a K-Means based "bag-of-mouths" model, which extracts visual features around the mouth region and a relational autoencoder, which addresses spatio-temporal aspects of videos. We explore multiple methods for the combination of cues from these modalities into one common classifier. This achieves a considerably greater accuracy than predictions from our strongest single-modality classifier. Our method was the winning submission in the 2013 EmotiW challenge and achieved a test set accuracy of 47.67% on the 2014 dataset

Clustering of LMS Use Strategies with Autoencoders

Learning Management Systems provide teachers with many functionalities to offer materials to students, interact with them and manage their courses. Recognizing teachers’ instructing styles from their course designs would allow recommendations and best practices to be made. We propose a method that determines teaching style in an unsupervised way from the course structure and use patterns. We define a course classification approach based on deep learning and clustering. We first use an autoencoder to reduce the dimensionality of the input data, while extracting the most important characteristics; thus, we obtain a latent representation of the courses. We then apply clustering techniques to the latent data to group courses based on their use patterns. The results show that this technique improves the clustering performance while avoiding the manual data pre-processing work. Furthermore, the obtained model defines seven course typologies that are clearly related to different use patterns of Learning Management Systems

Face Recognition System based on Wavelet, PCA-LDA and SVM

Face recognition has become one of the important research areas that is used in wide range of applications such as security systems, Information security and personal identification etc. In this paper, we deal with a face recognition system using wavelet transformation to preprocessing of face image and combination of Linear Discriminate Analysis and Principle Components Analysis, algorithm to extract features. Finally, support vector machine algorithm is used for classification. An experiment on Olivetti-Oracle Research Laboratory face is implemented to test performance of the proposed method. The results of this experiment showed the reliability and effectiveness of the proposed method. Keywords: Face recognition, Principal Component Analyses, Linear Discriminate Analysis, Support Vector Machine

Analyzing Domestic Abuse using Natural Language Processing on Social Media Data

Social media and social networking play a major role in billions of lives. Publicly available posts on websites such as Twitter, Reddit, Tumblr, and Facebook can contain deeply personal accounts of the lives of users – and the crises they face. Health woes, family concerns, accounts of bullying, and any number of other issues that people face every day are detailed on a massive scale online. Utilizing natural language processing and machine learning techniques, these data can be analyzed to understand societal and public health issues. Expensive surveys need not be conducted with automatic understanding of social media data, allowing faster, cost-effective data collection and analysis that can shed light on sociologically important problems. In this thesis, discussions of domestic abuse in social media are analyzed. The efficacy of classifiers that detect text discussing abuse is examined and computationally extracted characteristics of these texts are analyzed for a comprehensive view into the dynamics of abusive relationships. Analysis reveals micro-narratives in reasons for staying in versus leaving abusive relationships, as well as the stakeholders and actions in these relationships. Findings are consistent across various methods, correspond to observations in clinical literature, and affirm the relevance of natural language processing techniques for exploring issues of social importance in social media

Understanding novice programmer behavior on introductory courses - Learning analytics approach

It is not easy to learn programming. This is why increasing theoretical and practical knowledge in programming education benefits both the educators as well as the students. To allow the students to gain maximal benefit from their studies, the educator must be able to recognize the students who are struggling with learning programming. Learning analytics provides a possible solution to this problem. This thesis demonstrates a novel method to model programmer behavior by using Markov Models. Programming fulfills the Markov property, because the success of the next attempt to compile or execute code is not influenced by the previous attempts; only by the current skill level of the programmer. The model is built using a state machine, which consists of states representing the different phases of the programming process. The state machine contains eight different states and 29different state transition possibilities. A Markov chain corresponding to a specific student can be computed using this state machine and then used with, for example machine learning algorithms. The data for this thesis was collected from a total of five different introductory programming courses, which used either the Java or Python programming languages. The dataset contains 1174 unique students, who made 544 835 total submissions to411 unique assignments. All programming courses were given in Turku, during2017-2021.This thesis provides a theoretical basis for modeling students (Markov Models) and offers a practical method to model students using Markov Models. This thesis only applies unsupervised machine learning methods to the data, specifically the K-Means clustering algorithm. However, supervised methods may also be used. The usefulness of the model is demonstrated by clustering students into three statistically similar clusters: students who perform well, average and poorly. The model is also applied to recognize the programming language used, based only on the transitions within the state machine.--- Ohjelmoinnin oppiminen ei ole helppoa. Tästä syystä ohjelmoinnin opetuksenteoreettinen ja käytännön edistäminen hyödyttää paitsi nykyisin ohjelmointia opettavia, myös opiskelijoita. Jotta opiskelijat voivat saavuttaa maksimaalisenhyödyn opiskelustaan, opettajan täytyy voida tunnistaa ne opiskelijat, joille ohjelmoinnin opiskelu tuottaa hankaluuksia. Oppimisanalytiikka tarjoaa tähän mahdollisuuden. Tämä väitöskirja esittelee tavan mallintaa ohjelmoinnin opiskelijoidenkäyttäytymistä käyttämällä Markovin malleja. Ohjelmoijan käyttäytyminen toteuttaa Markovin ominaisuuden, sillä ohjelmoijan koodin ajoyrityksen onnistumiseen vaikuttaa ainoastaan ohjelmoijan senhetkinen taitotaso; aikaisemmilla yrityksillä ei ole vaikutusta tuleviin kertoihin. Malli rakennetaan käyttämällä tilakonetta, jonka jokainen tila vastaa ohjelmointiprosessin vaihetta. Tilakoneessa on yhteensä kahdeksan eri tilaa ja 29 erilaista tilan muutosmahdollisuutta. Tilakoneesta lasketaan opiskelijaa vastaava Markovin ketju, mitä voidaan käyttää esimerkiksi koneoppimisalgoritmien kanssa. Dataa tähän väitöskirjaan kerättiin yhteensä viidestä ohjelmoinninperuskurssista, joissa käytettiin joko Java- tai Python-ohjelmointikieltä. Opiskelijoita kursseilla oli yhteensä 1174. Opiskelijat tekivät yhteensä 544-835 ohjelmointitehtävän palautusta 411 ohjelmointitehtävään. Kaikki ohjelmointikurssit pidettiin Turussa, vuosina 2017-2021 Tämä väitöskirja tarjoaa teoreettisen pohjan ohjelmoinnin opiskelijoidenmallintamiseen (Markovin mallit) ja tarjoaa menetelmän, jolla Markovin malleja käyttämällä voi mallintaa ohjelmoinnin opiskelijoita. Malliin sovelletaan vain ohjaamattomia koneoppimismenetelmiä, erityisesti K-Means clustering -algoritmia. Tässä väitöskirjassa osoitan myös teoreettisen mallin muutamia käytännönsovelluksia luokittelemalla opiskelijoita samoja ominaisuuksia sisältäviin luokkiin. Malli opetetaan erottelemaan opiskelijat kolmeen ryhmään: hyvin, keskiverrosti ja huonosti pärjääviin. Mallia sovelletaan onnistuneesti myös tunnistamaan käytetty ohjelmointikieli käyttämällä vain tilakoneen tilasiirtymiä

Introduction: The Third International Conference on Epigenetic Robotics

This paper summarizes the paper and poster contributions to the Third International Workshop on Epigenetic Robotics. The focus of this workshop is on the cross-disciplinary interaction of developmental psychology and robotics. Namely, the general goal in this area is to create robotic models of the psychological development of various behaviors. The term "epigenetic" is used in much the same sense as the term "developmental" and while we could call our topic "developmental robotics", developmental robotics can be seen as having a broader interdisciplinary emphasis. Our focus in this workshop is on the interaction of developmental psychology and robotics and we use the phrase "epigenetic robotics" to capture this focus

Towards Universal Image Embeddings: A Large-Scale Dataset and Challenge for Generic Image Representations

Fine-grained and instance-level recognition methods are commonly trained and evaluated on specific domains, in a model per domain scenario. Such an approach, however, is impractical in real large-scale applications. In this work, we address the problem of universal image embedding, where a single universal model is trained and used in multiple domains. First, we leverage existing domain-specific datasets to carefully construct a new large-scale public benchmark for the evaluation of universal image embeddings, with 241k query images, 1.4M index images and 2.8M training images across 8 different domains and 349k classes. We define suitable metrics, training and evaluation protocols to foster future research in this area. Second, we provide a comprehensive experimental evaluation on the new dataset, demonstrating that existing approaches and simplistic extensions lead to worse performance than an assembly of models trained for each domain separately. Finally, we conducted a public research competition on this topic, leveraging industrial datasets, which attracted the participation of more than 1k teams worldwide. This exercise generated many interesting research ideas and findings which we present in detail. Project webpage: https://cmp.felk.cvut.cz/univ_emb/Comment: ICCV 2023 Accepte