Developing a comprehensive framework for multimodal feature extraction

Feature extraction is a critical component of many applied data science workflows. In recent years, rapid advances in artificial intelligence and machine learning have led to an explosion of feature extraction tools and services that allow data scientists to cheaply and effectively annotate their data along a vast array of dimensions---ranging from detecting faces in images to analyzing the sentiment expressed in coherent text. Unfortunately, the proliferation of powerful feature extraction services has been mirrored by a corresponding expansion in the number of distinct interfaces to feature extraction services. In a world where nearly every new service has its own API, documentation, and/or client library, data scientists who need to combine diverse features obtained from multiple sources are often forced to write and maintain ever more elaborate feature extraction pipelines. To address this challenge, we introduce a new open-source framework for comprehensive multimodal feature extraction. Pliers is an open-source Python package that supports standardized annotation of diverse data types (video, images, audio, and text), and is expressly with both ease-of-use and extensibility in mind. Users can apply a wide range of pre-existing feature extraction tools to their data in just a few lines of Python code, and can also easily add their own custom extractors by writing modular classes. A graph-based API enables rapid development of complex feature extraction pipelines that output results in a single, standardized format. We describe the package's architecture, detail its major advantages over previous feature extraction toolboxes, and use a sample application to a large functional MRI dataset to illustrate how pliers can significantly reduce the time and effort required to construct sophisticated feature extraction workflows while increasing code clarity and maintainability

Revisiting “Recognizing Human Activities User- Independently on Smartphones Based on Accelerometer Data” – What Has Happened Since 2012?

Our article “Recognizing human activities user-independently on smartphones based on accelerometer data” was published in the International Journal of Interactive Multimedia and Artificial Intelligence (IJIMAI) in 2012. In 2018, it was selected as the most outstanding article published in the 10 years of IJIMAI life. To celebrate the 10th anniversary of IJIMAI, in this article we will introduce what has happened in the field of human activity recognition and wearable sensor-based recognition since 2012, and especially, this article concentrates on introducing our work since 2012

Comparing Social Science and Computer Science Workflow Processes for Studying Group Interactions

In this article, a team of authors from the Geeks and Groupies workshop, in Leiden, the Netherlands, compare prototypical approaches to studying group interaction in social science and computer science disciplines, which we call workflows. To help social and computer science scholars understand and manage these differences, we organize workflow into three major stages: research design, data collection, and analysis. For each stage, we offer a brief overview on how scholars from each discipline work. We then compare those approaches and identify potential synergies and challenges. We conclude our article by discussing potential directions for more integrated and mutually beneficial collaboration that go beyond the producer–consumer model

Psychological research in the digital age

The smartphone has become an important personal companion in our daily lives. Each time we use the device, we generate data that provides information about ourselves. This data, in turn, is valuable to science because it objectively reflects our everyday behavior and experiences. In this way, smartphones enable research that is closer to everyday life than traditional laboratory experiments and questionnaire-based methods. While data collected with smartphones are increasingly being used in the field of personality psychology, new digital technologies can also be leveraged to collect and analyze large-scale unobtrusively sensed data in other areas of psychological research. This dissertation, therefore, explores the insights that smartphone sensing reveals for psychological research using two examples, situation and affect research, making a twofold research contribution. First, in two empirical studies, different data types of smartphone-sensed data, such as GPS or phone data, were combined with experience-sampled self-report, and classical questionnaire data to gain valuable insights into individual behavior, thinking, and feeling in everyday life. Second, predictive modeling techniques were applied to analyze the large, high-dimensional data sets collected by smartphones. To gain a deeper understanding of the smartphone data, interpretable variables were extracted from the raw sensing data, and the predictive performance of various machine learning algorithms was compared. In summary, the empirical findings suggest that smartphone data can effectively capture certain situational and behavioral indicators of psychological phenomena in everyday life. However, in certain research areas such as affect research, smartphone data should only complement, but not completely replace, traditional questionnaire-based data as well as other data sources such as neurophysiological indicators. The dissertation also concludes that the use of smartphone sensor data introduces new difficulties and challenges for psychological research and that traditional methods and perspectives are reaching their limits. The complexity of data collection, processing, and analysis requires established guidelines for study design, interdisciplinary collaboration, and theory-driven research that integrates explanatory and predictive approaches. Accordingly, further research is needed on how machine learning models and other big data methods in psychology can be reconciled with traditional theoretical approaches. Only in this way can we move closer to the ultimate goal of psychology to better understand, explain, and predict human behavior and experiences and their interplay with everyday situations

Py-Feat: Python Facial Expression Analysis Toolbox

Studying facial expressions is a notoriously difficult endeavor. Recent advances in the field of affective computing have yielded impressive progress in automatically detecting facial expressions from pictures and videos. However, much of this work has yet to be widely disseminated in social science domains such as psychology. Current state of the art models require considerable domain expertise that is not traditionally incorporated into social science training programs. Furthermore, there is a notable absence of user-friendly and open-source software that provides a comprehensive set of tools and functions that support facial expression research. In this paper, we introduce Py-Feat, an open-source Python toolbox that provides support for detecting, preprocessing, analyzing, and visualizing facial expression data. Py-Feat makes it easy for domain experts to disseminate and benchmark computer vision models and also for end users to quickly process, analyze, and visualize face expression data. We hope this platform will facilitate increased use of facial expression data in human behavior research.Comment: 25 pages, 3 figures, 5 table

Understanding Teacher Morale

This study emerged from discussions within the Policy and Planning Council of the Metropolitan Educational Research Consortium (MERC), a research alliance between Virginia Commonwealth University’s School of Education and seven surrounding school divisions. The project has two goals. The first goal is to develop an understanding of the factors that impact teachers’ experience of their work in the current PK12 public school context. Although this topic could be, and has been, investigated through a number of lenses (e.g., burnout, trust, motivation), this project focuses on the idea of teacher morale, a choice that will be discussed in detail in the next section of the report. The study addresses the following three questions: 1. How do teachers experience job satisfaction and morale? 2. What are the dynamics between a teacher’s job related ideal and the professional culture of the school that support or hinder the experience of job satisfaction and morale? 3. How do differences between schools related to policy context and social context affect the dynamics of job satisfaction and morale? To answer these questions MERC assembled a research team comprised of a university researcher, graduate students, and a team of school personnel from the MERC school divisions. Over the course of two years, the team developed a conceptual framework for understanding teacher morale, designed a research study that involved observing and interviewing teachers (n=44) across three purposefully selected middle schools in the Richmond region, and then collected and analyzed the data. This report shares both the process and the findings of this collaborative research effort. The second goal of this research project is to support action by local policy makers, school division leaders, central office personnel, principals, and teachers. The study was commissioned by local school leaders not just to document and reflect on teacher morale, but more importantly to do something about it. As argued above, teachers and the conditions of teachers’ work matters for our students, our schools, and the well being of our communities and society. In this regard, this report is only one piece of this project’s action and impact plan. While the report does contain a series of recommendations based on findings and how they can be used, the release of the report is tied to additional dissemination and professional development efforts designed to effect change

Gay Male Identity in the Context of College: Implications for Development, Support, and Campus Climate

This dissertation includes three articles that explore the relationship between gay identity and the college environment. The college environment has been shown to affect students’ attitudes, beliefs, and personal development in various ways, including aspects of individuals’ identity and attitudes towards social and political issues in society. D’Augelli’s (1994) lesbian-gay-bisexual (LGB) identity development framework provides both a priori knowledge of issues associated with gay identity and a lens through which findings are analyzed in each of the articles included in this dissertation. The first article examines the relationship between first-year college students’ personal characteristics and their attitudes towards same-sex relationships. Given the importance of peers as “valued others” to gay individuals, as well as the role that students play in establishing campus climate, the first article has implications for how the college environment is experienced by gay individuals. The second article explores the identity development of Black gay male college students. This article attempts to test the applicability of D’Augelli’s framework for racial minorities and for contemporary college students who also identify as gay. The third article included in this dissertation focuses on the representations of gay male college students in the online community called Facebook. Since representations are expressions of identity, this article has significance for understanding how gay male college students internalize information about their gay identity and selectively represent that identity to others. Considered together, these articles hold significance for researchers who study LGB individuals in higher education and administrators who work with LGB individuals on college campuses. Additionally, a revised theoretical framework that accounts for the findings discussed within these three articles is presented in the final chapter