Annotating Social Media Data From Vulnerable Populations: Evaluating Disagreement Between Domain Experts and Graduate Student Annotators

Researchers in computer science have spent considerable time developing methods to increase the accuracy and richness of annotations. However, there is a dearth in research that examines the positionality of the annotator, how they are trained and what we can learn from disagreements between different groups of annotators. In this study, we use qualitative analysis, statistical and computational methods to compare annotations between Chicago-based domain experts and graduate students who annotated a total of 1,851 tweets with images that are a part of a larger corpora associated with the Chicago Gang Intervention Study, which aims to develop a computational system that detects aggression and loss among gang-involved youth in Chicago. We found evidence to support the study of disagreement between annotators and underscore the need for domain expertise when reviewing Twitter data from vulnerable populations. Implications for annotation and content moderation are discussed

Multimodal Social Media Analysis for Gang Violence Prevention

Gang violence is a severe issue in major cities across the U.S. and recent studies [Patton et al. 2017] have found evidence of social media communications that can be linked to such violence in communities with high rates of exposure to gang activity. In this paper we partnered computer scientists with social work researchers, who have domain expertise in gang violence, to analyze how public tweets with images posted by youth who mention gang associations on Twitter can be leveraged to automatically detect psychosocial factors and conditions that could potentially assist social workers and violence outreach workers in prevention and early intervention programs. To this end, we developed a rigorous methodology for collecting and annotating tweets. We gathered 1,851 tweets and accompanying annotations related to visual concepts and the psychosocial codes: aggression, loss, and substance use. These codes are relevant to social work interventions, as they represent possible pathways to violence on social media. We compare various methods for classifying tweets into these three classes, using only the text of the tweet, only the image of the tweet, or both modalities as input to the classifier. In particular, we analyze the usefulness of mid-level visual concepts and the role of different modalities for this tweet classification task. Our experiments show that individually, text information dominates classification performance of the loss class, while image information dominates the aggression and substance use classes. Our multimodal approach provides a very promising improvement (18% relative in mean average precision) over the best single modality approach. Finally, we also illustrate the complexity of understanding social media data and elaborate on open challenges

Computational Approaches to Subjective Interpretation of Multimedia Messages

Nowadays a large part of communication is taking place on social media platforms such as Twitter, Facebook, Instagram, or YouTube, where messages often include multimedia contents (e.g., images, GIFs or videos). Since such messages are in digital form, computers can in principle process them in order to make our lives more convenient and help us overcome arising issues. However, these goals require the ability to capture what these messages mean to us, that is, how we interpret them from our own subjective points of view. Thus, the main goal of this dissertation is to advance a machine's ability to interpret social media contents in a more natural, subjective way. To this end, three research questions are addressed. The first question aims at answering "How to model human interpretation for machine learning?" We describe a way of modeling interpretation which allows for analyzing single or multiple ways of interpretation of both humans and computer models within the same theoretic framework. In a comprehensive survey we collect various possibilities for such a computational analysis. Particularly interesting are machine learning approaches where a single neural network learns multiple ways of interpretation. For example, a neural network can be trained to predict user-specific movie ratings from movie features and user ID, and can then be analyzed to understand how users rate movies. This is a promising direction, as neural networks are capable of learning complex patterns. However, how analysis results depend on network architecture is a largely unexplored topic. For the example of movie ratings, we show that the way of combining information for prediction can affect both prediction performance and what the network learns about the various ways of interpretation (corresponding to users). Since some application-specific details for dealing with human interpretation only become visible when going deeper into particular use-cases, the other two research questions of this dissertation are concerned with two selected application domains: Subjective visual interpretation and gang violence prevention. The first application study deals with subjectivity that comes from personal attitudes and aims at answering "How can we predict subjective image interpretation one would expect from the general public on photo-sharing platforms such as Flickr?" The predictions in this case take the form of subjective concepts or phrases. Our study on gang violence prevention is more community-centered and considers the question "How can we automatically detect tweets of gang members which could potentially lead to violence?" There, the psychosocial codes aggression, loss and substance use serve as proxy to estimate the subjective implications of online messages. In these two distinct application domains, we develop novel machine learning models for predicting subjective interpretations of images or tweets with images, respectively. In the process of building these detection tools, we also create three different datasets which we share with the research community. Furthermore, we see that some domains such as Chicago gangs require special care due to high vulnerability of involved users. This motivated us to establish and describe an in-depth collaboration between social work researchers and computer scientists. As machine learning is incorporating more and more subjective components and gaining societal impact, we have good reason to believe that similar collaborations between the humanities and computer science will become increasingly necessary to advance the field in an ethical way

Mathematical Modeling of Synaptic Transmission at Chemical Synapses

Synapses are connections between different nerve cells that form an essential link in neural signal transmission. It is generally distinguished between electrical and chemical synapses, where chemical synapses are more common in the human brain and are also the type we deal with in this work. In chemical synapses, small container-like objects called vesicles fill with neurotransmitter and expel them from the cell during synaptic transmission. This process is vital for communication between neurons. However, to the best of our knowledge no mathematical models that take different filling states of the vesicles into account have been developed before this thesis was written. In this thesis we propose a novel mathematical model for modeling synaptic transmission at chemical synapses which includes the description of vesicles of different filling states. The model consists of a transport equation (for the vesicle growth process) plus three ordinary differential equations (ODEs) and focuses on the presynapse and synaptic cleft. The well-posedness is proved in detail for this partial differential equation (PDE) system. We also propose a few different variations and related models. In particular, an ODE system is derived and a delay differential equation (DDE) system is formulated. We then use nonlinear optimization methods for data fitting to test some of the models on data made available to us by the Animal Physiology group at TU Kaiserslautern

Computational Approaches to Subjective Interpretation of Multimedia Messages

Nowadays a large part of communication is taking place on social media platforms such as Twitter, Facebook, Instagram, or YouTube, where messages often include multimedia contents (e.g., images, GIFs or videos). Since such messages are in digital form, computers can in principle process them in order to make our lives more convenient and help us overcome arising issues. However, these goals require the ability to capture what these messages mean to us, that is, how we interpret them from our own subjective points of view. Thus, the main goal of this dissertation is to advance a machine's ability to interpret social media contents in a more natural, subjective way. To this end, three research questions are addressed. The first question aims at answering "How to model human interpretation for machine learning?" We describe a way of modeling interpretation which allows for analyzing single or multiple ways of interpretation of both humans and computer models within the same theoretic framework. In a comprehensive survey we collect various possibilities for such a computational analysis. Particularly interesting are machine learning approaches where a single neural network learns multiple ways of interpretation. For example, a neural network can be trained to predict user-specific movie ratings from movie features and user ID, and can then be analyzed to understand how users rate movies. This is a promising direction, as neural networks are capable of learning complex patterns. However, how analysis results depend on network architecture is a largely unexplored topic. For the example of movie ratings, we show that the way of combining information for prediction can affect both prediction performance and what the network learns about the various ways of interpretation (corresponding to users). Since some application-specific details for dealing with human interpretation only become visible when going deeper into particular use-cases, the other two research questions of this dissertation are concerned with two selected application domains: Subjective visual interpretation and gang violence prevention. The first application study deals with subjectivity that comes from personal attitudes and aims at answering "How can we predict subjective image interpretation one would expect from the general public on photo-sharing platforms such as Flickr?" The predictions in this case take the form of subjective concepts or phrases. Our study on gang violence prevention is more community-centered and considers the question "How can we automatically detect tweets of gang members which could potentially lead to violence?" There, the psychosocial codes aggression, loss and substance use serve as proxy to estimate the subjective implications of online messages. In these two distinct application domains, we develop novel machine learning models for predicting subjective interpretations of images or tweets with images, respectively. In the process of building these detection tools, we also create three different datasets which we share with the research community. Furthermore, we see that some domains such as Chicago gangs require special care due to high vulnerability of involved users. This motivated us to establish and describe an in-depth collaboration between social work researchers and computer scientists. As machine learning is incorporating more and more subjective components and gaining societal impact, we have good reason to believe that similar collaborations between the humanities and computer science will become increasingly necessary to advance the field in an ethical way

Mathematical Modeling of Synaptic Transmission at Chemical Synapses

Synapses are connections between different nerve cells that form an essential link in neural signal transmission. It is generally distinguished between electrical and chemical synapses, where chemical synapses are more common in the human brain and are also the type we deal with in this work. In chemical synapses, small container-like objects called vesicles fill with neurotransmitter and expel them from the cell during synaptic transmission. This process is vital for communication between neurons. However, to the best of our knowledge no mathematical models that take different filling states of the vesicles into account have been developed before this thesis was written. In this thesis we propose a novel mathematical model for modeling synaptic transmission at chemical synapses which includes the description of vesicles of different filling states. The model consists of a transport equation (for the vesicle growth process) plus three ordinary differential equations (ODEs) and focuses on the presynapse and synaptic cleft. The well-posedness is proved in detail for this partial differential equation (PDE) system. We also propose a few different variations and related models. In particular, an ODE system is derived and a delay differential equation (DDE) system is formulated. We then use nonlinear optimization methods for data fitting to test some of the models on data made available to us by the Animal Physiology group at TU Kaiserslautern