ConStance: Modeling Annotation Contexts to Improve Stance Classification

Manual annotations are a prerequisite for many applications of machine learning. However, weaknesses in the annotation process itself are easy to overlook. In particular, scholars often choose what information to give to annotators without examining these decisions empirically. For subjective tasks such as sentiment analysis, sarcasm, and stance detection, such choices can impact results. Here, for the task of political stance detection on Twitter, we show that providing too little context can result in noisy and uncertain annotations, whereas providing too strong a context may cause it to outweigh other signals. To characterize and reduce these biases, we develop ConStance, a general model for reasoning about annotations across information conditions. Given conflicting labels produced by multiple annotators seeing the same instances with different contexts, ConStance simultaneously estimates gold standard labels and also learns a classifier for new instances. We show that the classifier learned by ConStance outperforms a variety of baselines at predicting political stance, while the model's interpretable parameters shed light on the effects of each context.Comment: To appear at EMNLP 201

Zero-shot Model Diagnosis

When it comes to deploying deep vision models, the behavior of these systems must be explicable to ensure confidence in their reliability and fairness. A common approach to evaluate deep learning models is to build a labeled test set with attributes of interest and assess how well it performs. However, creating a balanced test set (i.e., one that is uniformly sampled over all the important traits) is often time-consuming, expensive, and prone to mistakes. The question we try to address is: can we evaluate the sensitivity of deep learning models to arbitrary visual attributes without an annotated test set? This paper argues the case that Zero-shot Model Diagnosis (ZOOM) is possible without the need for a test set nor labeling. To avoid the need for test sets, our system relies on a generative model and CLIP. The key idea is enabling the user to select a set of prompts (relevant to the problem) and our system will automatically search for semantic counterfactual images (i.e., synthesized images that flip the prediction in the case of a binary classifier) using the generative model. We evaluate several visual tasks (classification, key-point detection, and segmentation) in multiple visual domains to demonstrate the viability of our methodology. Extensive experiments demonstrate that our method is capable of producing counterfactual images and offering sensitivity analysis for model diagnosis without the need for a test set.Comment: Accepted in CVPR 202

Learning visual saliency by combining feature maps in a nonlinear manner using AdaBoost

To predict where subjects look under natural viewing conditions, biologically inspired saliency models decompose visual input into a set of feature maps across spatial scales. The output of these feature maps are summed to yield the final saliency map. We studied the integration of bottom-up feature maps across multiple spatial scales by using eye movement data from four recent eye tracking datasets. We use AdaBoost as the central computational module that takes into account feature selection, thresholding, weight assignment, and integration in a principled and nonlinear learning framework. By combining the output of feature maps via a series of nonlinear classifiers, the new model consistently predicts eye movements better than any of its competitors

Exploring the Use of Social Media to Infer Relationships Between Demographics, Psychographics and Vaccine Hesitancy

The growing popularity of social media as a platform to obtain information and share one\u27s opinions on various topics makes it a rich source of information for research. In this study, we aimed to develop a framework to infer relationships between demographic and psychographic characteristics of a user and their opinion on a specific narrative - in this case, their stance on taking the COVID-19 vaccine. Twitter was the chosen platform due to the large USA user base and easily available data. Demographic traits included Race, Age, Gender, and Human-vs-Organization Status. Psychographic traits included the Big Five personality traits (Conscientiousness, Neuroticism, Openness, Agreeableness, Extraversion), Risk Seeking, Risk Aversion, Inward Focus, and Outward Focus. Our pipeline involved preprocessing the data, labelling tweets as vaccine-hesitant using distant supervision, training a vaccine hesitancy classifier to classify a second dataset, obtaining demographic and psychographic inferences for each user, and finally running a logistic regression with vaccine hesitancy as the dependent variable and sets of demographic and psychographic characteristics as the independent variable. We achieved an F1 score of 0.947 for our classifier and found statistically significant trends in vaccine hesitancy for race, age, gender, and human-vs- organization status. On the other hand, there were no significant relationships between any of the psychographic traits and vaccine hesitancy. It should be noted that this study was not pre-registered and the values for all variables (dependent and independent) come from noisy classifiers. As such, these results should only be viewed as a preliminary analysis of the demographic and psychographic factors correlated with vaccine hesitancy. We conclude that such a framework is a useful tool to identify the relations between different demographics and popular narratives. Further work and better data are necessary to improve the framework to the point where the strength of the correlations can be considered and not just the overall relationships. Furthermore, while psychographic traits yielded no significant results, there were several limitations in their inference, and focusing on improving psychographic trait inference is an important avenue for future studies

Context Embedding Networks

Low dimensional embeddings that capture the main variations of interest in collections of data are important for many applications. One way to construct these embeddings is to acquire estimates of similarity from the crowd. However, similarity is a multi-dimensional concept that varies from individual to individual. Existing models for learning embeddings from the crowd typically make simplifying assumptions such as all individuals estimate similarity using the same criteria, the list of criteria is known in advance, or that the crowd workers are not influenced by the data that they see. To overcome these limitations we introduce Context Embedding Networks (CENs). In addition to learning interpretable embeddings from images, CENs also model worker biases for different attributes along with the visual context i.e. the visual attributes highlighted by a set of images. Experiments on two noisy crowd annotated datasets show that modeling both worker bias and visual context results in more interpretable embeddings compared to existing approaches.Comment: CVPR 2018 spotligh

Cyberbullying Classification based on Social Network Analysis

With the popularity of social media platforms such as Facebook, Twitter, and Instagram, people widely share their opinions and comments over the Internet. Exten- sive use of social media has also caused a lot of problems. A representative problem is Cyberbullying, which is a serious social problem, mostly among teenagers. Cyber- bullying occurs when a social media user posts aggressive words or phrases to harass other users, and that leads to negatively affects on their mental and social well-being. Additionally, it may ruin the reputation of that media. We are considering the problem of detecting posts that are aggressive. Moreover, we try to detect Cyberbullies. In this research, we study Cyberbullying as a classification problem by combining text mining techniques, and the graph of the social network relationships based on a dataset from Twitter. We create an new dataset that has more information for each tweet (post). We improve the classification accuracy by considering the additional social network features based on the user’s follower list and retweet information