When in doubt ask the crowd : leveraging collective intelligence for improving event detection and machine learning

[no abstract

Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey

Image classification systems recently made a giant leap with the advancement of deep neural networks. However, these systems require an excessive amount of labeled data to be adequately trained. Gathering a correctly annotated dataset is not always feasible due to several factors, such as the expensiveness of the labeling process or difficulty of correctly classifying data, even for the experts. Because of these practical challenges, label noise is a common problem in real-world datasets, and numerous methods to train deep neural networks with label noise are proposed in the literature. Although deep neural networks are known to be relatively robust to label noise, their tendency to overfit data makes them vulnerable to memorizing even random noise. Therefore, it is crucial to consider the existence of label noise and develop counter algorithms to fade away its adverse effects to train deep neural networks efficiently. Even though an extensive survey of machine learning techniques under label noise exists, the literature lacks a comprehensive survey of methodologies centered explicitly around deep learning in the presence of noisy labels. This paper aims to present these algorithms while categorizing them into one of the two subgroups: noise model based and noise model free methods. Algorithms in the first group aim to estimate the noise structure and use this information to avoid the adverse effects of noisy labels. Differently, methods in the second group try to come up with inherently noise robust algorithms by using approaches like robust losses, regularizers or other learning paradigms

A study of security issues of mobile apps in the android platform using machine learning approaches

Mobile app poses both traditional and new potential threats to system security and user privacy. There are malicious apps that may do harm to the system, and there are mis-behaviors of apps, which are reasonable and legal when not abused, yet may lead to real threats otherwise. Moreover, due to the nature of mobile apps, a running app in mobile devices may be only part of the software, and the server side behavior is usually not covered by analysis. Therefore, direct analysis on the app itself may be incomplete and additional sources of information are needed. In this dissertation, we discuss how we can apply machine learning techniques in multiple tasks for security issues in regard of mobile apps in the Android platform. These include malicious apps detection and security risk estimation of apps. Both direct sources of information from the developer of apps and indirect sources of information from user comments are utilized in these tasks. We also propose comparison of these different sources in the task of security risk estimation to point out the necessity of usage of indirect sources in mobile app security tasks

Crowdsourcing for Engineering Design: Objective Evaluations and Subjective Preferences

Crowdsourcing enables designers to reach out to large numbers of people who may not have been previously considered when designing a new product, listen to their input by aggregating their preferences and evaluations over potential designs, aiming to improve ``good'' and catch ``bad'' design decisions during the early-stage design process. This approach puts human designers--be they industrial designers, engineers, marketers, or executives--at the forefront, with computational crowdsourcing systems on the backend to aggregate subjective preferences (e.g., which next-generation Brand A design best competes stylistically with next-generation Brand B designs?) or objective evaluations (e.g., which military vehicle design has the best situational awareness?). These crowdsourcing aggregation systems are built using probabilistic approaches that account for the irrationality of human behavior (i.e., violations of reflexivity, symmetry, and transitivity), approximated by modern machine learning algorithms and optimization techniques as necessitated by the scale of data (millions of data points, hundreds of thousands of dimensions). This dissertation presents research findings suggesting the unsuitability of current off-the-shelf crowdsourcing aggregation algorithms for real engineering design tasks due to the sparsity of expertise in the crowd, and methods that mitigate this limitation by incorporating appropriate information for expertise prediction. Next, we introduce and interpret a number of new probabilistic models for crowdsourced design to provide large-scale preference prediction and full design space generation, building on statistical and machine learning techniques such as sampling methods, variational inference, and deep representation learning. Finally, we show how these models and algorithms can advance crowdsourcing systems by abstracting away the underlying appropriate yet unwieldy mathematics, to easier-to-use visual interfaces practical for engineering design companies and governmental agencies engaged in complex engineering systems design.PhDDesign ScienceUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/133438/1/aburnap_1.pd

Speaker-adaptive multimodal prediction model for listener responses

The goal of this paper is to analyze and model the variability in speaking styles in dyadic interactions and build a predictive algorithm for listener responses that is able to adapt to these different styles. The end result of this research will be a virtual human able to automatically respond to a human speaker with proper listener responses (e.g., head nods). Our novel speaker-adaptive prediction model is created from a corpus of dyadic interactions where speaker variability is analyzed to identify a subset of prototypical speaker styles. During a live interaction our prediction model automatically identifies the closest prototypical speaker style and predicts listener responses based on this ``communicative style". Central to our approach is the idea of ``speaker profile" which uniquely identifies each speaker and enables the matching between prototypical speakers and new speakers. The paper shows the merits of our speaker-adaptive listener response prediction model by showing improvement over a state-of-the-art approach which does not adapt to the speaker. Besides the merits of speaker-adapta-tion, our experiments highlights the importance of using multimodal features when comparing speakers to select the closest prototypical speaker style

Bayesian nonparametrics for crowdsourcing

Supervised machine learning relies on a labeled training set, whose size is closely related to the achievable performance of any learning algorithm. Thanks to the progresses in ubiquitous computing, networks, and data acquisition and storage technologies, the availability of data is no longer a problem. Nowadays, we can easily gather massive unlabeled datasets in a short period of time. Traditionally, the labeling was performed by a small set of experts so as to control the quality and the consistency of the annotations. When dealing with large datasets this approach is no longer feasible and the labeling process becomes the bottleneck. Crowdsourcing has been proven to be an effective and efficient tool to annotate large datasets. By distributing the labeling process across a potentially unlimited pool of annotators, it allows building large labeled datasets in a short period of time at a low cost. However, this comes at the expenses of a variable quality of the annotations, i.e. we need to deal with a large set of annotators of possibly unknown and variable expertise. In this new setting, methods to combine the annotations to produce reliable estimates of the ground truth are necessary. In this thesis, we tackle the problem of aggregating the information coming from a set of different annotators in a multi-class classification setting. We assume that no information about the expertise of the annotators or the ground truth of the instances is available. In particular, we focus on the potential advantages of using Bayesian Nonparametric models to build interpretable solutions for crowdsourcing applications. Bayesian Nonparametric models are Bayesian models which set a prior probability on an infinite-dimensional parameter space. After seeing a finite training sample, the posterior probability ends up using a finite number of parameters. Therefore, the complexity of the model depends on the training set and we can infer it from the data, avoiding the use of expensive model selection algorithms. We focus our efforts on two specific problems. Firstly, we claim that considering the existence of clusters of annotators in this aggregation step can improve the overall performance of the system. This is especially important in early stages of crowdsourcing implementations, when the number of annotations is low. At this stage there is not enough information to accurately estimate the bias introduced by each annotator separately, so we have to resort to models that consider the statistical links among them. In addition, finding these clusters is interesting in itself, as knowing the behavior of the pool of annotators allows implementing efficient active learning strategies. Based on this, we propose in two new fully unsupervised models based on a Chinese Restaurant Process prior and a hierarchical structure that allows inferring these groups jointly with the ground truth and the properties of the annotators. The second problem is modeling inconsistent annotators. The performance of the annotators can be in-homogeneous across the instance space due to several factors like his past experience with similar cases. To capture this behavior, we proposed an algorithm that uses a Dirichlet Process Mixture model to divide the instance space in different areas across which the annotators are consistent. The algorithm allows us to infer the characteristics of each annotator in each of the identified areas, the ground truth of the training set, as well as building a classifier for test examples. In addition, it offers an interpretable solution allowing to better understanding the decision process undertaken by the annotators, and implement schemes to improve the overall performance of the system. We propose efficient approximate inference algorithms based on Markov Chain Monte Carlo sampling and variational inference, using auxiliary variables to deal with non-conjugacies when needed. Finally, we perform experiments, both on synthetic and real databases, to show the advantages of our models over state-of-the-art algorithms.Todo aprendizaje máquina supervisado descansa sobre un conjunto de entrenamiento etiquetado cuyo tamaño muestral está directamente relacionado con el rendimiento final del algoritmo. Gracias a los avances en computación ubicua, redes y tecnologías de adquisición y almacenamiento de datos, la disponibilidad de datos con que entrenar estos algoritmos ha dejado de ser un problema. Actualmente, podemos fácilmente reunir enormes conjuntos de datos no etiquetados en cortos periodos de tiempo. Tradicionalmente, el etiquetado de estos datos, era realizado por un pequeño conjunto de expertos a fin de controlar la calidad final y la consistencia de las anotaciones. Cuando nos enfrentamos a grandes conjuntos de datos, esta forma de proceder deja de ser factible, convirtiéndose el etiquetado en un cuello de botella. Crowdsourcing ha probado ser una herramienta efectiva y eficiente para anotar grandes conjuntos de datos en aprendizaje máquina. Mediante la distribución del proceso de etiquetado a un, potencialmente ilimitado, conjunto de anotadores, permite construir grandes conjuntos de datos etiquetados en un corto periodo de tiempo y a un bajo coste. Sin embargo, todo esto tiene como precio una pérdida sobre el control de la calidad de las anotaciones. Nos enfrentamos ahora a un gran conjunto de anotadores cuya experiencia es variable y desconocida. En este nuevo escenario, métodos de combinación de las anotaciones para dar lugar a estimaciones fiables de la etiqueta verdadera son necesarios. En esta tesis, abordamos el problema de agregar la información procedente de diferentes anotadores en un problema de clasificación multi-clase. Asumimos que no existe información disponible acerca de la experiencia de los anotadores o la etiqueta verdadera de las muestras. En concreto, nos centramos en las ventajas potenciales de usar modelos bayesianos no paramétricos para construir soluciones interpretables para aplicaciones de crowdsourcing. Los modelos bayesianos no paramétricos son modelos Bayesianos que definen una probabilidad a priori sobre un espacio de parámetros con infinitas dimensiones. Tras observar una muestra de entrenamiento finita, la probabilidad a posteriori termina usando un número finito de parámetros. Por tanto, la complejidad del modelo depende del conjunto de entrenamiento usado que es inferida a partir de los datos, evitando el uso de costosos algoritmos para selección de modelos. Nos centramos en dos problemas específicos. En primer lugar, defendemos que tener en cuenta la existencia de grupos de anotadores en la etapa de agregación, puede mejorar el rendimiento global del sistema. Esto es especialmente importante en fases tempranas de la implementación del sistema de crowdsourcing, cuando el número de anotaciones en bajo. En esta fase no hay suficiente información para estimar con precisión el sesgo introducido por cada anotador por separado, por lo que tenemos que recurrir a modelos que tengan en cuenta las dependencias estadísticas entre los distintos anotadores. Además, encontrar estos grupos de anotadores es un problema interesante por sí mismo, pues el conocer el comportamiento de nuestros anotadores nos permite implementar estrategias eficientes de aprendizaje activo. Basándonos en esta hipótesis, proponemos dos nuevos modelos no supervisados haciendo uso de un prior Chinese Restaurant Process y una estructura jerárquica que nos permite inferir los grupos de anotadores así como sus propiedades y las etiquetas verdaderas. El segundo problema es el modelado de anotadores inconsistentes. El rendimiento de los anotadores puede ser no homogéneo en el espacio muestral debido a diferentes factores tales como sus experiencias pasadas con casos similares. Para capturar este comportamiento, proponemos un algoritmo que usa un modelo Dirichlet Process Mixture con el objetivo de dividir el espacio muestral en diferentes áreas en las cuales los anotadores son consistentes. El algoritmo nos permite inferir las características de cada anotador en cada una de las áreas identificadas, las etiquetas verdaderas de nuestras muestras de entrenamiento, así como construir un clasificador para futuras muestras. Además, ofrece una solución interpretable permitiendo una mejor comprensión del proceso de decisión adoptado por los anotadores, así como implementar estrategias para mejorar el rendimiento global del sistema. Proponemos algoritmos de inferencia aproximada eficientes basados en muestreo Markov Chain Monte Carlo e inferencia variacional, usando variables auxiliares para lidiar con modelos de observación no conjugados cuando así se requiera. Finalmente, realizamos experimentos con bases de datos sintéticas y reales a fin de mostrar las ventajas de nuestros modelos con respecto al estado del arte.This work was partially supported by the "Formación de Profesorado Universitario" fellowship from the Spanish Ministry of Education (FPU AP2009-1513).Programa Oficial de Doctorado en Multimedia y ComunicacionesPresidente: Joaquín Míguez Arenas.- Secretario: Alberto Suárez González.- Vocal: Finale Doshi-Vele

Detecting Political Framing Shifts and the Adversarial Phrases within\\ Rival Factions and Ranking Temporal Snapshot Contents in Social Media

abstract: Social Computing is an area of computer science concerned with dynamics of communities and cultures, created through computer-mediated social interaction. Various social media platforms, such as social network services and microblogging, enable users to come together and create social movements expressing their opinions on diverse sets of issues, events, complaints, grievances, and goals. Methods for monitoring and summarizing these types of sociopolitical trends, its leaders and followers, messages, and dynamics are needed. In this dissertation, a framework comprising of community and content-based computational methods is presented to provide insights for multilingual and noisy political social media content. First, a model is developed to predict the emergence of viral hashtag breakouts, using network features. Next, another model is developed to detect and compare individual and organizational accounts, by using a set of domain and language-independent features. The third model exposes contentious issues, driving reactionary dynamics between opposing camps. The fourth model develops community detection and visualization methods to reveal underlying dynamics and key messages that drive dynamics. The final model presents a use case methodology for detecting and monitoring foreign influence, wherein a state actor and news media under its control attempt to shift public opinion by framing information to support multiple adversarial narratives that facilitate their goals. In each case, a discussion of novel aspects and contributions of the models is presented, as well as quantitative and qualitative evaluations. An analysis of multiple conflict situations will be conducted, covering areas in the UK, Bangladesh, Libya and the Ukraine where adversarial framing lead to polarization, declines in social cohesion, social unrest, and even civil wars (e.g., Libya and the Ukraine).Dissertation/ThesisDoctoral Dissertation Computer Science 201