Spam elimination and bias correction : ensuring label quality in crowdsourced tasks.

Crowdsourcing is proposed as a powerful mechanism for accomplishing large scale tasks via anonymous workers online. It has been demonstrated as an effective and important approach for collecting labeled data in application domains which require human intelligence, such as image labeling, video annotation, natural language processing, etc. Despite the promises, one big challenge still exists in crowdsourcing systems: the difficulty of controlling the quality of crowds. The workers usually have diverse education levels, personal preferences, and motivations, leading to unknown work performance while completing a crowdsourced task. Among them, some are reliable, and some might provide noisy feedback. It is intrinsic to apply worker filtering approach to crowdsourcing applications, which recognizes and tackles noisy workers, in order to obtain high-quality labels. The presented work in this dissertation provides discussions in this area of research, and proposes efficient probabilistic based worker filtering models to distinguish varied types of poor quality workers. Most of the existing work in literature in the field of worker filtering either only concentrates on binary labeling tasks, or fails to separate the low quality workers whose label errors can be corrected from the other spam workers (with label errors which cannot be corrected). As such, we first propose a Spam Removing and De-biasing Framework (SRDF), to deal with the worker filtering procedure in labeling tasks with numerical label scales. The developed framework can detect spam workers and biased workers separately. The biased workers are defined as those who show tendencies of providing higher (or lower) labels than truths, and their errors are able to be corrected. To tackle the biasing problem, an iterative bias detection approach is introduced to recognize the biased workers. The spam filtering algorithm proposes to eliminate three types of spam workers, including random spammers who provide random labels, uniform spammers who give same labels for most of the items, and sloppy workers who offer low accuracy labels. Integrating the spam filtering and bias detection approaches into aggregating algorithms, which infer truths from labels obtained from crowds, can lead to high quality consensus results. The common characteristic of random spammers and uniform spammers is that they provide useless feedback without making efforts for a labeling task. Thus, it is not necessary to distinguish them separately. In addition, the removal of sloppy workers has great impact on the detection of biased workers, with the SRDF framework. To combat these problems, a different way of worker classification is presented in this dissertation. In particular, the biased workers are classified as a subcategory of sloppy workers. Finally, an ITerative Self Correcting - Truth Discovery (ITSC-TD) framework is then proposed, which can reliably recognize biased workers in ordinal labeling tasks, based on a probabilistic based bias detection model. ITSC-TD estimates true labels through applying an optimization based truth discovery method, which minimizes overall label errors by assigning different weights to workers. The typical tasks posted on popular crowdsourcing platforms, such as MTurk, are simple tasks, which are low in complexity, independent, and require little time to complete. Complex tasks, however, in many cases require the crowd workers to possess specialized skills in task domains. As a result, this type of task is more inclined to have the problem of poor quality of feedback from crowds, compared to simple tasks. As such, we propose a multiple views approach, for the purpose of obtaining high quality consensus labels in complex labeling tasks. In this approach, each view is defined as a labeling critique or rubric, which aims to guide the workers to become aware of the desirable work characteristics or goals. Combining the view labels results in the overall estimated labels for each item. The multiple views approach is developed under the hypothesis that workers\u27 performance might differ from one view to another. Varied weights are then assigned to different views for each worker. Additionally, the ITSC-TD framework is integrated into the multiple views model to achieve high quality estimated truths for each view. Next, we propose a Semi-supervised Worker Filtering (SWF) model to eliminate spam workers, who assign random labels for each item. The SWF approach conducts worker filtering with a limited set of gold truths available as priori. Each worker is associated with a spammer score, which is estimated via the developed semi-supervised model, and low quality workers are efficiently detected by comparing the spammer score with a predefined threshold value. The efficiency of all the developed frameworks and models are demonstrated on simulated and real-world data sets. By comparing the proposed frameworks to a set of state-of-art methodologies, such as expectation maximization based aggregating algorithm, GLAD and optimization based truth discovery approach, in the domain of crowdsourcing, up to 28.0% improvement can be obtained for the accuracy of true label estimation

Distributed Data Management in Vehicular Networks Using Mobile Agents

En los últimos años, las tecnologías de la información y las comunicaciones se han incorporado al mundo de la automoción gracias a sus avances, y han permitido la creación de dispositivos cada vez más pequeños y potentes. De esta forma, los vehículos pueden ahora incorporar por un precio asequible equipos informáticos y de comunicaciones.En este escenario, los vehículos que circulan por una determinada zona (como una ciudad o una autopista) pueden comunicarse entre ellos usando dispositivos inalámbricos que les permiten intercambiar información con otros vehículos cercanos, formando así una red vehicular ad hoc, o VANET (Vehicular Ad hoc Network). En este tipo de redes, las comunicaciones se establecen con conexiones punto a punto por medio de dispositivos tipo Wi-Fi, que permiten la comunicación con otros del mismo tipo dentro de su alcance, sin que sea necesaria la existencia previa de una infraestructura de comunicaciones como ocurre con las tecnologías de telefonía móvil (como 3G/4G), que además requieren de una suscripción y el pago de una tarifa para poder usarlas.Cada vehículo puede enviar información y recibirla de diversos orígenes, como el propio vehículo (por medio de los sensores que lleva incorporados), otros vehículos que se encuentran cerca, así como de la infraestructura de tráfico presente en las carreteras (como semáforos, señales, paneles electrónicos de información, cámaras de vigilancia, etc.). Todos estas fuentes pueden transmitir datos de diversa índole, como información de interés para los conductores (por ejemplo, atascos de tráfico o accidentes en la vía), o de cualquier otro tipo, mientras sea posible digitalizarla y enviarla a través de una red.Todos esos datos pueden ser almacenados localmente en los ordenadores que llevan los vehículos a medida que son recibidos, y sería muy interesante poder sacarles partido por medio de alguna aplicación que los explotara. Por ejemplo, podrían utilizarse los vehículos como plataformas móviles de sensores que obtengan datos de los lugares por los que viajan. Otro ejemplo de aplicación sería la de ayudar a encontrar plazas de aparcamiento libres en una zona de una ciudad, usando la información que suministrarían los vehículos que dejan una plaza libre.Con este fin, en esta tesis se ha desarrollado una propuesta de la gestión de datos basada en el uso de agentes móviles para poder hacer uso de la información presente en una VANET de forma eficiente y flexible. Esta no es una tarea trivial, ya que los datos se encuentran dispersos entre los vehículos que forman la red, y dichos vehículos están constantemente moviéndose y cambiando de posición. Esto hace que las conexiones de red establecidas entre ellos sean inestables y de corta duración, ya que están constantemente creándose y destruyéndose a medida que los vehículos entran y salen del alcance de sus comunicaciones debido a sus movimientos.En un escenario tan complicado, la aproximación que proponemos permite que los datos sean localizados, y que se puedan hacer consultas sobre ellos y transmitirlos de un sitio cualquiera de la VANET a otro, usando estrategias multi-salto que se adaptan a las siempre cambiantes posiciones de los vehículos. Esto es posible gracias a la utilización de agentes móviles para el procesamiento de datos, ya que cuentan con una serie de propiedades (como su movilidad, autonomía, adaptabilidad, o inteligencia), que hace que sean una elección muy apropiada para este tipo de entorno móvil y con un elevado grado de incertidumbre.La solución propuesta ha sido extensamente evaluada y probada por medio de simulaciones, que demuestran su buen rendimiento y fiabilidad en redes vehiculares con diferentes condiciones y en diversos escenarios.<br /

Principles for Designing Context-Aware Applications for Physical Activity Promotion

Mobile devices with embedded sensors have become commonplace, carried by billions of people worldwide. Their potential to influence positive health behaviors such as physical activity in people is just starting to be realized. Two critical ingredients, an accurate understanding of human behavior and use of that knowledge for building computational models, underpin all emerging behavior change applications. Early research prototypes suggest that such applications would facilitate people to make difficult decisions to manage their complex behaviors. However, the progress towards building real-world systems that support behavior change has been much slower than expected. The extreme diversity in real-world contextual conditions and user characteristics has prevented the conception of systems that scale and support end-users’ goals. We believe that solutions to the many challenges of designing context-aware systems for behavior change exist in three areas: building behavior models amenable to computational reasoning, designing better tools to improve our understanding of human behavior, and developing new applications that scale existing ways of achieving behavior change. With physical activity as its focus, this thesis addresses some crucial challenges that can move the field forward. Specifically, this thesis provides the notion of sweet spots, a phenomenological account of how people make and execute their physical activity plans. The key contribution of this concept is in its potential to improve the predictability of computational models supporting physical activity planning. To further improve our understanding of the dynamic nature of human behavior, we designed and built Heed, a low-cost, distributed and situated self-reporting device. Heed’s single-purpose and situated nature proved its use as the preferred device for self-reporting in many contexts. We finally present a crowdsourcing system that leverages expert knowledge to write personalized behavior change messages for large-scale context-aware applications.PHDInformationUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/144089/1/gparuthi_1.pd

Proximity as a Service via Cellular Network-Assisted Mobile Device-to-Device

PhD ThesisThe research progress of communication has brought a lot of novel technologies to meet the multi-dimensional demands such as pervasive connection, low delay and high bandwidth. Device-to-Device (D2D) communication is a way to no longer treat the User Equipment (UEs) as a terminal, but rather as a part of the network for service provisioning. This thesis decouples UEs into service providers (helpers) and service requesters. By collaboration among proximal devices, with the coordination of cellular networks, some local tasks can be achieved, such as coverage extension, computation o oading, mobile crowdsourcing and mobile crowdsensing. This thesis proposes a generic framework Proximity as a Service (PaaS) for increasing the coverage with demands of service continuity. As one of the use cases, the optimal helper selection algorithm of PaaS for increasing the service coverage with demands of service continuity is called ContAct based Proximity (CAP). Mainly, fruitful contact information (e.g., contact duration, frequency, and interval) is captured, and is used to handle ubiquitous proximal services through the optimal selection of helpers. The nature of PaaS is evaluated under the Helsinki city scenario, with movement model of Points Of Interest (POI) and with critical factors in uencing the service demands (e.g., success ratio, disruption duration and frequency). Simulation results show the advantage of CAP, in both success ratio and continuity of the service (outputs). Based on this perspective, metrics such as service success ratio and continuity as a service evaluation of the PaaS are evaluated using the statistical theory of the Design Of Experiments (DOE). DOE is used as there are many dimensions to the state space (access tolerance, selected helper number, helper access limit, and transmit range) that can in uence the results. A key contribution of this work is that it brings rigorous statistical experiment design methods into the research into mobile computing. Results further reveal the influence of four factors (inputs), e.g., service tolerance, number of helpers allocated, the number of concurrent devices supported by each helper and transmit range. Based on this perspective, metrics such as service success ratio and continuity are evaluated using DOE. The results show that transmit range is the most dominant factor. The number of selected helpers is the second most dominant factor. Since di erent factors have di erent regression levels, a uni ed 4 level full factorial experiment and a cubic multiple regression analysis have been carried out. All the interactions and the corresponding coe cients have been found. This work is the rst one to evaluate LTE-Direct and WiFi-Direct in an opportunistic proximity service. The contribution of the results for industry is to guide how many users need to cooperate to enable mobile computing and for academia. This reveals the facts that: 1, in some cases, the improvement of spectrum e ciency brought by D2D is not important; 2, nodal density and the resources used in D2D air-interfaces are important in the eld of mobile computing. This work built a methodology to study the D2D networks with a di erent perspective (PaaS)

Proceedings of the 2nd Conference on Production Systems and Logistics (CPSL 2021)

Proceedings of the CPSL 202

Foundations of Human-Aware Planning -- A Tale of Three Models

abstract: A critical challenge in the design of AI systems that operate with humans in the loop is to be able to model the intentions and capabilities of the humans, as well as their beliefs and expectations of the AI system itself. This allows the AI system to be "human- aware" -- i.e. the human task model enables it to envisage desired roles of the human in joint action, while the human mental model allows it to anticipate how its own actions are perceived from the point of view of the human. In my research, I explore how these concepts of human-awareness manifest themselves in the scope of planning or sequential decision making with humans in the loop. To this end, I will show (1) how the AI agent can leverage the human task model to generate symbiotic behavior; and (2) how the introduction of the human mental model in the deliberative process of the AI agent allows it to generate explanations for a plan or resort to explicable plans when explanations are not desired. The latter is in addition to traditional notions of human-aware planning which typically use the human task model alone and thus enables a new suite of capabilities of a human-aware AI agent. Finally, I will explore how the AI agent can leverage emerging mixed-reality interfaces to realize effective channels of communication with the human in the loop.Dissertation/ThesisDoctoral Dissertation Computer Science 201