Assuring privacy and reliability in crowdsourcing with coding

Abstract—Crowd workers are often unreliable and anonymous. Hence there is a need to ensure reliable work delivery while preserving some level of privacy to the requester’s data. For this purpose, we use a combination of random perturbation to mask the sensitive data and error-correcting codes for quality assurance. We also consider the possibility of collusion attacks by malicious crowd workers. We develop mathematical models to study the precise tradeoffs between task performance quality, level of privacy against collusion attacks, and cost of invoking a large crowd. Such a study provides design strategies and principles for crowd work. The use of classification codes may improve efficiency considerably. We also comment on the applicability of these techniques for scalable assessment in education via peer grading, e.g. for massive open online courses (MOOCs). I swear, by Galois and all the gods, that I will be true to the noble traditions of coding theory; that I will speak of it solely in the secret language of higher algebra known only to my fellow initiates; and above all, that I will vigilantly guard our sacred theory from those who would profane it by practical applications. — James L. Massey (1971) Today coding theory is badly profaned and most coding theorists are contaminated by practical applications. This is because practical application of error-correcting codes have been rapidly spreading all over the area of digital communi-cation and storage systems. Mainly this is due to the advance of semiconductor technology. — Hideki Imai (1991) [1] I

Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?

We explore the design of an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final classification decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. Additionally, the workers report quantized confidence levels when they are able to submit definitive answers. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance. We obtain a couterintuitive result that the classification performance does not benefit from workers reporting quantized confidence. Therefore, the crowdsourcing system designer should employ the reject option without requiring confidence reporting.Comment: 6 pages, 4 figures, SocialSens 2017. arXiv admin note: text overlap with arXiv:1602.0057

A European research roadmap for optimizing societal impact of big data on environment and energy efficiency

We present a roadmap to guide European research efforts towards a socially responsible big data economy that maximizes the positive impact of big data in environment and energy efficiency. The goal of the roadmap is to allow stakeholders and the big data community to identify and meet big data challenges, and to proceed with a shared understanding of the societal impact, positive and negative externalities, and concrete problems worth investigating. It builds upon a case study focused on the impact of big data practices in the context of Earth Observation that reveals both positive and negative effects in the areas of economy, society and ethics, legal frameworks and political issues. The roadmap identifies European technical and non-technical priorities in research and innovation to be addressed in the upcoming five years in order to deliver societal impact, develop skills and contribute to standardization.Comment: 6 pages, 2 figures, 1 tabl

Reliable Inference from Unreliable Agents

Distributed inference using multiple sensors has been an active area of research since the emergence of wireless sensor networks (WSNs). Several researchers have addressed the design issues to ensure optimal inference performance in such networks. The central goal of this thesis is to analyze distributed inference systems with potentially unreliable components and design strategies to ensure reliable inference in such systems. The inference process can be that of detection or estimation or classification, and the components/agents in the system can be sensors and/or humans. The system components can be unreliable due to a variety of reasons: faulty sensors, security attacks causing sensors to send falsified information, or unskilled human workers sending imperfect information. This thesis first quantifies the effect of such unreliable agents on the inference performance of the network and then designs schemes that ensure a reliable overall inference. In the first part of this thesis, we study the case when only sensors are present in the system, referred to as sensor networks. For sensor networks, the presence of malicious sensors, referred to as Byzantines, are considered. Byzantines are sensors that inject false information into the system. In such systems, the effect of Byzantines on the overall inference performance is characterized in terms of the optimal attack strategies. Game-theoretic formulations are explored to analyze two-player interactions. Next, Byzantine mitigation schemes are designed that address the problem from the system\u27s perspective. These mitigation schemes are of two kinds: Byzantine identification schemes and Byzantine tolerant schemes. Using learning based techniques, Byzantine identification schemes are designed that learn the identity of Byzantines in the network and use this information to improve system performance. When such schemes are not possible, Byzantine tolerant schemes using error-correcting codes are developed that tolerate the effect of Byzantines and maintain good performance in the network. Error-correcting codes help in correcting the erroneous information from these Byzantines and thereby counter their attack. The second line of research in this thesis considers humans-only networks, referred to as human networks. A similar research strategy is adopted for human networks where, the effect of unskilled humans sharing beliefs with a central observer called \emph{CEO} is analyzed, and the loss in performance due to the presence of such unskilled humans is characterized. This problem falls under the family of problems in information theory literature referred to as the \emph{CEO Problem}, but for belief sharing. The asymptotic behavior of the minimum achievable mean squared error distortion at the CEO is studied in the limit when the number of agents $L$ and the sum rate $R$ tend to infinity. An intermediate regime of performance between the exponential behavior in discrete CEO problems and the $1/R$ behavior in Gaussian CEO problems is established. This result can be summarized as the fact that sharing beliefs (uniform) is fundamentally easier in terms of convergence rate than sharing measurements (Gaussian), but sharing decisions is even easier (discrete). Besides theoretical analysis, experimental results are reported for experiments designed in collaboration with cognitive psychologists to understand the behavior of humans in the network. The act of fusing decisions from multiple agents is observed for humans and the behavior is statistically modeled using hierarchical Bayesian models. The implications of such modeling on the design of large human-machine systems is discussed. Furthermore, an error-correcting codes based scheme is proposed to improve system performance in the presence of unreliable humans in the inference process. For a crowdsourcing system consisting of unskilled human workers providing unreliable responses, the scheme helps in designing easy-to-perform tasks and also mitigates the effect of erroneous data. The benefits of using the proposed approach in comparison to the majority voting based approach are highlighted using simulated and real datasets. In the final part of the thesis, a human-machine inference framework is developed where humans and machines interact to perform complex tasks in a faster and more efficient manner. A mathematical framework is built to understand the benefits of human-machine collaboration. Such a study is extremely important for current scenarios where humans and machines are constantly interacting with each other to perform even the simplest of tasks. While machines perform best in some tasks, humans still give better results in tasks such as identifying new patterns. By using humans and machines together, one can extract complete information about a phenomenon of interest. Such an architecture, referred to as Human-Machine Inference Networks (HuMaINs), provides promising results for the two cases of human-machine collaboration: \emph{machine as a coach} and \emph{machine as a colleague}. For simple systems, we demonstrate tangible performance gains by such a collaboration which provides design modules for larger, and more complex human-machine systems. However, the details of such larger systems needs to be further explored

Succeeding with Smart People Initiatives: Difficulties and Preconditions for Smart City Initiatives that Target Citizens

Smart City is a paradigm for the development of urban spaces through the implementation of state-of-the-art ICT. There are two main approaches when developing Smart Cities: top-down and bottom-up. Based on the bottom-up approach, the concepts of Smart People and Smart Communities have emerged as dimensions of the Smart City, advocating for the engagement of citizens in Smart People initiatives. The aim of this research is both to find the types of Smart People initiatives and to identify their difficulties and preconditions for success. However, such initiatives that aim to (1) leverage the citizens intellectually and (2) use citizens as a source of input for ideas and innovation, are understudied. Therefore, this research proposes a concentrated framework of Smart People initiatives from an extensive literature review. On one hand, this framework contributes with a common ground and vocabulary that facilitates the dialogue within and between practitioners and academia. On the other hand, the identification of difficulties and preconditions guides the academia and practitioners in how to successfully account for citizens in the Smart City. From the literature review and the conduct of case studies of five European cities, participation came out as the key difficulty across both types of Smart People initiatives and cases, closely followed by awareness, motivation and complexity

A Value-sensitive Design Perspective of Cryptocurrencies: A Research Agenda

Cryptocurrencies and their underlying blockchain technology have begun to transform numerous industries. Although we have seen an uptrend in the types of created cryptocurrencies, it has not yet translated into mainstream adoption., In this paper, we use value-sensitive design principles to identify values among current and potential cryptocurrency adopters. Using Bitcoin as the context for this qualitative research study, we use grounded theory analytical techniques to discover manifested values among users and non-users. We develop a cryptocurrency value-sensitive design framework to summarize our results. As our main contribution, we offer a research agenda based on the cryptocurrency stakeholders’ underlying value system. This agenda can help information systems scholars apply this value-sensitive design perspective to their own cryptocurrency research

Crowdsourcing Strategizing: A View From the Top

Crowdsourcing strategizing is the application of crowdsourcing for organizational strategy development. While crowdsourcing is experiencing popularity in application and discussion, the concept is not new. However, literature on the value of crowdsourcing strategizing is not widespread in academic or business works. This qualitative case study explored crowdsourcing strategizing in Richmond, Virginia metro area nonprofits. The study was conducted to explore the lack of understanding on the value of crowdsourcing strategizing, with a focus on leaderships perspective of value. The results showed that nonprofit leaders found value in the crowdsourced data gathered through crowdsourcing strategizing