An introduction to crowdsourcing for language and multimedia technology research

Language and multimedia technology research often relies on large manually constructed datasets for training or evaluation of algorithms and systems. Constructing these datasets is often expensive with significant challenges in terms of recruitment of personnel to carry out the work. Crowdsourcing methods using scalable pools of workers available on-demand offers a flexible means of rapid low-cost construction of many of these datasets to support existing research requirements and potentially promote new research initiatives that would otherwise not be possible

Citizen Social Lab: A digital platform for human behaviour experimentation within a citizen science framework

Cooperation is one of the behavioral traits that define human beings, however we are still trying to understand why humans cooperate. Behavioral experiments have been largely conducted to shed light into the mechanisms behind cooperation and other behavioral traits. However, most of these experiments have been conducted in laboratories with highly controlled experimental protocols but with varied limitations which limits the reproducibility and the generalization of the results obtained. In an attempt to overcome these limitations, some experimental approaches have moved human behavior experimentation from laboratories to public spaces, where behaviors occur naturally, and have opened the participation to the general public within the citizen science framework. Given the open nature of these environments, it is critical to establish the appropriate protocols to maintain the same data quality that one can obtain in the laboratories. Here, we introduce Citizen Social Lab, a software platform designed to be used in the wild using citizen science practices. The platform allows researchers to collect data in a more realistic context while maintaining the scientific rigour, and it is structured in a modular and scalable way so it can also be easily adapted for online or brick-and-mortar experimental laboratories. Following citizen science guidelines, the platform is designed to motivate a more general population into participation, but also to promote engaging and learning of the scientific research process. We also review the main results of the experiments performed using the platform up to now, and the set of games that each experiment includes. Finally, we evaluate some properties of the platform, such as the heterogeneity of the samples of the experiments and their satisfaction level, and the parameters that demonstrate the robustness of the platform and the quality of the data collected.Comment: 17 pages, 11 figures and 4 table

The “self-bad, partner-worse” strategy inhibits cooperation in networked populations

The emergence and maintenance of cooperation is a popular topic in studies of information sciences and evolutionary game theory. In two-player iterated games, memory in terms of the outcome of previous interactions and the strategy choices of co-players are of great referential significance for subsequent strategy actions. It is generally recognized that there is no single simple and overarching strategy whereby one player X can unilaterally achieve a higher payoff than his opponent Y, irrespective of Y's strategy and response. In this paper, we demonstrate that such strategies do exist in diverse networked populations. More precisely, (i) such strategies can obtain a low payoff for the focal player, however, they also lead to an even lower payoff for that player's partner, in turn lowering benefits of the overall populations; (ii) they are capable of winning with a high probability against opponents with an unknown strategy; and (iii) they have a survival advantage and robust fitness in terms of evolutionary processes. We refer to these as the “self-bad, partner-worse” (SBPW) strategies. Results presented here add to previous studies on strategy evolution in the context of social dilemmas and hint at some insights concerning cooperation promotion mechanisms among networked populations

MMZDA: Enabling Social Welfare Maximization in Cross-Silo Federated Learning

—As one of the typical settings of Federated Learning (FL), cross-silo FL allows organizations to jointly train an optimal Machine Learning (ML) model. In this case, some organizations may try to obtain the global model without contributing their local training, lowering the social welfare. In this paper, we model the interactions among organizations in cross-silo FL as a public goods game for the first time and theoretically prove that there exists a social dilemma where the maximum social welfare is not achieved in Nash equilibrium. To overcome this social dilemma, we employ the Multi-player Multi-action ZeroDeterminant (MMZD) strategy to maximize the social welfare. With the help of the MMZD, an individual organization can unilaterally control the social welfare without extra cost. Since the MMZDstrategy can be adopted by all organizations, we further study the scenario where multiple organizations jointly adopt the MMZD strategy and form an MMZD Alliance (MMZDA). We prove theoretically that the MMZDA strategy strengthens the control of the maximum social welfare. Experimental results validate that the MMZD strategy is effective in maximizing the social welfare and the MMZDA can achieve a larger maximum value