2,204 research outputs found
Peer Prediction for Peer Review: Designing a Marketplace for Ideas
The paper describes a potential platform to facilitate academic peer review
with emphasis on early-stage research. This platform aims to make peer review
more accurate and timely by rewarding reviewers on the basis of peer prediction
algorithms. The algorithm uses a variation of Peer Truth Serum for
Crowdsourcing (Radanovic et al., 2016) with human raters competing against a
machine learning benchmark. We explain how our approach addresses two large
productive inefficiencies in science: mismatch between research questions and
publication bias. Better peer review for early research creates additional
incentives for sharing it, which simplifies matching ideas to teams and makes
negative results and p-hacking more visible
PoFEL: Energy-efficient Consensus for Blockchain-based Hierarchical Federated Learning
Facilitated by mobile edge computing, client-edge-cloud hierarchical
federated learning (HFL) enables communication-efficient model training in a
widespread area but also incurs additional security and privacy challenges from
intermediate model aggregations and remains the single point of failure issue.
To tackle these challenges, we propose a blockchain-based HFL (BHFL) system
that operates a permissioned blockchain among edge servers for model
aggregation without the need for a centralized cloud server. The employment of
blockchain, however, introduces additional overhead. To enable a compact and
efficient workflow, we design a novel lightweight consensus algorithm, named
Proof of Federated Edge Learning (PoFEL), to recycle the energy consumed for
local model training. Specifically, the leader node is selected by evaluating
the intermediate FEL models from all edge servers instead of other
energy-wasting but meaningless calculations. This design thus improves the
system efficiency compared with traditional BHFL frameworks. To prevent model
plagiarism and bribery voting during the consensus process, we propose
Hash-based Commitment and Digital Signature (HCDS) and Bayesian Truth
Serum-based Voting (BTSV) schemes. Finally, we devise an incentive mechanism to
motivate continuous contributions from clients to the learning task.
Experimental results demonstrate that our proposed BHFL system with the
corresponding consensus protocol and incentive mechanism achieves
effectiveness, low computational cost, and fairness
Bayesian markets to elicit private information
Financial markets reveal what investors think about the future, and prediction markets are used to forecast election results. Could markets also encourage people to reveal private information, such as subjective judgments (e.g., “Are you satisfied with your life?”) or unverifiable facts? This paper shows how to design such markets, called Bayesian markets. People trade an asset whose value represents the proportion of affirmative answers to a question. Their trading position then reveals their own answer to the question. The results of this paper are based on a Bayesian setup in which people use their private information (their “type”) as a signal. Hence, beliefs about others’ types are correlated with one’s own type. Bayes
Experimental philosophy and the incentivisation challenge : a proposed application of the Bayesian Truth Serum
A key challenge in experimental social science research is the incentivisation of subjects such that they take the tasks presented to them seriously and answer honestly. If subject responses can be evaluated against an objective baseline, a standard way of incentivising participants is by rewarding them monetarily as a function of their performance. However, the subject area of experimental philosophy is such that this mode of incentivisation is not applicable as participant responses cannot easily be scored along a true-false spectrum by the experimenters. We claim that experimental philosophers’ neglect of and claims of unimportance about incentivisation mechanisms in their surveys and experiments has plausibly led to poorer data quality and worse conclusions drawn overall, potentially threatening the research programme of experimental philosophy in the long run. As a solution to this, we propose the adoption of the Bayesian Truth Serum, an incentive-compatible mechanism used in economics and marketing, designed for eliciting honest responding in subjective data designs by rewarding participant answers that are surprisingly common. We argue that the Bayesian Truth Serum (i) adequately addresses the issue of incentive compatibility in subjective data research designs and (ii) that it should be applied to the vast majority of research in experimental philosophy. Further, we (iii) provide an empirical application of the method, demonstrating its qualified impact on the distribution of answers on a number of standard experimental philosophy items and outline guidance for researchers aiming to apply this mechanism in future research by specifying the additional costs and design steps involved.Publisher PDFPeer reviewe
- …