Crowdsourcing Relevance: Two Studies on Assessment

Crowdsourcing has become an alternative approach to collect relevance judgments at large scale. In this thesis, we focus on some specific aspects related to time, scale, and agreement. First, we address the issue of the time factor in gathering relevance label: we study how much time the judges need to assess documents. We conduct a series of four experiments which unexpectedly reveal us how introducing time limitations leads to benefits in terms of the quality of the results. Furthermore, we discuss strategies aimed to determine the right amount of time to make available to the workers for the relevance assessment, in order to both guarantee the high quality of the gathered results and the saving of the valuable resources of time and money. Then we explore the application of magnitude estimation, a psychophysical scaling technique for the measurement of sensation, for relevance assessment. We conduct a large-scale user study across 18 TREC topics, collecting more than 50,000 magnitude estimation judgments, which result to be overall rank-aligned with ordinal judgments made by expert relevance assessors. We discuss the benefits, the reliability of the judgements collected, and the competitiveness in terms of assessor cost. We also report some preliminary results on the agreement among judges. Often, the results of crowdsourcing experiments are affected by noise, that can be ascribed to lack of agreement among workers. This aspect should be considered as it can affect the reliability of the gathered relevance labels, as well as the overall repeatability of the experiments.openDottorato di ricerca in Informatica e scienze matematiche e fisicheopenMaddalena, Edd

Multidimensional news quality: A comparison of crowdsourcing and nichesourcing

In the age of fake news and of filter bubbles, assessing the quality of information is a compelling issue: it is important for users to understand the quality of the information they consume online. We report on our experiment aimed at understanding if workers from the crowd can be a suitable alternative to

Fact-Checking at Scale with Crowdsourcing: Experiments and Lessons Learned

In this paper, we present our journey in exploring the use of crowdsourcing for fact-checking. We discuss our early experiments aimed towards the identification of the best possible setting for misinformation assessment using crowdsourcing. Our results indicate that the crowd can effectively address misinformation at scale, showing some degree of correlation with experts. We also highlight the influence of worker background on the quality of truthfulness assessments

Preliminary results from a crowdsourcing experiment in immunohistochemistry

Background: Crowdsourcing, i.e., the outsourcing of tasks typically performed by a few experts to a large crowd as an open call, has been shown to be reasonably effective in many cases, like Wikipedia, the Chess match of Kasparov against the world in 1999, and several others. The aim of the present paper is to describe the setup of an experimentation of crowdsourcing techniques applied to the quantification of immunohistochemistry. Methods: Fourteen Images from MIB1-stained breast specimens were first manually counted by a pathologist, then submitted to a crowdsourcing platform through a specifically developed application. 10 positivity evaluations for each image have been collected and summarized using their median. The positivity values have been then compared to the gold standard provided by the pathologist by means of Spearman correlation. Results: Contributors were in total 28, and evaluated 4.64 images each on average. Spearman correlation between gold and crowdsourced positivity percentages is 0.946 (p < 0.001). Conclusions: Aim of the experiment was to understand how to use crowdsourcing for an image analysis task that is currently time-consuming when done by human experts. Crowdsourced work can be used in various ways, in particular statistically agregating data to reduce identification errors. However, in this preliminary experimentation we just considered the most basic indicator, that is the median positivity percentage, which provided overall good results. This method might be more aimed to research than routine: when a large number of images are in need of ad-hoc evaluation, crowdsourcing may represent a quick answer to the need. \ua9 Della Mea et al

Observation of the B0 → ρ0ρ0 decay from an amplitude analysis of B0 → (π+π−)(π+π−) decays

Proton–proton collision data recorded in 2011 and 2012 by the LHCb experiment, corresponding to an integrated luminosity of 3.0 fb−1 , are analysed to search for the charmless B0→ρ0ρ0 decay. More than 600 B0→(π+π−)(π+π−) signal decays are selected and used to perform an amplitude analysis, under the assumption of no CP violation in the decay, from which the B0→ρ0ρ0 decay is observed for the first time with 7.1 standard deviations significance. The fraction of B0→ρ0ρ0 decays yielding a longitudinally polarised final state is measured to be fL=0.745−0.058+0.048(stat)±0.034(syst) . The B0→ρ0ρ0 branching fraction, using the B0→ϕK⁎(892)0 decay as reference, is also reported as B(B0→ρ0ρ0)=(0.94±0.17(stat)±0.09(syst)±0.06(BF))×10−6

Measurement of the (eta c)(1S) production cross-section in proton-proton collisions via the decay (eta c)(1S) -&gt; p(p)over-bar

The production of the $\eta_c (1S)$ state in proton-proton collisions is probed via its decay to the $p \bar{p}$ final state with the LHCb detector, in the rapidity range $2.0 6.5$ GeV/c. The cross-section for prompt production of $\eta_c (1S)$ mesons relative to the prompt $J/\psi$ cross-section is measured, for the first time, to be $\sigma_{\eta_c (1S)}/\sigma_{J/\psi} = 1.74 \pm 0.29 \pm 0.28 \pm 0.18 _{B}$ at a centre-of-mass energy $\sqrt{s} = 7$ TeV using data corresponding to an integrated luminosity of 0.7 fb$^{-1}$, and $\sigma_{\eta_c (1S)}/\sigma_{J/\psi} = 1.60 \pm 0.29 \pm 0.25 \pm 0.17 _{B}$ at $\sqrt{s} = 8$ TeV using 2.0 fb$^{-1}$. The uncertainties quoted are, in order, statistical, systematic, and that on the ratio of branching fractions of the $\eta_c (1S)$ and $J/\psi$ decays to the $p \bar{p}$ final state. In addition, the inclusive branching fraction of $b$-hadron decays into $\eta_c (1S)$ mesons is measured, for the first time, to be $B ( b \rightarrow \eta_c X ) = (4.88 \pm 0.64 \pm 0.25 \pm 0.67 _{B}) \times 10^{-3}$, where the third uncertainty includes also the uncertainty on the $J/\psi$ inclusive branching fraction from $b$-hadron decays. The difference between the $J/\psi$ and $\eta_c (1S)$ meson masses is determined to be $114.7 \pm 1.5 \pm 0.1$ MeV/c$^2$.The production of the $\eta _c (1S)$ state in proton-proton collisions is probed via its decay to the $p\overline{p}$ final state with the LHCb detector, in the rapidity range $2.0 6.5 \mathrm{{\,GeV/}{ c}}$ . The cross-section for prompt production of $\eta _c (1S)$ mesons relative to the prompt ${{ J}}/{\psi }$ cross-section is measured, for the first time, to be $\sigma _{\eta _c (1S)}/\sigma _{{{{ J}}/{\psi }}} = 1.74\, \pm \,0.29\, \pm \, 0.28\, \pm \,0.18 _{{\mathcal{B}}}$ at a centre-of-mass energy ${\sqrt{s}} = 7 {~\mathrm{TeV}}$ using data corresponding to an integrated luminosity of 0.7 fb$^{-1}$ , and $\sigma _{\eta _c (1S)}/\sigma _{{{{ J}}/{\psi }}} = 1.60 \pm 0.29 \pm 0.25 \pm 0.17 _{{\mathcal{B}}}$ at ${\sqrt{s}} = 8 {~\mathrm{TeV}}$ using 2.0 fb$^{-1}$ . The uncertainties quoted are, in order, statistical, systematic, and that on the ratio of branching fractions of the $\eta _c (1S)$ and ${{ J}}/{\psi }$ decays to the $p\overline{p}$ final state. In addition, the inclusive branching fraction of ${b}$ -hadron decays into $\eta _c (1S)$ mesons is measured, for the first time, to be ${\mathcal{B}}( b {\rightarrow } \eta _c X ) = (4.88\, \pm \,0.64\, \pm \,0.29\, \pm \, 0.67 _{{\mathcal{B}}}) \times 10^{-3}$ , where the third uncertainty includes also the uncertainty on the ${{ J}}/{\psi }$ inclusive branching fraction from ${b}$ -hadron decays. The difference between the ${{ J}}/{\psi }$ and $\eta _c (1S)$ meson masses is determined to be $114.7 \pm 1.5 \pm 0.1 {\mathrm {\,MeV\!/}c^2}$ .The production of the $\eta_c (1S)$ state in proton-proton collisions is probed via its decay to the $p \bar{p}$ final state with the LHCb detector, in the rapidity range $2.0 6.5$ GeV/c. The cross-section for prompt production of $\eta_c (1S)$ mesons relative to the prompt $J/\psi$ cross-section is measured, for the first time, to be $\sigma_{\eta_c (1S)}/\sigma_{J/\psi} = 1.74 \pm 0.29 \pm 0.28 \pm 0.18 _{B}$ at a centre-of-mass energy $\sqrt{s} = 7$ TeV using data corresponding to an integrated luminosity of 0.7 fb$^{-1}$, and $\sigma_{\eta_c (1S)}/\sigma_{J/\psi} = 1.60 \pm 0.29 \pm 0.25 \pm 0.17 _{B}$ at $\sqrt{s} = 8$ TeV using 2.0 fb$^{-1}$. The uncertainties quoted are, in order, statistical, systematic, and that on the ratio of branching fractions of the $\eta_c (1S)$ and $J/\psi$ decays to the $p \bar{p}$ final state. In addition, the inclusive branching fraction of $b$-hadron decays into $\eta_c (1S)$ mesons is measured, for the first time, to be $B ( b \rightarrow \eta_c X ) = (4.88 \pm 0.64 \pm 0.29 \pm 0.67 _{B}) \times 10^{-3}$, where the third uncertainty includes also the uncertainty on the $J/\psi$ inclusive branching fraction from $b$-hadron decays. The difference between the $J/\psi$ and $\eta_c (1S)$ meson masses is determined to be $114.7 \pm 1.5 \pm 0.1$ MeV/c$^2$

A study of CP violation in B-+/- -&gt; DK +/- and B-+/- -&gt; D pi(+/-) decays with D -&gt; (KSK +/-)-K-0 pi(-/+) final states

A first study of CP violation in the decay modes $B^\pm\to [K^0_{\rm S} K^\pm \pi^\mp]_D h^\pm$ and $B^\pm\to [K^0_{\rm S} K^\mp \pi^\pm]_D h^\pm$, where $h$ labels a $K$ or $\pi$ meson and $D$ labels a $D^0$ or $\overline{D}^0$ meson, is performed. The analysis uses the LHCb data set collected in $pp$ collisions, corresponding to an integrated luminosity of 3 fb$^{-1}$. The analysis is sensitive to the CP-violating CKM phase $\gamma$ through seven observables: one charge asymmetry in each of the four modes and three ratios of the charge-integrated yields. The results are consistent with measurements of $\gamma$ using other decay modes

Study of the rare B-s(0) and B-0 decays into the pi(+) pi(-) mu(+) mu(-) final state

A search for the rare decays $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and $B^0 \to \pi^+\pi^-\mu^+\mu^-$ is performed in a data set corresponding to an integrated luminosity of 3.0 fb$^{-1}$ collected by the LHCb detector in proton-proton collisions at centre-of-mass energies of 7 and 8 TeV. Decay candidates with pion pairs that have invariant mass in the range 0.5-1.3 GeV/$c^2$ and with muon pairs that do not originate from a resonance are considered. The first observation of the decay $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and the first evidence of the decay $B^0 \to \pi^+\pi^-\mu^+\mu^-$ are obtained and the branching fractions are measured to be $\mathcal{B}(B_s^0 \to \pi^+\pi^-\mu^+\mu^-)=(8.6\pm 1.5\,({\rm stat}) \pm 0.7\,({\rm syst})\pm 0.7\,({\rm norm}))\times 10^{-8}$ and $\mathcal{B}(B^0 \to \pi^+\pi^-\mu^+\mu^-)=(2.11\pm 0.51\,({\rm stat}) \pm 0.15\,({\rm syst})\pm 0.16\,({\rm norm}) )\times 10^{-8}$, where the third uncertainty is due to the branching fraction of the decay $B^0\to J/\psi(\to \mu^+\mu^-)K^*(890)^0(\to K^+\pi^-)$, used as a normalisation.A search for the rare decays Bs0→π+π−μ+μ− and B0→π+π−μ+μ− is performed in a data set corresponding to an integrated luminosity of 3.0 fb−1 collected by the LHCb detector in proton–proton collisions at centre-of-mass energies of 7 and 8 TeV . Decay candidates with pion pairs that have invariant mass in the range 0.5–1.3 GeV/c2 and with muon pairs that do not originate from a resonance are considered. The first observation of the decay Bs0→π+π−μ+μ− and the first evidence of the decay B0→π+π−μ+μ− are obtained and the branching fractions, restricted to the dipion-mass range considered, are measured to be B(Bs0→π+π−μ+μ−)=(8.6±1.5 (stat)±0.7 (syst)±0.7(norm))×10−8 and B(B0→π+π−μ+μ−)=(2.11±0.51(stat)±0.15(syst)±0.16(norm))×10−8 , where the third uncertainty is due to the branching fraction of the decay B0→J/ψ(→μ+μ−)K⁎(892)0(→K+π−) , used as a normalisation.A search for the rare decays Bs0→π+π−μ+μ− and B0→π+π−μ+μ− is performed in a data set corresponding to an integrated luminosity of 3.0 fb−1 collected by the LHCb detector in proton–proton collisions at centre-of-mass energies of 7 and 8 TeV . Decay candidates with pion pairs that have invariant mass in the range 0.5–1.3 GeV/c2 and with muon pairs that do not originate from a resonance are considered. The first observation of the decay Bs0→π+π−μ+μ− and the first evidence of the decay B0→π+π−μ+μ− are obtained and the branching fractions, restricted to the dipion-mass range considered, are measured to be B(Bs0→π+π−μ+μ−)=(8.6±1.5 (stat)±0.7 (syst)±0.7(norm))×10−8 and B(B0→π+π−μ+μ−)=(2.11±0.51(stat)±0.15(syst)±0.16(norm))×10−8 , where the third uncertainty is due to the branching fraction of the decay B0→J/ψ(→μ+μ−)K⁎(892)0(→K+π−) , used as a normalisation.A search for the rare decays $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and $B^0 \to \pi^+\pi^-\mu^+\mu^-$ is performed in a data set corresponding to an integrated luminosity of 3.0 fb$^{-1}$ collected by the LHCb detector in proton-proton collisions at centre-of-mass energies of 7 and 8 TeV. Decay candidates with pion pairs that have invariant mass in the range 0.5-1.3 GeV/$c^2$ and with muon pairs that do not originate from a resonance are considered. The first observation of the decay $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and the first evidence of the decay $B^0 \to \pi^+\pi^-\mu^+\mu^-$ are obtained and the branching fractions, restricted to the dipion-mass range considered, are measured to be $\mathcal{B}(B_s^0 \to \pi^+\pi^-\mu^+\mu^-)=(8.6\pm 1.5\,({\rm stat}) \pm 0.7\,({\rm syst})\pm 0.7\,({\rm norm}))\times 10^{-8}$ and $\mathcal{B}(B^0 \to \pi^+\pi^-\mu^+\mu^-)=(2.11\pm 0.51\,({\rm stat}) \pm 0.15\,({\rm syst})\pm 0.16\,({\rm norm}) )\times 10^{-8}$, where the third uncertainty is due to the branching fraction of the decay $B^0\to J/\psi(\to \mu^+\mu^-)K^*(890)^0(\to K^+\pi^-)$, used as a normalisation