Mixture Model Parameters.

<p>Each observation made by the expert can take one of three category values from . The parameters governing the model are and , where is the true category of the observed data point, and and correspond to the expert labelling the data point as or , respectively.</p

Experimental results on the level of agreement among experts.

<p>More than 8,000 RBC images were remotely presented to these experts, using the interface that is illustrated in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0046192#pone-0046192-g001" target="_blank">Figure 1</a>. For example, at least five out of nine experts agree on 97% of images labelled as negative, 64% labelled as positive, and 7% labelled as questionable; whereas only one out of nine experts at any given time agrees on the full set of labels (i.e., no two experts agree completely!). Note that these percentages are based on individual RBC images. For example, in the 3 column, it is not the same 3 experts who agree on the images but possibly different sets of 3 experts for different images.</p

Decoding model of the proposed setup.

<p>The expert responses are treated as the observed variables and the true image labels as the latent variables in a mixture model with parameters . Expectation Maximisation (EM) is used to obtain the Maximum Likelihood solution to the data.</p

The browser-based interface for remote cell labelling.

<p>Each expert is allowed to navigate through the database of cell images, eliminating the infected cells and marking those that are questionable (i.e., cannot be reliably labelled as infected or uninfected).</p

Experimental results on the level of <i>self-inconsistency</i> of each expert within each category.

<p>More than 8,000 RBC images were separately presented to these experts, using the interface that is illustrated in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0046192#pone-0046192-g001" target="_blank">Figure 1</a>. For example, expert 1 changes her/his decision regarding what s/he labelled as negative 1.6% of the time. Similarly s/he changed her/his mind for 1.3% and 1.8% of the positive and questionable images, respectively.</p

Sample cells classified by the proposed methodology.

<p>Each observation made by the expert can take one of three category values from . The parameters governing the model are and , where is the true category of the observed data point, and and correspond to the expert labelling the data point as 0 or 1, respectively. Please refer to the <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0046192#s2" target="_blank">Methods</a> Section for further details.</p

Forward model of the proposed setup.

<p>There are a total of images with possible labels from being sent to experts. The expert labels each image with a certain probability . The final dataset consists of an matrix of values from the set .</p

Experimental performance metrics of the experts.

<p>The metrics are calculated after combining the responses of all the experts using EM and then assuming the results to be correct. , , , , where , and correspond to the number of true positive, true negative, false positive, and false negative labels respectively.</p

Performance results from 9 simulated experts with varying average ensemble accuracies.

<p>We can see that the combined accuracy (in green) is always higher than the maximum accuracy of the ensemble. Refer to the <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0046192#s2" target="_blank">Methods</a> section for details.</p

Proposed platform.

<p><b>A</b>) Biomedical data (e.g., images of thin blood smear samples) from individual light microscopes all around the world are transmitted to data centres where they are pre-processed and digitally distributed among gamers, which in turn diagnose and transmit their responses back. These individual results of the gamers are then fused toward a final diagnosis, the result of which is transmitted back to the point-of-care or the clinic/hospital. In the map above, orange-coloured regions show locations where risk of contraction of malaria still exists. <b>B</b>) Block diagram of the presented platform.</p