Sources of inconsistency in focus selection between experimenters.

<p>Matches are shown in green and unmatched points in magenta. (A) Dual assignment of a single maximum by experimenter 2 (‘double clicking’); (B) mislabelling of non-true maxima; (C) interpretation of diffraction-limited foci as a single focus or dual foci (‘doublet’); and (D) arbitrary selection of different background levels to determine inclusion of foci in the analysis. Arrows indicate the discordant foci in magenta.</p

Parameters obtained from a single image show wide variation in performance when applied across the entire dataset.

<p>The heatmap shows the F1 scores for the optimal parameters from the training image (‘Image used for training’) applied to all 21 assigned images from each experimenter (‘Image with manual focus selection’ (‘Test’)). P1, P2, and P3 refer to the experimenter ID. Arrows labelled A to C are discussed in the main text. The colour key and histogram of all F1 scores are shown above the heatmap.</p

The average focus intensity is higher with increasing agreement between experimenters (cluster size or concordance).

<p>Greedy clustering was performed by first comparing foci selected by two experimenters using an 8 pixel radius. When the focus was selected by both experimenters, a cluster of size 2 was generated. Foci that were only picked by a single experimenter were used to seed a cluster of size 1. The clusters were compared to the third experimenter and any matches used to increase the cluster sizes such that matches between all three experimenters gave rise to cluster size 3. Clustering was performed using experimenter order 123, 231 and 312 and all results combined for plotting (A). Because experimenters clicked on slightly different parts of the focus, the focus intensity was calculated as the average pixel value selected by the experimenters. The normalized values for the cluster were calculated by dividing the average pixel value by the median value of all of the selected foci from the same image. The horizontal, dashed line indicates a normalized value of 1. The analysis of individual images are shown in (B).</p

Example image and manual focus identification.

<p>(A) Spread nuclear DNA from budding yeast meiosis stained with DAPI (DNA, left) and antibodies against Zip3-GFP (middle). The merged image is shown in false colour on the right. Scale bar  =  2 µm. (B) Experimenter-labelled maxima (foci) of Zip3-GFP within the DNA region (determined by the experimenter). (C) Quantification of foci from the same 21 images by three different experimenters (P1 to P3). ‘Expert’ refers to scientists with previous knowledge of Zip3 or Msh4 quantification. Magenta bars represent the arithmetic mean, the black bar in the box-and-whisker plot shows the median value and all individual data points are shown as dots. Whiskers extend 1.5× of the interquartile range or to the minimum/maximum value, when these fall within 1.5× of the interquartile range.</p

Agreement of focus selection between any two experimenters decreases with reduced focus intensity.

<p>Foci were arranged by pixel intensity and divided into quartiles. The match statistics (Jaccard scores) were computed for each quartile (Q1–Q4) and the entire set (‘All’). Images are arranged in descending order of overall Jaccard score. Q1 consisted of the 25% of foci with the lowest intensity, Q2 the 26^th–50^th percentile (‘low intensity’); Q3 the 51^st–75^th percentile (‘medium intensity’), and Q4 the 25% of foci with the highest intensity (‘highest intensity’).</p

Training on multiple images improves consistency in focus detection.

<p>The FindFoci Optimiser was trained using 1 to 21 images. When the number of combinations of images used for training was small (n = 1 or n = 20 has 21 possible combinations; n = 2 or n = 19 has 210 possible combination) all possible combinations were used. For larger numbers of possible combinations, a random subset of 100 combinations were used to select images for training of the algorithm. The parameters obtained from training were tested against the manually marked images from each experimenter and the F1-score calculated. P1 to P3 refers to experimenter ID, with the first mention indicating the images used for training and the second referring to the manually-assigned images against which the parameter combination was assessed. <i>I.e.</i> ‘P1 versus P2' means that P1's marked images were used for training and the parameters were used to predict focus selection by P2.</p

The FindFoci algorithm closely matches human assignments^a.

a<p>The FindFoci algorithm was trained on individual images and the highest F1-score reported and averaged over the 21 images per experimenter. The analysis was performed using the original selected pixel coordinates (‘Original’) or the coordinates after alignment to their appropriate local maximum (‘Aligned’).</p><p>The FindFoci algorithm closely matches human assignments^{<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0114749#nt102" target="_blank">a</a>}.</p

Comparison of focus assignments between experimenters.

<p>An example image shows matches (within 8 pixels) of two experimenters in green using the position of experimenter P1. Unmatched foci that were selected only by experimenter P1 are shown in yellow and unmatched foci selected only by experimenter P2 are indicated in blue. The Jaccard score for this comparison was 0.81. Only spots within the DAPI stained region were extracted for analysis. Arrows indicate examples where experimenters have clicked a short distance from a maximum.</p

Comparison of manual focus selection by two different experimenters across 21 images^a.

a<p>Matches were iteratively assigned using the closest pairs within 8 pixels. The table shows the comparison of the raw spot assignments and the aligned spot assignments where points are moved to their closest local maximum (see methods for details). Scores are averaged over 21 images.</p><p>Comparison of manual focus selection by two different experimenters across 21 images^{<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0114749#nt101" target="_blank">a</a>}.</p

Comparison of F1-scores from human assignments or automated focus assignment to other experimenters^a.

a<p>The average F1 score between experimenters across 21 images using the original manually assigned foci from each experimenter (‘Original’) or the automated focus assignment trained across a dataset (‘Algorithm’) when compared against the ‘ground truth’, defined by another experimenter. Matches were iteratively assigned using the closest pairs within 8 pixels. Values are shown to three significant figures.</p><p>Comparison of F1-scores from human assignments or automated focus assignment to other experimenters^{<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0114749#nt103" target="_blank">a</a>}.</p