Each system is evaluated against each of the four generated silver standard corpora and, in addition, against the gold annotations of the ShARe/CLEF corpus.

<p>Concept recognition systems were evaluated by obtaining precision, recall and F1-measure based on the different applied annotation sets in this study. Each of the double-pointed arrows symbolises both types of evaluation: an <i>exact</i> and a <i>sentence-based</i>.</p

Example sentence illustrating the annotations obtained from the four concept recognition systems and propagation to silver standard corpus.

<p>Each of the four concept recognition systems assigns annotations to the textual content of the corpora independent from the other systems. If two systems annotate the same text span, e.g. <i>blood pressure</i> (CUI:C1272641) or <i>quality of life</i> (CUI:0518214) in the case of a sentence-based matching, this text annotation is included into the silver standard corpus of the text source.</p

Generation of silver standard annotations depends on individual, independent concept recognition system annotations.

<p>Each system assigns annotations for each of the four corpora independently. After all the annotations are obtained, the systems’ outputs are harmonised on a corpus level. Two out of four systems have to agree on an annotation in order for it to be propagated to the silver annotation set.</p

A summary of the areas encompassed by the ARRIVE guidelines.

<p>The ARRIVE guidelines are organised into twenty sections, which cover all areas of a typical research manuscript. <i>Image credit</i>: NC3Rs: <a href="https://www.nc3rs.org.uk/arrive-guidelines" target="_blank">https://www.nc3rs.org.uk/arrive-guidelines</a>.</p

Request for Community partnership in data resource licensing planning

<p>We write an open letter to the NIH Data Research Council to initiate a dialog regarding NIH decisions on data use agreements and licenses. We are members of NIH-funded research groups that collect and/or integrate biomedical data from diverse sources for the purpose of advancing diagnosis, prognosis, treatment selection, and mechanistic discovery. </p><p><br></p><p>We welcome additional signatories here:</p><p>https://docs.google.com/document/d/1fbwKxnPu5f1YXlMM6UMyfBqHx_Inz86tKzwRFO4W8jQ<br></p><br><p>Summary:</p><br><ul><li><p>The current diversity of data use agreements and licenses significantly hampers the ability to reuse and redistribute data in various informatics contexts.</p></li><li><p>We believe that any mandatory data licensing policy must also include a plan for ensuring access, sustainability, and data quality. </p></li><li><p>We request community partnership with NIH to develop common licensing and data reuse plans. </p></li></ul