Report on the 1st Simulation for Information Retrieval Workshop (Sim4IR 2021) at SIGIR 2021

Simulation is used as a low-cost and repeatable means of experimentation. As Information Retrieval (IR) researchers, we are no strangers to the idea of using simulation within our own field---such as the traditional means of IR system evaluation as manifested through the Cranfield paradigm. While simulation has been used in other areas of IR research (such as the study of user behaviours), we argue that the potential for using simulation has been recognised by relatively few IR researchers so far. To this end, the Sim4IR workshop was held online on July 15th, 2021 in conjunction with ACM SIGIR 2021. Building on past efforts, the goal of the workshop was to create a forum for researchers and practitioners to promote methodology and development of more widespread use of simulation for IR evaluation. Around 80 participants took part over two sessions. A total of two keynotes, three original paper presentations, and eight 'encore talks' were presented. The main conclusions from the resultant discussion were that simulation has the potential to offer solutions to the limitations of existing evaluation methodologies, but there is more research needed toward developing realistic user simulators; and the development and sharing of simulators, in the form of toolkits and online services, is critical for successful uptake.publishedVersio

Implementation of Early Next-Generation Sequencing for Inborn Errors of Immunity:A Prospective Observational Cohort Study of Diagnostic Yield and Clinical Implications in Dutch Genome Diagnostic Centers

OBJECTIVE: Inborn errors of immunity (IEI) are a heterogeneous group of disorders, affecting different components of the immune system. Over 450 IEI related genes have been identified, with new genes continually being recognized. This makes the early application of next-generation sequencing (NGS) as a diagnostic method in the evaluation of IEI a promising development. We aimed to provide an overview of the diagnostic yield and time to diagnosis in a cohort of patients suspected of IEI and evaluated by an NGS based IEI panel early in the diagnostic trajectory in a multicenter setting in the Netherlands. STUDY DESIGN: We performed a prospective observational cohort study. We collected data of 165 patients with a clinical suspicion of IEI without prior NGS based panel evaluation that were referred for early NGS using a uniform IEI gene panel. The diagnostic yield was assessed in terms of definitive genetic diagnoses, inconclusive diagnoses and patients without abnormalities in the IEI gene panel. We also assessed time to diagnosis and clinical implications. RESULTS: For children, the median time from first consultation to diagnosis was 119 days versus 124 days for adult patients (U=2323; p=0.644). The median turn-around time (TAT) of genetic testing was 56 days in pediatric patients and 60 days in adult patients (U=1892; p=0.191). A definitive molecular diagnosis was made in 25/65 (24.6%) of pediatric patients and 9/100 (9%) of adults. Most diagnosed disorders were identified in the categories of immune dysregulation (n=10/25; 40%), antibody deficiencies (n=5/25; 20%), and phagocyte diseases (n=5/25; 20%). Inconclusive outcomes were found in 76/165 (46.1%) patients. Within the patient group with a genetic diagnosis, a change in disease management occurred in 76% of patients. CONCLUSION: In this cohort, the highest yields of NGS based evaluation for IEI early in the diagnostic trajectory were found in pediatric patients, and in the disease categories immune dysregulation and phagocyte diseases. In cases where a definitive diagnosis was made, this led to important disease management implications in a large majority of patients. More research is needed to establish a uniform diagnostic pathway for cases with inconclusive diagnoses, including variants of unknown significance

Oral application of L-menthol in the heat: From pleasure to performance

When menthol is applied to the oral cavity it presents with a familiar refreshing sensation and cooling mint flavour. This may be deemed hedonic in some individuals, but may cause irritation in others. This variation in response is likely dependent upon trigeminal sensitivity toward cold stimuli, suggesting a need for a menthol solution that can be easily personalised. Menthol’s characteristics can also be enhanced by matching colour to qualitative outcomes; a factor which can easily be manipulated by practitioners working in athletic or occupational settings to potentially enhance intervention efficacy. This presentation will outline the efficacy of oral menthol application for improving time trial performance to date, either via swilling or via co-ingestion with other cooling strategies, with an emphasis upon how menthol can be applied in ecologically valid scenarios. Situations in which performance is not expected to be enhanced will also be discussed. An updated model by which menthol may prove hedonic, satiate thirst and affect ventilation will also be presented, with the potential performance implications of these findings discussed and modelled. Qualitative reflections from athletes that have implemented menthol mouth swilling in competition, training and maximal exercise will also be included

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical science

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical science.</p

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency–Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe