Information retrieval (IR) and the paradox of change: An analysis using the philosophy of Parmenides

Purpose – This paper aims to explore whether philosophical insights from Plato's dialogue “Parmenides” on the complex and often paradoxical nature of change can illuminate the nature of information retrieval (IR). IR is modelled as a dialectic process involving mutually dependent yet conflicting forces between the subjective and the objective. These forces operate to produce change in the subjective experience of users (becoming informed) through facilitating a relationship with objective documents. Accurately modelling, predicting and enabling this process remains a persistent problem for IR and this paper seeks to examine the extent to which this is because of the nature of change. Design/methodology/approach – The paper is a conceptual analysis and literature review. Findings – The problem of change (what it is, how it happens and how we can know it has happened) is essential to our understanding of information as information normally implies some kind of change in knowledge state. Any process of change, however, on examination of its qualities, appears to necessitate the combination of irreconcilable and conflicting forces. The apparent contradictions within the existence of change as discussed in “Parmenides” also exist in IR on both a theoretical and a technical level. Research limitations/implications – Change is a central concept for information in general and IR in particular. A deeper understanding of the paradoxical nature of change can provide new insights into IR theory and practice. Originality/value – The paper presents a new historical philosophical perspective on the nature of change and applies it to current IR problems

Symbiosis between the TRECVid benchmark and video libraries at the Netherlands Institute for Sound and Vision

Audiovisual archives are investing in large-scale digitisation efforts of their analogue holdings and, in parallel, ingesting an ever-increasing amount of born- digital files in their digital storage facilities. Digitisation opens up new access paradigms and boosted re-use of audiovisual content. Query-log analyses show the shortcomings of manual annotation, therefore archives are complementing these annotations by developing novel search engines that automatically extract information from both audio and the visual tracks. Over the past few years, the TRECVid benchmark has developed a novel relationship with the Netherlands Institute of Sound and Vision (NISV) which goes beyond the NISV just providing data and use cases to TRECVid. Prototype and demonstrator systems developed as part of TRECVid are set to become a key driver in improving the quality of search engines at the NISV and will ultimately help other audiovisual archives to offer more efficient and more fine-grained access to their collections. This paper reports the experiences of NISV in leveraging the activities of the TRECVid benchmark

Experiences from the ImageCLEF Medical Retrieval and Annotation Tasks

The medical tasks in ImageCLEF have been run every year from 2004-2018 and many different tasks and data sets have been used over these years. The created resources are being used by many researchers well beyond the actual evaluation campaigns and are allowing to compare the performance of many techniques on the same grounds and in a reproducible way. Many of the larger data sets are from the medical literature, as such images are easier to obtain and to share than clinical data, which was used in a few smaller ImageCLEF challenges that are specifically marked with the disease type and anatomic region. This chapter describes the main results of the various tasks over the years, including data, participants, types of tasks evaluated and also the lessons learned in organizing such tasks for the scientific community

VISCERAL ::evaluation-as-a-service for medical imaging

Systematic evaluation has had a strong impact on many data analysis domains, for example, TREC and CLEF in information retrieval, ImageCLEF in image retrieval, and many challenges in conferences such as MICCAI for medical imaging and ICPR for pattern recognition. With Kaggle, a platform for machine learning challenges has also had a significant success in crowdsourcing solutions. This shows the importance to systematically evaluate algorithms and that the impact is far larger than simply evaluating a single system. Many of these challenges also showed the limits of the commonly used paradigm to prepare a data collection and tasks, distribute these and then evaluate the participants’ submissions. Extremely large datasets are cumbersome to download, while shipping hard disks containing the data becomes impractical. Confidential data can often not be shared, for example medical data, and also data from company repositories. Real-time data will never be available via static data collections as the data change over time and data preparation often takes much time. The Evaluation-as-a-Service (EaaS) paradigm tries to find solutions for many of these problems and has been applied in the VISCERAL project. In EaaS, the data are not moved but remain on a central infrastructure. In the case of VISCERAL, all data were made available in a cloud environment. Participants were provided with virtual machines on which to install their algorithms. Only a small part of the data, the training data, was visible to participants. The major part of the data, the test data, was only accessible to the organizers who ran the algorithms in the participants’ virtual machines on the test data to obtain impartial performance measures