Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation

This paper presents an overview of the ImageCLEF 2018 evaluation campaign, an event that was organized as part of the CLEF (Conference and Labs of the Evaluation Forum) Labs 2018. ImageCLEF is an ongoing initiative (it started in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval with the aim of providing information access to collections of images in various usage scenarios and domains. In 2018, the 16th edition of ImageCLEF ran three main tasks and a pilot task: (1) a caption prediction task that aims at predicting the caption of a figure from the biomedical literature based only on the figure image; (2) a tuberculosis task that aims at detecting the tuberculosis type, severity and drug resistance from CT (Computed Tomography) volumes of the lung; (3) a LifeLog task (videos, images and other sources) about daily activities understanding and moment retrieval, and (4) a pilot task on visual question answering where systems are tasked with answering medical questions. The strong participation, with over 100 research groups registering and 31 submitting results for the tasks, shows an increasing interest in this benchmarking campaign

Analysing TB severity levels with an enhanced deep residual learning– depth-resnet

This work responds to the Competition of Tuberculosis Task organised by imageCLEF 2018. While Task #3 appears to be challenging, the experience was very enjoyable. If time had been permitted, it was certain that more accurate results could have been achieved. The authors submitted 2 runs. Based on the given training datasets with severity levels of 1 to 5, an enhanced deep residual learning architecture, depthResNet, is developed and applied to train the datasets to classify 5 categories. The datasets are pre-processed with each volume being segmented into twenty- 128×128×depth blocks with ~64 pixel overlaps. While each block has been predicted with a severity level, assembling all constituent block scores together to give an overall label for the concerned volume tends to be more challenging. Since the probability of high severity is not provided from the training datasets, which bears little resemblance to the classification probability, the submission of probability for the first run was manually assigned as 0.9, 0.7, 0.5, 0.3, and 0.1 to severity levels of 1 to 5 respectively. After the deadline was extended, the model was re-trained with frame numbers increased from 1 to 8, which takes much longer to train. In addition, a new measure was introduced to calculate the overall probability of high severity based on the block scores. As a result, with regard to classification accuracy, the 2nd submitted run achieved place 14 over a total of 36 submissions, a significant improvement from position of 35 from the first run

ImageCLEF 2019: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications

This paper presents an overview of the foreseen ImageCLEF 2019 lab that will be organized as part of the Conference and Labs of the Evaluation Forum - CLEF Labs 2019. ImageCLEF is an ongoing evaluation initiative (started in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2019, the 17th edition of ImageCLEF will run four main tasks: (i) a Lifelog task (videos, images and other sources) about daily activities understanding, retrieval and summarization, (ii) a Medical task that groups three previous tasks (caption analysis, tuberculosis prediction, and medical visual question answering) with newer data, (iii) a new Coral task about segmenting and labeling collections of coral images for 3D modeling, and (iv) a new Security task addressing the problems of automatically identifying forged content and retrieve hidden information. The strong participation, with over 100 research groups registering and 31 submitting results for the tasks in 2018 shows an important interest in this benchmarking campaign and we expect the new tasks to attract at least as many researchers for 2019

ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications

This paper presents an overview of the 2020 ImageCLEF lab that will be organized as part of the Conference and Labs of the Evaluation Forum - CLEF Labs 2020 in Thessaloniki, Greece. ImageCLEF is an ongoing evaluation initiative (run since 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2020, the 18th edition of ImageCLEF will organize four main tasks: (i) a Lifelog task (videos, images and other sources) about daily activity understanding, retrieval and summarization, (ii) a Medical task that groups three previous tasks (caption analysis, tuberculosis prediction, and medical visual question answering) with new data and adapted tasks, (iii) a Coral task about segmenting and labeling collections of coral images for 3D modeling, and a new (iv) Web user interface task addressing the problems of detecting and recognizing hand drawn website UIs (User Interfaces) for generating automatic code. The strong participation, with over 235 research groups registering and 63 submitting over 359 runs for the tasks in 2019 shows an important interest in this benchmarking campaign. We expect the new tasks to attract at least as many researchers for 2020

Overview of the ImageCLEF 2018 Caption Prediction Tasks

The caption prediction task is in 2018 in its second edition after the task was first run in the same format in 2017. For 2018 the database was more focused on clinical images to limit diversity. As automatic methods with limited manual control were used to select images, there is still an important diversity remaining in the image data set. Participation was relatively stable compared to 2017. Usage of external data was restricted in 2018 to limit critical remarks regarding the use of external resources by some groups in 2017. Results show that this is a difficult task but that large amounts of training data can make it possible to detect the general topics of an image from the biomedical literature. For an even better comparison it seems important to filter the concepts for the images that are made available. Very general concepts (such as “medical image”) need to be removed, as they are not specific for the images shown, and also extremely rare concepts with only one or two examples can not really be learned. Providing more coherent training data or larger quantities can also help to learn such complex models

ImageCLEF 2019: Multimedia Retrieval in Medicine, Lifelogging, Security and Nature

This paper presents an overview of the ImageCLEF 2019 lab, organized as part of the Conference and Labs of the Evaluation Forum - CLEF Labs 2019. ImageCLEF is an ongoing evaluation initiative (started in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2019, the 17th edition of ImageCLEF runs four main tasks: (i) a medical task that groups three previous tasks (caption analysis, tuberculosis prediction, and medical visual question answering) with new data, (ii) a lifelog task (videos, images and other sources) about daily activities understanding, retrieval and summarization, (iii) a new security task addressing the problems of automatically identifying forged content and retrieve hidden information, and (iv) a new coral task about segmenting and labeling collections of coral images for 3D modeling. The strong participation, with 235 research groups registering, and 63 submitting over 359 runs, shows an important interest in this benchmark campaign

Analysis of tuberculosis severity levels from CT pulmonary images based on enhanced residual deep learning architecture

This research investigates the application of CT pulmonary images to the detection and characterisation of TB at five levels of severity, in order to monitor the efficacy of treatment. To contend with smaller datasets (i.e. in hundreds) and the characteristics of CT TB images in which abnormalities occupy only limited regions, a 3D block-based residual deep learning network (ResNet) coupled with injection of depth information (depth-Resnet) at each layer was implemented. Progress in evaluation has been accomplished in two ways. One is to assess the proposed depth-Resnet in prediction of severity scores and another is to analyse the probability of high severity of TB. For the former, delivered results are of 92.70 ± 5.97% and 67.15 ± 1.69% for proposed depth-Resnet and ResNet-50 respectively. For the latter, two additional measures are put forward, which are calculated using (1) the overall severity (1 to 5) probability, and (2) separate probabilities of both high severity (scores of 1 to 3) and low severity (scores of 4 and 5) respectively, when scores of 1 to 5 are mapped into initial probabilities of (0.9, 0.7, 0.5, 0.3, 0.2) respectively. As a result, these measures achieve the averaged accuracies of 75.88% and 85.29% for both methods respectively

Overview of ImageCLEFlifelog 2018: daily living understanding and lifelog moment retrieval

Benchmarking in Multimedia and Retrieval related research fields has a long tradition and important position within the community. Benchmarks such as the MediaEval Multimedia Benchmark or CLEF are well established and also served by the community. One major goal of these competitions beside of comparing different methods and approaches is also to create or promote new interesting research directions within multimedia. For example the Medico task at MediaEval with the goal of medical related multimedia analysis. Although lifelogging creates a lot of attention in the community which is shown by several workshops and special session hosted about the topic. Despite of that there exist also some lifelogging related benchmarks. For example the previous edition of the lifelogging task at ImageCLEF. The last years ImageCLEFlifelog task was well received but had some barriers that made it difficult for some researchers to participate (data size, multi modal features, etc.) The ImageCLEFlifelog 2018 tries to overcome these problems and make the task accessible for an even broader audience (eg, pre-extracted features are provided). Furthermore, the task is divided into two subtasks (challenges). The two challenges are lifelog moment retrieval (LMRT) and the Activities of Daily Living understanding (ADLT). All in all seven teams participated with a total number of 41 runs which was an significant increase compared to the previous year