Object-Proposal Evaluation Protocol is 'Gameable'

Object proposals have quickly become the de-facto pre-processing step in a number of vision pipelines (for object detection, object discovery, and other tasks). Their performance is usually evaluated on partially annotated datasets. In this paper, we argue that the choice of using a partially annotated dataset for evaluation of object proposals is problematic -- as we demonstrate via a thought experiment, the evaluation protocol is 'gameable', in the sense that progress under this protocol does not necessarily correspond to a "better" category independent object proposal algorithm. To alleviate this problem, we: (1) Introduce a nearly-fully annotated version of PASCAL VOC dataset, which serves as a test-bed to check if object proposal techniques are overfitting to a particular list of categories. (2) Perform an exhaustive evaluation of object proposal methods on our introduced nearly-fully annotated PASCAL dataset and perform cross-dataset generalization experiments; and (3) Introduce a diagnostic experiment to detect the bias capacity in an object proposal algorithm. This tool circumvents the need to collect a densely annotated dataset, which can be expensive and cumbersome to collect. Finally, we plan to release an easy-to-use toolbox which combines various publicly available implementations of object proposal algorithms which standardizes the proposal generation and evaluation so that new methods can be added and evaluated on different datasets. We hope that the results presented in the paper will motivate the community to test the category independence of various object proposal methods by carefully choosing the evaluation protocol.Comment: 15 pages, 11 figures, 4 table

A Proposal for Semantic Map Representation and Evaluation

Semantic mapping is the incremental process of “mapping” relevant information of the world (i.e., spatial information, temporal events, agents and actions) to a formal description supported by a reasoning engine. Current research focuses on learning the semantic of environments based on their spatial location, geometry and appearance. Many methods to tackle this problem have been proposed, but the lack of a uniform representation, as well as standard benchmarking suites, prevents their direct comparison. In this paper, we propose a standardization in the representation of semantic maps, by defining an easily extensible formalism to be used on top of metric maps of the environments. Based on this, we describe the procedure to build a dataset (based on real sensor data) for benchmarking semantic mapping techniques, also hypothesizing some possible evaluation metrics. Nevertheless, by providing a tool for the construction of a semantic map ground truth, we aim at the contribution of the scientific community in acquiring data for populating the dataset

Mid-term report for the CORE Organic II funded project. “Innovative cropping Practices to increase soil health of organic fruit tree orchards” BIO-INCROP

Activities performed in the first part of BIO-INCROP project concern five of the eight main objectives fixed in the project proposal. They are: Evaluation of soil borne pest and pathogens involved in replant disease Role of rhizospheric bacterial and fungal communities in plant health Selection of naturally available resources to increase microbial diversity and biomass Compost and organic amendments Evaluation of biologically active formulates The document reports main research results and shows main items of dissemination activity performed in the first part of the project

A Social Rehabilitation Program Implementation and Analysis: A Demonstration-Experimental Proposal

The intent of this proposal is to institute a social rehabilitation program on a ward of approximately 40-50 patients and then to assess and analyze the results of the program. Evaluation is to be undertaken through the utilization of a control group of approximately the same size, composition, and general philosophy located within the same hospital. In designing a research proposal for such a hospital, three important issues must be addressed as the effects of the program have ramifications that radiate throughout the hospital structure. The three issues, in the order of the manner they will be outlined, are operational problems, administrative problems, and finally methodological issues

Multilingual interactive experiments with Flickr

This paper presents a proposal for iCLEF 2006, the interactive track of the CLEF cross-language evaluation campaign. In the past, iCLEF has addressed applications such as information retrieval and question answering. However, for 2006 the focus has turned to text-based image retrieval from Flickr. We describe Flickr, the challenges this kind of collection presents to cross-language researchers, and suggest initial iCLEF tasks

Development of a novel virtual coordinate measuring machine

Existing VCMMs (virtual coordinate measuring machine) have been mainly developed to either simulate the measurement process hence enabling the off-line programming, or to perform error analysis and uncertainty evaluation. Their capability and performance could be greatly improved if there is a complete solution to cover the whole process and provide an integrated environment. The aim of this study is to develop such a VCMM that not only supports measurement process simulation, but also performs uncertainty evaluation. It makes use of virtual reality techniques to provide an accurate model of a physical CMM, together with uncertainty evaluation. An interface is also provided to communicate with CMM controller, allowing the measuring programs generated and simulated in the VCMM to be executed or tested on the physical CMM afterwards. This paper discusses the proposal of a novel VCMM design and the preliminary results

A PROPOSAL TO REFINE CONCEPT MAPPING FOR EFFECTIVE SCIENCE LEARNING

Concept maps are found to be useful in eliciting knowledge, meaningful learning, evaluation of understanding and in studying the nature of changes taking place during cognitive development, particularly in the classroom. Several experts have claimed the effectiveness of this tool for learning science. We agree with the claim, but the effectiveness will improve only if we gradually introduce a certain amount of discipline in constructing the maps. The discipline is warranted, we argue, because science thrives to be an unambiguous and rigorously structured body of knowledge. Since learning science may be seen as a process where a novice is expected to be transformed into an expert, we use the context of learning science for making the proposal. Further, we identify certain anomalies in the evaluation of concept maps, and suggest that the evaluation should be based on semantics of the linking words (relation types) and not on graphical criteria alone.\u