'Part'ly first among equals: Semantic part-based benchmarking for state-of-the-art object recognition systems

An examination of object recognition challenge leaderboards (ILSVRC, PASCAL-VOC) reveals that the top-performing classifiers typically exhibit small differences amongst themselves in terms of error rate/mAP. To better differentiate the top performers, additional criteria are required. Moreover, the (test) images, on which the performance scores are based, predominantly contain fully visible objects. Therefore, `harder' test images, mimicking the challenging conditions (e.g. occlusion) in which humans routinely recognize objects, need to be utilized for benchmarking. To address the concerns mentioned above, we make two contributions. First, we systematically vary the level of local object-part content, global detail and spatial context in images from PASCAL VOC 2010 to create a new benchmarking dataset dubbed PPSS-12. Second, we propose an object-part based benchmarking procedure which quantifies classifiers' robustness to a range of visibility and contextual settings. The benchmarking procedure relies on a semantic similarity measure that naturally addresses potential semantic granularity differences between the category labels in training and test datasets, thus eliminating manual mapping. We use our procedure on the PPSS-12 dataset to benchmark top-performing classifiers trained on the ILSVRC-2012 dataset. Our results show that the proposed benchmarking procedure enables additional differentiation among state-of-the-art object classifiers in terms of their ability to handle missing content and insufficient object detail. Given this capability for additional differentiation, our approach can potentially supplement existing benchmarking procedures used in object recognition challenge leaderboards.Comment: Extended version of our ACCV-2016 paper. Author formatting modifie

ICT Test Bed Project: benchmarking procedure analyses: project year 2, 2004

ICT Test Bed evaluation analysis of attainment data of participating schools

Evaluating Local Community Methods in Networks

We present a new benchmarking procedure that is unambiguous and specific to local community-finding methods, allowing one to compare the accuracy of various methods. We apply this to new and existing algorithms. A simple class of synthetic benchmark networks is also developed, capable of testing properties specific to these local methods.Comment: 8 pages, 9 figures, code included with sourc

Benchmarking procedure and analyses with performance data from 2006 (Year 4): technical report 21, 2006 (T21/06)

ICT TB Benchmarking and attainment analysi

Growth Rates Preservation (GRP) temporal benchmarking: Drawbacks and alternative solutions

Benchmarking monthly or quarterly series to annual data is a common practice in many National Statistical Institutes. The benchmarking problem arises when time series data for the same target variable are measured at different frequencies and there is a need to remove discrepancies between the sums of the sub-annual values and their annual benchmarks. Several benchmarking methods are available in the literature. The Growth Rates Preservation (GRP) benchmarking procedure is often considered the best method. It is often claimed that this procedure is grounded on an ideal movement preservation principle. However, we show that there are important drawbacks to GRP, relevant for practical applications, that are unknown in the literature. Alternative benchmarking models will be considered that do not suffer from some of GRP\u2019s side effects

Facility layout problem: Bibliometric and benchmarking analysis

Facility layout problem is related to the location of departments in a facility area, with the aim of determining the most effective configuration. Researches based on different approaches have been published in the last six decades and, to prove the effectiveness of the results obtained, several instances have been developed. This paper presents a general overview on the extant literature on facility layout problems in order to identify the main research trends and propose future research questions. Firstly, in order to give the reader an overview of the literature, a bibliometric analysis is presented. Then, a clusterization of the papers referred to the main instances reported in literature was carried out in order to create a database that can be a useful tool in the benchmarking procedure for researchers that would approach this kind of problems

Cancer gene prioritization by integrative analysis of mRNA expression and DNA copy number data: a comparative review

A variety of genome-wide profiling techniques are available to probe complementary aspects of genome structure and function. Integrative analysis of heterogeneous data sources can reveal higher-level interactions that cannot be detected based on individual observations. A standard integration task in cancer studies is to identify altered genomic regions that induce changes in the expression of the associated genes based on joint analysis of genome-wide gene expression and copy number profiling measurements. In this review, we provide a comparison among various modeling procedures for integrating genome-wide profiling data of gene copy number and transcriptional alterations and highlight common approaches to genomic data integration. A transparent benchmarking procedure is introduced to quantitatively compare the cancer gene prioritization performance of the alternative methods. The benchmarking algorithms and data sets are available at http://intcomp.r-forge.r-project.orgComment: PDF file including supplementary material. 9 pages. Preprin

The Enhanced Denton's Benchmarking Procedure for Extrapolation. Matrix Formulation and Practical Issues.

Statistical agencies make often recourse to benchmarking to estimate Quarterly National Accounts. One of the most used benchmarking procedure is the one based on the Proportional First Di®erences (PFD) criterion, according to the original proposal by Denton (1971). Despite its good properties for distribution, this method might give unsatisfactory results in the extrapolation of quarters of the current year. An enhanced version of the Denton PFD method has been suggested by the International Monetary Fund (IMF) to improve forecasting accuracy. In this paper we provide a matrix formalization of the enhanced solution, and analyze its properties through artificial data. Finally, we critically review the shortcut version of the enhanced method proposed by the IMF, which is currently in use in some statistical agencies