A quantum leap in informal benchmarking : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Organisational Excellence at Massey University, Palmerston North, Manawatu, New Zealand

Despite the paucity of available literature on informal benchmarking and the consequential lack of its understanding, informal benchmarking has outranked established benchmarking (formal), placing 4th out of 20 of the most used business improvement tools, based on a 2008 Global Benchmarking Network (GBN) survey of 450 organisations worldwide. This paradox is exacerbated by the growing popularity of informal benchmarking, even though it is not correspondingly as effective as it is widely used. Therefore, two significant gaps need be filled: firstly, to develop a theoretical understanding of, and secondly, to investigate how to increase the effectiveness of informal benchmarking as an organisational improvement tool. A pragmatic mixed method quantitative-qualitative sequential design using an abductivedeductive- inductive approach is adopted. The product of abduction is a preliminary conceptual model of informal benchmarking from the transdisciplinary academic review of benchmarking, informal learning, organisation learning and knowledge management, augmented by concepts on quantum thinking, innovation and positive deviance. The model informs the quantitative survey questionnaire, whose deductive results of 81 survey responses from 14 countries informs the in-depth semi-structured interviews of 16 informants from 7 countries, the resulting dataset being inductively coded into conceptuallydriven dendrograms. The integrated findings refine the conceptual model of informal benchmarking, and develops a toolset-based application model (a pragmatic outcome of the conceptual model), a maturity assessment framework and an eco-system strategy. From here, an informal benchmarking roadmap is synthesised, representing a sustainable platform for informal benchmarking to be deployed as an effective organisational improvement initiative. The research sets the stage for a leap in scholarly understanding of informal benchmarking in the wider context of business and organisational improvement, and offers organisational improvement practitioners an invaluable cost-effective solution in a time-scarce executive world. This pragmatic study of informal benchmarking has possibly unleashed a different epistemological stance within the benchmarking field, by advocating an organic approach to benchmarking, in contrast to the highly methodical approaches associated with conventional benchmarking

REGULATORY BENCHMARKING IN CENTRAL EUROPE: CURRENT PRACTICE AND POSSIBILITIES OF DEVELOPMENT FOR THE ENERGY SECTOR

Benchmarking is a technique of performance evaluation in which comparisons are made to benchmarks that represent external performance standards. In the field of regulation of public utilities, benchmarking can be used as an element of performance-based regulation or as a pure regulatory method, called yardstick competition. In the absence of competition, benchmarking can be used to simulate competitive pressures by comparing a regulated firm's performance against an efficient standard. \\\\\\\\r\\\\\\\\nThe aim of this paper is to examine the Central European regulatory benchmarking practices in the energy sector, namely the electricity and natural gas distribution industries, and to analyse the possibilities of further development of regulatory benchmarking in this region. The countries onto which we focus are Czech Republic, Slovakia, Hungary, Poland, Germany, Austria and Switzerland. In the region of Central Europe there are still significant differences between countries, especially in terms of experiences in modern regulation, regulatory methods and practices, level of economic development etc. Differences are considerable especially between Western countries (Germany, Austria and Switzerland) and the countries of former Eastern Bloc (Czech Republic, Slovakia, Hungary and Poland). As a result, the degree of the use of regulatory benchmarking is also very diverse within this region. \\\\\\\\r\\\\\\\\nIn the first part of the paper, we develop basic theoretic concepts of economic regulation. Then we describe the most frequently used regulatory methods - cost-of-service regulation, incentive regulation and yardstick competition - and we deal with common regulatory benchmarking techniques, describe their principles and main challenges. Subsequenty, we provide an overview of regulatory methods and benchmarking practices for each country in the region of interest. In the final part of the paper, we analyse the challenges and possibilities for further development of regulatory benchmarking in the Central Europe. \\\\\\\\r\\\\\\\\nWe have found that except for Switzerland, all Central European regulatory regimes are based on some form of incentive regulation. The most sophisticated methods of benchmarking are used in Germany and Austria. In these countries, benchmarking is used in both electricity and natural gas industries. The Polish regulator is using a benchmarking method in cost efficiency analysis in electricity distribution. In Hungary, a specific method of benchmarking is used. In Switzerland, no benchmarking is used at present. In Czech Republic and Slovakia, some principles of benchmarking are adopted, but not directly to the revenue requirements setting. We summarize that the possibilities for development of regulatory benchmarking in the Central Europe could be extended by reducing market concentration, unbundling large vertically integrated companies, establishing a closer cooperation with the private sector and closer harmonization of regulatory frameworks.regulatory benchmarking, public utilities benchmarking, Central European utilites, public utilites regulation, regulatory framework

EU and OECD benchmarking and peer review compared

Benchmarking and peer review are essential elements of the so-called EU open method of coordination (OMC) which has been contested in the literature for lack of effectiveness. In this paper we compare benchmarking and peer review procedures as used by the EU with those used by the OECD. Different types of benchmarking and peer review are distinguished and pitfalls for (international) benchmarking are discussed. We find that the OECD has a clear single objective for its benchmarking and peer review activities (i.e. horizontal policy transfers) whereas the EU suffers from a mix of objectives (a. horizontal policy learning; b. EU wide vertical policy coordination and c. multilateral monitoring and surveillance under the shadow of hierarchy). Whereas the OECD is able to skirt around most of the benchmarking pitfalls, this is not the case for the EU. It is argued that, rather than continue working with the panacea OMC benchmarking and peer review currently represents, EU benchmarking should take a number of more distinct shapes in order to improve effectiveness. Moreover, in some areas benchmarking and peer review are not sufficient coordination tools, and are at best additional to those means of coordination that include enforceable sanctions

'Part'ly first among equals: Semantic part-based benchmarking for state-of-the-art object recognition systems

An examination of object recognition challenge leaderboards (ILSVRC, PASCAL-VOC) reveals that the top-performing classifiers typically exhibit small differences amongst themselves in terms of error rate/mAP. To better differentiate the top performers, additional criteria are required. Moreover, the (test) images, on which the performance scores are based, predominantly contain fully visible objects. Therefore, `harder' test images, mimicking the challenging conditions (e.g. occlusion) in which humans routinely recognize objects, need to be utilized for benchmarking. To address the concerns mentioned above, we make two contributions. First, we systematically vary the level of local object-part content, global detail and spatial context in images from PASCAL VOC 2010 to create a new benchmarking dataset dubbed PPSS-12. Second, we propose an object-part based benchmarking procedure which quantifies classifiers' robustness to a range of visibility and contextual settings. The benchmarking procedure relies on a semantic similarity measure that naturally addresses potential semantic granularity differences between the category labels in training and test datasets, thus eliminating manual mapping. We use our procedure on the PPSS-12 dataset to benchmark top-performing classifiers trained on the ILSVRC-2012 dataset. Our results show that the proposed benchmarking procedure enables additional differentiation among state-of-the-art object classifiers in terms of their ability to handle missing content and insufficient object detail. Given this capability for additional differentiation, our approach can potentially supplement existing benchmarking procedures used in object recognition challenge leaderboards.Comment: Extended version of our ACCV-2016 paper. Author formatting modifie

BENCHMARKING - A VALID STRATEGY FOR THE LONG TERM?

The present paper work deals with a popular method for developingrequirements and setting goals ââ¬â benchmarking. It contains general aspects about thispowerful performance improvement tool, including types of benchmarking, steps tofollow in Benchmarking analysis, its goals, the benefits in using it and some dangerscaused by using it, also; the whole paperwork can be considered as being a plea forcontinuous, ongoing, unending improvement in management context and sustains theidea that benchmarking enables decision-makers to understand exactly how muchimprovement they will need to accomplish in order to achieve superior performance. Wedecided to broach this issue because even if it is an actual one, none of the existingarticles did not attempt to answer whether or not benchmarking is a valid long-termstrategy that should be implemented by nowaday’s companies. The case study examinesthe benchmarking initiatives taken by Xerox, one of the world's leading copiercompanies, as a part of its 'Leadership through Quality' program; the case discusses thebenchmarking concept and its implementation in various processes at Xerox and it alsoexplores the positive impact of benchmarking practices on this company.benchmarking, management, strategy;

SMaSH: A Benchmarking Toolkit for Human Genome Variant Calling

Motivation: Computational methods are essential to extract actionable information from raw sequencing data, and to thus fulfill the promise of next-generation sequencing technology. Unfortunately, computational tools developed to call variants from human sequencing data disagree on many of their predictions, and current methods to evaluate accuracy and computational performance are ad-hoc and incomplete. Agreement on benchmarking variant calling methods would stimulate development of genomic processing tools and facilitate communication among researchers. Results: We propose SMaSH, a benchmarking methodology for evaluating human genome variant calling algorithms. We generate synthetic datasets, organize and interpret a wide range of existing benchmarking data for real genomes, and propose a set of accuracy and computational performance metrics for evaluating variant calling methods on this benchmarking data. Moreover, we illustrate the utility of SMaSH to evaluate the performance of some leading single nucleotide polymorphism (SNP), indel, and structural variant calling algorithms. Availability: We provide free and open access online to the SMaSH toolkit, along with detailed documentation, at smash.cs.berkeley.edu

Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology

Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology.Comment: Minor update