Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology.Comment: Minor update

A comparative study of benchmarking approaches for non-domestic buildings: Part 1 – Top-down approach

Benchmarking plays an important role in improving energy efficiency of non-domestic buildings. A review of energy benchmarks that underpin the UK’s Display Energy Certificate (DEC) scheme have prompted necessities to explore the benefits and limitations of using various methods to derive energy benchmarks. The existing methods were reviewed and grouped into top-down and bottom-up approaches based on the granularity of the data used. In the study, two top-down methods, descriptive statistics and artificial neural networks (ANN), were explored for the purpose of benchmarking energy performances of schools. The results were used to understand the benefits of using these benchmarks for assessing energy efficiency of buildings and the limitations that affect the robustness of the derived benchmarks. Compared to the bottom-up approach, top-down approaches were found to be beneficial in gaining insight into how peers perform. The relative rather than absolute feedback on energy efficiency meant that peer pressure was a motivator for improvement. On the other hand, there were limitations with regard to the extent to which the energy efficiency of a building could be accurately assessed using the top-down benchmarks. Moreover, difficulties in acquiring adequate data were identified as a key limitation to using the top-down approach for benchmarking non-domestic buildings. The study suggested that there are benefits in rolling out of DECs to private sector buildings and that there is a need to explore more complex methods to provide more accurate indication of energy efficiency in non-domestic buildings

Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology

The Strategy of the Commons: Modelling the Annual Cost of Successful ICT Services for European Research

The provision of ICT services for research is increasingly using Cloud services to complement the traditional federation of computing centres. Due to the complex funding structure and differences in the basic business model, comparing the cost-effectiveness of these options requires a new approach to cost assessment. This paper presents a cost assessment method addressing the limitations of the standard methods and some of the initial results of the study. This acts as an illustration of the kind of cost assessment issues high-utilisation rate ICT services should consider when choosing between different infrastructure options. The research is co-funded by the European Commission Seventh Framework Programme through the e-FISCAL project (contract number RI-283449)

Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness

In recent years, the notion of local robustness (or robustness for short) has emerged as a desirable property of deep neural networks. Intuitively, robustness means that small perturbations to an input do not cause the network to perform misclassifications. In this paper, we present a novel algorithm for verifying robustness properties of neural networks. Our method synergistically combines gradient-based optimization methods for counterexample search with abstraction-based proof search to obtain a sound and ({\delta}-)complete decision procedure. Our method also employs a data-driven approach to learn a verification policy that guides abstract interpretation during proof search. We have implemented the proposed approach in a tool called Charon and experimentally evaluated it on hundreds of benchmarks. Our experiments show that the proposed approach significantly outperforms three state-of-the-art tools, namely AI^2 , Reluplex, and Reluval

Dynamic Power Management for Neuromorphic Many-Core Systems

This work presents a dynamic power management architecture for neuromorphic many core systems such as SpiNNaker. A fast dynamic voltage and frequency scaling (DVFS) technique is presented which allows the processing elements (PE) to change their supply voltage and clock frequency individually and autonomously within less than 100 ns. This is employed by the neuromorphic simulation software flow, which defines the performance level (PL) of the PE based on the actual workload within each simulation cycle. A test chip in 28 nm SLP CMOS technology has been implemented. It includes 4 PEs which can be scaled from 0.7 V to 1.0 V with frequencies from 125 MHz to 500 MHz at three distinct PLs. By measurement of three neuromorphic benchmarks it is shown that the total PE power consumption can be reduced by 75%, with 80% baseline power reduction and a 50% reduction of energy per neuron and synapse computation, all while maintaining temporary peak system performance to achieve biological real-time operation of the system. A numerical model of this power management model is derived which allows DVFS architecture exploration for neuromorphics. The proposed technique is to be used for the second generation SpiNNaker neuromorphic many core system