88,274 research outputs found
The DEVStone Metric: Performance Analysis of DEVS Simulation Engines
The DEVStone benchmark allows us to evaluate the performance of
discrete-event simulators based on the DEVS formalism. It provides model sets
with different characteristics, enabling the analysis of specific issues of
simulation engines. However, this heterogeneity hinders the comparison of the
results among studies, as the results obtained on each research work depend on
the chosen subset of DEVStone models. We define the DEVStone metric based on
the DEVStone synthetic benchmark and provide a mechanism for specifying
objective ratings for DEVS-based simulators. This metric corresponds to the
average number of times that a simulator can execute a selection of 12 DEVStone
models in one minute. The variety of the chosen models ensures we measure
different particularities provided by DEVStone. The proposed metric allows us
to compare various simulators and to assess the impact of new features on their
performance. We use the DEVStone metric to compare some popular DEVS-based
simulators
On a Catalogue of Metrics for Evaluating Commercial Cloud Services
Given the continually increasing amount of commercial Cloud services in the
market, evaluation of different services plays a significant role in
cost-benefit analysis or decision making for choosing Cloud Computing. In
particular, employing suitable metrics is essential in evaluation
implementations. However, to the best of our knowledge, there is not any
systematic discussion about metrics for evaluating Cloud services. By using the
method of Systematic Literature Review (SLR), we have collected the de facto
metrics adopted in the existing Cloud services evaluation work. The collected
metrics were arranged following different Cloud service features to be
evaluated, which essentially constructed an evaluation metrics catalogue, as
shown in this paper. This metrics catalogue can be used to facilitate the
future practice and research in the area of Cloud services evaluation.
Moreover, considering metrics selection is a prerequisite of benchmark
selection in evaluation implementations, this work also supplements the
existing research in benchmarking the commercial Cloud services.Comment: 10 pages, Proceedings of the 13th ACM/IEEE International Conference
on Grid Computing (Grid 2012), pp. 164-173, Beijing, China, September 20-23,
201
Sketch-based 3D Shape Retrieval using Convolutional Neural Networks
Retrieving 3D models from 2D human sketches has received considerable
attention in the areas of graphics, image retrieval, and computer vision.
Almost always in state of the art approaches a large amount of "best views" are
computed for 3D models, with the hope that the query sketch matches one of
these 2D projections of 3D models using predefined features.
We argue that this two stage approach (view selection -- matching) is
pragmatic but also problematic because the "best views" are subjective and
ambiguous, which makes the matching inputs obscure. This imprecise nature of
matching further makes it challenging to choose features manually. Instead of
relying on the elusive concept of "best views" and the hand-crafted features,
we propose to define our views using a minimalism approach and learn features
for both sketches and views. Specifically, we drastically reduce the number of
views to only two predefined directions for the whole dataset. Then, we learn
two Siamese Convolutional Neural Networks (CNNs), one for the views and one for
the sketches. The loss function is defined on the within-domain as well as the
cross-domain similarities. Our experiments on three benchmark datasets
demonstrate that our method is significantly better than state of the art
approaches, and outperforms them in all conventional metrics.Comment: CVPR 201
Load-Varying LINPACK: A Benchmark for Evaluating Energy Efficiency in High-End Computing
For decades, performance has driven the high-end computing (HEC) community. However, as highlighted in recent exascale studies that chart a path from petascale to exascale computing, power consumption is fast becoming the major design constraint in HEC. Consequently, the HEC community needs to address this issue in future petascale and exascale computing systems.
Current scientific benchmarks, such as LINPACK and SPEChpc, only evaluate HEC systems when running at full throttle, i.e., 100% workload, resulting in a focus on performance and ignoring the issues of power and energy consumption. In contrast, efforts like SPECpower evaluate the energy efficiency of a compute server at varying workloads. This is analogous to evaluating the energy efficiency (i.e., fuel efficiency) of an automobile at varying speeds (e.g., miles per gallon highway versus city). SPECpower, however, only evaluates the energy efficiency of a single compute server rather than a HEC system; furthermore, it is based on SPEC's Java Business Benchmarks (SPECjbb) rather than a scientific benchmark. Given the absence of a load-varying scientific benchmark to evaluate the energy efficiency of HEC systems at different workloads, we propose the load-varying LINPACK (LV-LINPACK) benchmark. In this paper, we identify application parameters that affect performance and provide a methodology to vary the workload of LINPACK, thus enabling a more rigorous study of energy efficiency in supercomputers, or more generally, HEC
Essential guidelines for computational method benchmarking
In computational biology and other sciences, researchers are frequently faced
with a choice between several computational methods for performing data
analyses. Benchmarking studies aim to rigorously compare the performance of
different methods using well-characterized benchmark datasets, to determine the
strengths of each method or to provide recommendations regarding suitable
choices of methods for an analysis. However, benchmarking studies must be
carefully designed and implemented to provide accurate, unbiased, and
informative results. Here, we summarize key practical guidelines and
recommendations for performing high-quality benchmarking analyses, based on our
experiences in computational biology.Comment: Minor update
Recommended from our members
Reactivity to sustainability metrics: A configurational study of motivation and capacity
Previous research on reactivity – defined as changing organisational behaviour to better conform to the criteria of measurement in response to being measured – has found significant variation in company responses towards sustainability metrics. We propose that reactivity is driven by dialogue, motivation and capacity in a configurational way. Empirically, we use fuzzy set Qualitative Comparative Analysis (fsQCA) to analyse company responses to the sustainability index FTSE4Good. We find evidence of complimentary and substitute effects between motivation and capacity. Based on these effects we develop a typology of reactivity to sustainability metrics, which also theorises the use of metrics as tools for performance feedback and the building of calculative capacity. We show that when reactivity is studied configurationally, we can identify previously underacknowledged types of responses. We discuss the theoretical and practical implications for studying and using sustainability metrics as governance tools for responsible behaviour
- …