19,300 research outputs found
On a Catalogue of Metrics for Evaluating Commercial Cloud Services
Given the continually increasing amount of commercial Cloud services in the
market, evaluation of different services plays a significant role in
cost-benefit analysis or decision making for choosing Cloud Computing. In
particular, employing suitable metrics is essential in evaluation
implementations. However, to the best of our knowledge, there is not any
systematic discussion about metrics for evaluating Cloud services. By using the
method of Systematic Literature Review (SLR), we have collected the de facto
metrics adopted in the existing Cloud services evaluation work. The collected
metrics were arranged following different Cloud service features to be
evaluated, which essentially constructed an evaluation metrics catalogue, as
shown in this paper. This metrics catalogue can be used to facilitate the
future practice and research in the area of Cloud services evaluation.
Moreover, considering metrics selection is a prerequisite of benchmark
selection in evaluation implementations, this work also supplements the
existing research in benchmarking the commercial Cloud services.Comment: 10 pages, Proceedings of the 13th ACM/IEEE International Conference
on Grid Computing (Grid 2012), pp. 164-173, Beijing, China, September 20-23,
201
Building an Expert System for Evaluation of Commercial Cloud Services
Commercial Cloud services have been increasingly supplied to customers in
industry. To facilitate customers' decision makings like cost-benefit analysis
or Cloud provider selection, evaluation of those Cloud services are becoming
more and more crucial. However, compared with evaluation of traditional
computing systems, more challenges will inevitably appear when evaluating
rapidly-changing and user-uncontrollable commercial Cloud services. This paper
proposes an expert system for Cloud evaluation that addresses emerging
evaluation challenges in the context of Cloud Computing. Based on the knowledge
and data accumulated by exploring the existing evaluation work, this expert
system has been conceptually validated to be able to give suggestions and
guidelines for implementing new evaluation experiments. As such, users can
conveniently obtain evaluation experiences by using this expert system, which
is essentially able to make existing efforts in Cloud services evaluation
reusable and sustainable.Comment: 8 page, Proceedings of the 2012 International Conference on Cloud and
Service Computing (CSC 2012), pp. 168-175, Shanghai, China, November 22-24,
201
A Factor Framework for Experimental Design for Performance Evaluation of Commercial Cloud Services
Given the diversity of commercial Cloud services, performance evaluations of
candidate services would be crucial and beneficial for both service customers
(e.g. cost-benefit analysis) and providers (e.g. direction of service
improvement). Before an evaluation implementation, the selection of suitable
factors (also called parameters or variables) plays a prerequisite role in
designing evaluation experiments. However, there seems a lack of systematic
approaches to factor selection for Cloud services performance evaluation. In
other words, evaluators randomly and intuitively concerned experimental factors
in most of the existing evaluation studies. Based on our previous taxonomy and
modeling work, this paper proposes a factor framework for experimental design
for performance evaluation of commercial Cloud services. This framework
capsules the state-of-the-practice of performance evaluation factors that
people currently take into account in the Cloud Computing domain, and in turn
can help facilitate designing new experiments for evaluating Cloud services.Comment: 8 pages, Proceedings of the 4th International Conference on Cloud
Computing Technology and Science (CloudCom 2012), pp. 169-176, Taipei,
Taiwan, December 03-06, 201
Towards a Taxonomy of Performance Evaluation of Commercial Cloud Services
Cloud Computing, as one of the most promising computing paradigms, has become
increasingly accepted in industry. Numerous commercial providers have started
to supply public Cloud services, and corresponding performance evaluation is
then inevitably required for Cloud provider selection or cost-benefit analysis.
Unfortunately, inaccurate and confusing evaluation implementations can be often
seen in the context of commercial Cloud Computing, which could severely
interfere and spoil evaluation-related comprehension and communication. This
paper introduces a taxonomy to help profile and standardize the details of
performance evaluation of commercial Cloud services. Through a systematic
literature review, we constructed the taxonomy along two dimensions by
arranging the atomic elements of Cloud-related performance evaluation. As such,
this proposed taxonomy can be employed both to analyze existing evaluation
practices through decomposition into elements and to design new experiments
through composing elements for evaluating performance of commercial Cloud
services. Moreover, through smooth expansion, we can continually adapt this
taxonomy to the more general area of evaluation of Cloud Computing.Comment: 8 pages, Proceedings of the 5th International Conference on Cloud
Computing (IEEE CLOUD 2012), pp. 344-351, Honolulu, Hawaii, USA, June 24-29,
201
On Evaluating Commercial Cloud Services: A Systematic Review
Background: Cloud Computing is increasingly booming in industry with many
competing providers and services. Accordingly, evaluation of commercial Cloud
services is necessary. However, the existing evaluation studies are relatively
chaotic. There exists tremendous confusion and gap between practices and theory
about Cloud services evaluation. Aim: To facilitate relieving the
aforementioned chaos, this work aims to synthesize the existing evaluation
implementations to outline the state-of-the-practice and also identify research
opportunities in Cloud services evaluation. Method: Based on a conceptual
evaluation model comprising six steps, the Systematic Literature Review (SLR)
method was employed to collect relevant evidence to investigate the Cloud
services evaluation step by step. Results: This SLR identified 82 relevant
evaluation studies. The overall data collected from these studies essentially
represent the current practical landscape of implementing Cloud services
evaluation, and in turn can be reused to facilitate future evaluation work.
Conclusions: Evaluation of commercial Cloud services has become a world-wide
research topic. Some of the findings of this SLR identify several research gaps
in the area of Cloud services evaluation (e.g., the Elasticity and Security
evaluation of commercial Cloud services could be a long-term challenge), while
some other findings suggest the trend of applying commercial Cloud services
(e.g., compared with PaaS, IaaS seems more suitable for customers and is
particularly important in industry). This SLR study itself also confirms some
previous experiences and reveals new Evidence-Based Software Engineering (EBSE)
lessons
- …