1 research outputs found
BenchCouncil's View on Benchmarking AI and Other Emerging Workloads
This paper outlines BenchCouncil's view on the challenges, rules, and vision
of benchmarking modern workloads like Big Data, AI or machine learning, and
Internet Services. We conclude the challenges of benchmarking modern workloads
as FIDSS (Fragmented, Isolated, Dynamic, Service-based, and Stochastic), and
propose the PRDAERS benchmarking rules that the benchmarks should be specified
in a paper-and-pencil manner, relevant, diverse, containing different levels of
abstractions, specifying the evaluation metrics and methodology, repeatable,
and scaleable. We believe proposing simple but elegant abstractions that help
achieve both efficiency and general-purpose is the final target of benchmarking
in future, which may be not pressing. In the light of this vision, we shortly
discuss BenchCouncil's related projects.Comment: 7 page