4 research outputs found
CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarking
We present CodeReef - an open platform to share all the components necessary
to enable cross-platform MLOps (MLSysOps), i.e. automating the deployment of ML
models across diverse systems in the most efficient way. We also introduce the
CodeReef solution - a way to package and share models as non-virtualized,
portable, customizable and reproducible archive files. Such ML packages include
JSON meta description of models with all dependencies, Python APIs, CLI actions
and portable workflows necessary to automatically build, benchmark, test and
customize models across diverse platforms, AI frameworks, libraries, compilers
and datasets. We demonstrate several CodeReef solutions to automatically build,
run and measure object detection based on SSD-Mobilenets, TensorFlow and COCO
dataset from the latest MLPerf inference benchmark across a wide range of
platforms from Raspberry Pi, Android phones and IoT devices to data centers.
Our long-term goal is to help researchers share their new techniques as
production-ready packages along with research papers to participate in
collaborative and reproducible benchmarking, compare the different
ML/software/hardware stacks and select the most efficient ones on a Pareto
frontier using online CodeReef dashboards.Comment: Presented at the 1st Workshop on MLOps Systems co-located with the
3rd Conference on Machine Learning and Systems (MLSys'20) in Austin, TX, USA:
https://mlops-systems.github.io . A live interactive demo:
https://CodeReef.ai/dem
The Collective Knowledge project: making ML models more portable and reproducible with open APIs, reusable best practices and MLOps
This article provides an overview of the Collective Knowledge technology (CK
or cKnowledge). CK attempts to make it easier to reproduce ML&systems research,
deploy ML models in production, and adapt them to continuously changing data
sets, models, research techniques, software, and hardware. The CK concept is to
decompose complex systems and ad-hoc research projects into reusable
sub-components with unified APIs, CLI, and JSON meta description. Such
components can be connected into portable workflows using DevOps principles
combined with reusable automation actions, software detection plugins, meta
packages, and exposed optimization parameters. CK workflows can automatically
plug in different models, data and tools from different vendors while building,
running and benchmarking research code in a unified way across diverse
platforms and environments. Such workflows also help to perform whole system
optimization, reproduce results, and compare them using public or private
scoreboards on the CK platform (https://cKnowledge.io). For example, the
modular CK approach was successfully validated with industrial partners to
automatically co-design and optimize software, hardware, and machine learning
models for reproducible and efficient object detection in terms of speed,
accuracy, energy, size, and other characteristics. The long-term goal is to
simplify and accelerate the development and deployment of ML models and systems
by helping researchers and practitioners to share and reuse their knowledge,
experience, best practices, artifacts, and techniques using open CK APIs.Comment: arXiv admin note: text overlap with arXiv:2001.0793
Collective Knowledge: organizing research projects as a database of reusable components and portable workflows with common APIs
This article provides the motivation and overview of the Collective Knowledge
framework (CK or cKnowledge). The CK concept is to decompose research projects
into reusable components that encapsulate research artifacts and provide
unified application programming interfaces (APIs), command-line interfaces
(CLIs), meta descriptions and common automation actions for related artifacts.
The CK framework is used to organize and manage research projects as a database
of such components.
Inspired by the USB "plug and play" approach for hardware, CK also helps to
assemble portable workflows that can automatically plug in compatible
components from different users and vendors (models, datasets, frameworks,
compilers, tools). Such workflows can build and run algorithms on different
platforms and environments in a unified way using the universal CK program
pipeline with software detection plugins and the automatic installation of
missing packages.
This article presents a number of industrial projects in which the modular CK
approach was successfully validated in order to automate benchmarking,
auto-tuning and co-design of efficient software and hardware for machine
learning (ML) and artificial intelligence (AI) in terms of speed, accuracy,
energy, size and various costs. The CK framework also helped to automate the
artifact evaluation process at several computer science conferences as well as
to make it easier to reproduce, compare and reuse research techniques from
published papers, deploy them in production, and automatically adapt them to
continuously changing datasets, models and systems.
The long-term goal is to accelerate innovation by connecting researchers and
practitioners to share and reuse all their knowledge, best practices,
artifacts, workflows and experimental results in a common, portable and
reproducible format at https://cKnowledge.io .Comment: Accepted for Philosophical Transactions of the Royal Society. arXiv
admin note: text overlap with arXiv:2006.0716
MLModelScope: A Distributed Platform for Model Evaluation and Benchmarking at Scale
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced
at such a rapid pace that researchers are hard-pressed to analyze and study
them. The complicated procedures for evaluating innovations, along with the
lack of standard and efficient ways of specifying and provisioning ML/DL
evaluation, is a major "pain point" for the community. This paper proposes
MLModelScope, an open-source, framework/hardware agnostic, extensible and
customizable design that enables repeatable, fair, and scalable model
evaluation and benchmarking. We implement the distributed design with support
for all major frameworks and hardware, and equip it with web, command-line, and
library interfaces. To demonstrate MLModelScope's capabilities we perform
parallel evaluation and show how subtle changes to model evaluation pipeline
affects the accuracy and HW/SW stack choices affect performance