Machine Learning at Microsoft with ML .NET

Machine Learning is transitioning from an art and science into a technology available to every developer. In the near future, every application on every platform will incorporate trained models to encode data-based decisions that would be impossible for developers to author. This presents a significant engineering challenge, since currently data science and modeling are largely decoupled from standard software development processes. This separation makes incorporating machine learning capabilities inside applications unnecessarily costly and difficult, and furthermore discourage developers from embracing ML in first place. In this paper we present ML .NET, a framework developed at Microsoft over the last decade in response to the challenge of making it easy to ship machine learning models in large software applications. We present its architecture, and illuminate the application demands that shaped it. Specifically, we introduce DataView, the core data abstraction of ML .NET which allows it to capture full predictive pipelines efficiently and consistently across training and inference lifecycles. We close the paper with a surprisingly favorable performance study of ML .NET compared to more recent entrants, and a discussion of some lessons learned

GPU 환경에서 머신러닝 워크로드의 효율적인 실행

학위논문(박사) -- 서울대학교대학원 : 공과대학 컴퓨터공학부, 2023. 2. 전병곤.Machine learning (ML) workloads are becoming increasingly important in many types of real-world applications. We attribute this trend to the development of software systems for ML, which have facilitated the widespread adoption of heterogeneous accelerators such as GPUs. Todays ML software stack has made great improvements in terms of efficiency, however, not all use cases are well supported. In this dissertation, we study how to improve execution efficiency of ML workloads on GPUs from a software system perspective. We identify workloads where current systems for ML have inefficiencies in utilizing GPUs and devise new system techniques that handle those workloads efficiently. We first present Nimble, a ML execution engine equipped with carefully optimized GPU scheduling. The proposed scheduling techniques can be used to improve execution efficiency by up to 22.34×. Second, we propose Orca, an inference serving system specialized for Transformer-based generative models. By incorporating new scheduling and batching techniques, Orca significantly outperforms state-of-the-art systems – 36.9× throughput improvement at the same level of latency. The last topic of this dissertation is WindTunnel, a framework that translates classical ML pipelines into neural networks, providing GPU training capabilities for classical ML workloads. WindTunnel also allows joint training of pipeline components via backpropagation, resulting in improved accuracy over the original pipeline and neural network baselines.최근 경향을 보면 다양한 종류의 애플리케이션에서 머신 러닝(ML) 워크로드가 점 점 더 중요하게 활용되고 있다. 이는 ML용 시스템 소프트웨어의 개발을 통해 GPU 와 같은 이기종 가속기의 광범위한 활용이 가능해졌기 때문이다. 많은 연구자들의 관심 덕에 ML용 시스템 소프트웨어 스택은 분명 하루가 다르게 개선되고 있지만, 여전히 모든 사례에서 높은 효율성을 보여주지는 못한다. 이 학위논문에서는 시스 템 소프트웨어 관점에서 GPU 환경에서 ML 워크로드의 실행 효율성을 개선하는 방법을 연구한다. 구체적으로는 오늘날의 ML용 시스템이 GPU를 효율적으로 사 용하지 못하는 워크로드를 규명하고 더 나아가서 해당 워크로드를 효율적으로 처리할 수 있는 시스템 기술을 고안하는 것을 목표로 한다. 본 논문에서는 먼저 최적화된 GPU 스케줄링을 갖춘 ML 실행 엔진인 Nimble 을 소개한다. 새 스케줄링 기법을 통해 Nimble은 기존 대비 GPU 실행 효율성 을 최대 22.34배까지 향상시킬 수 있다. 둘째로 Transformer 기반의 생성 모델에 특화된 추론 서비스 시스템 Orca를 제안한다. 새로운 스케줄링 및 batching 기 술에 힘입어, Orca는 동일한 수준의 지연 시간을 기준으로 했을 때 기존 시스템 대비 36.9배 향상된 처리량을 보인다. 마지막으로 신경망을 사용하지 않는 고전 ML 파이프라인을 신경망으로 변환하는 프레임워크 WindTunnel을 소개한다. 이 를 통해 고전 ML 파이프라인 학습을 GPU를 사용해 진행할 수 있게 된다. 또한 WindTunnel은 gradient backpropagation을 통해 파이프라인의 여러 요소를 한 번에 공동으로 학습 할 수 있으며, 이를 통해 파이프라인의 정확도를 더 향상시킬 수 있음을 확인하였다.Chapter 1 Introduction 1 1.1 Motivation 1 1.2 Dissertation Overview 2 1.3 Previous Publications 4 1.4 Roadmap 5 Chapter 2 Background 6 2.1 ML Workloads 6 2.2 The GPU Execution Model 7 2.3 GPU Scheduling in ML Frameworks 8 2.4 Engine Scheduling in Inference Servers 10 2.5 Inference Procedure of Generative Models 11 Chapter 3 Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning 17 3.1 Introduction 17 3.2 Motivation 21 3.3 System Design 24 3.3.1 Ahead-of-time (AoT) Scheduling 25 3.3.2 Stream Assignment Algorithm 28 3.4 Evaluation 32 3.4.1 Inference Latency 36 3.4.2 Impact of Multi-stream Execution 36 3.4.3 Training Throughput 37 3.5 Summary 38 Chapter 4 Orca: A Distributed Serving System for Transformer-Based Generative Models 40 4.1 Introduction 40 4.2 Challenges and Proposed Solutions 44 4.3 Orca System Design 51 4.3.1 Distributed Architecture 51 4.3.2 Scheduling Algorithm 54 4.4 Implementation 60 4.5 Evaluation 61 4.5.1 Engine Microbenchmark 63 4.5.2 End-to-end Performance 66 4.6 Summary 71 Chapter 5 WindTunnel: Towards Differentiable ML Pipelines Beyond a Single Model 72 5.1 Introduction 72 5.2 Pipeline Translation 78 5.2.1 Translating Arithmetic Operators 80 5.2.2 Translating Algorithmic Operators: GBDT 81 5.2.3 Translating Algorithmic Operators for Categorical Features 85 5.2.4 Fine-Tuning 87 5.3 Implementation 87 5.4 Experiments 88 5.4.1 Experimental Setup 89 5.4.2 Overall Performance 94 5.4.3 Ablation Study 95 5.5 Summary 98 Chapter 6 Related Work 99 Chapter 7 Conclusion 105 Bibliography 107 Appendix A Appendix: Nimble 131 A.1 Proofs on the Stream Assignment Algorithm of Nimble 131 A.1.1 Proof of Theorem 1 132 A.1.2 Proof of Theorem 2 134 A.1.3 Proof of Theorem 3 135 A.1.4 Time Complexity Analysis 137 A.2 Evaluation Results on Various GPUs 139 A.3 Evaluation Results on Different Training Batch Sizes 139박

Service Abstractions for Scalable Deep Learning Inference at the Edge

Deep learning driven intelligent edge has already become a reality, where millions of mobile, wearable, and IoT devices analyze real-time data and transform those into actionable insights on-device. Typical approaches for optimizing deep learning inference mostly focus on accelerating the execution of individual inference tasks, without considering the contextual correlation unique to edge environments and the statistical nature of learning-based computation. Specifically, they treat inference workloads as individual black boxes and apply canonical system optimization techniques, developed over the last few decades, to handle them as yet another type of computation-intensive applications. As a result, deep learning inference on edge devices still face the ever increasing challenges of customization to edge device heterogeneity, fuzzy computation redundancy between inference tasks, and end-to-end deployment at scale. In this thesis, we propose the first framework that automates and scales the end-to-end process of deploying efficient deep learning inference from the cloud to heterogeneous edge devices. The framework consists of a series of service abstractions that handle DNN model tailoring, model indexing and query, and computation reuse for runtime inference respectively. Together, these services bridge the gap between deep learning training and inference, eliminate computation redundancy during inference execution, and further lower the barrier for deep learning algorithm and system co-optimization. To build efficient and scalable services, we take a unique algorithmic approach of harnessing the semantic correlation between the learning-based computation. Rather than viewing individual tasks as isolated black boxes, we optimize them collectively in a white box approach, proposing primitives to formulate the semantics of the deep learning workloads, algorithms to assess their hidden correlation (in terms of the input data, the neural network models, and the deployment trials) and merge common processing steps to minimize redundancy

Believability Assessment and Modelling in Video Games

Artificial Intelligence remains one of the most sought after subjects in computer science to this day. One of its subjects, and the focus of this thesis, is its application to video games as believable agents. This means focusing on implementing agents that behave like us rather than simply attempting to win, whether that means cooperating or competing like we do. Success in building more human-like characters can enhance immersion and enjoyment in games, thus potentially increasing its gameplay value. Ultimately, bringing benefits to the industry and academia. However, believability is a hard concept to define. It depends on how and what one considers to be ``believable'', which is often very subjective. This means that developing believable agents remains a sought out, albeit difficult, challenge. There are many approaches to development ranging from finite state machines or imitation learning to emotional models, with no single solution to creating a human-like agent. This problems remains when attempting to assess these solutions as well. Assessing the believability of agents, characters and simulated actors is also a core challenge for human-like behaviour. While numerous approaches are suggested in the literature, there is not a dominant solution for evaluation either. In addition, assessment rarely receives as much attention as development or modelling do. Mostly, it comes as a necessity of evaluating agents rather than focusing on how its process could affect the outcome of the evaluation itself. This thesis takes a different approach to developing believability and its assessment. For starters, it explores assessment first. In previous years, several researchers have tried to find ways of assessing human-like behaviour in games through adaptations of Turing Tests on their agents. Given the small pool of diversity of the explored parameters in believability assessment and a focus on programming the bots, this thesis starts by exploring different parameters for evaluating believability in video games. The objective of this work is to analyze the different ways believability can be assessed, for humans and non-player characters (NPCs) by comparing how results between them and scores are affected in both when changing the parameters. This thesis also explores the concept of believability and its need in video games in general. Another aspect of assessment explored in this thesis is believability's overall representation. Past research shows methodologies being limited to discrete and low-granularity representations of believable behaviour. This work will focus, for the first time, in viewing believability as a time-continuous phenomenon and explore the suitability of two different affect annotation schemes for its assessment. These techniques are also compared to previously used discrete methodologies, to understand how moment-to-moment assessment can contribute to these. In addition, this thesis studies the degree to which we can predict character believability in a continuous fashion. This is achieved by training random forest models to predict believability based on annotations of the context extracted from a game. It is then that this thesis tackles development. For this work, different solutions are combined into one and in a different order: this time-continuous data based on peoples' assessment of believability is modelled and integrated into a game agent to affect its behaviour. This results in a final comparison between two agents, where one uses a believability biased model and the other does not. Showing that biasing agents' behaviour with assessment data can increase their overall believability