Sphynx: ReLU-Efficient Network Design for Private Inference

The emergence of deep learning has been accompanied by privacy concerns surrounding users' data and service providers' models. We focus on private inference (PI), where the goal is to perform inference on a user's data sample using a service provider's model. Existing PI methods for deep networks enable cryptographically secure inference with little drop in functionality; however, they incur severe latency costs, primarily caused by non-linear network operations (such as ReLUs). This paper presents Sphynx, a ReLU-efficient network design method based on micro-search strategies for convolutional cell design. Sphynx achieves Pareto dominance over all existing private inference methods on CIFAR-100. We also design large-scale networks that support cryptographically private inference on Tiny-ImageNet and ImageNet

Speeding up deep neural architecture search for wearable activity recognition with early prediction of converged performance

Neural architecture search (NAS) has the potential to uncover more performant networks for human activity recognition from wearable sensor data. However, a naive evaluation of the search space is computationally expensive. We introduce neural regression methods for predicting the converged performance of a deep neural network (DNN) using validation performance in early epochs and topological and computational statistics. Our approach shows a significant improvement in predicting converged testing performance over a naive approach taking the ranking of the DNNs at an early epoch as an indication of their ranking on convergence. We apply this to the optimization of the convolutional feature extractor of an LSTM recurrent network using NAS with deep Q-learning, optimizing the kernel size, number of kernels, number of layers, and the connections between layers, allowing for arbitrary skip connections and dimensionality reduction with pooling layers. We find architectures which achieve up to 4% better F1 score on the recognition of gestures in the Opportunity dataset than our implementation of DeepConvLSTM and 0.8% better F1 score than our implementation of state-of-the-art model Attend and Discriminate, while reducing the search time by more than 90% over a random search. This opens the way to rapidly search for well-performing dataset-specific architectures. We describe the computational implementation of the system (software frameworks, computing resources) to enable replication of this work. Finally, we lay out several future research directions for NAS which the community may pursue to address ongoing challenges in human activity recognition, such as optimizing architectures to minimize power, minimize sensor usage, or minimize training data needs

Модуль ітераційної побудови моделей машинного навчання для розпізнавання іменованих сутностей

Магістерська дисертація: 92 с., 10 рис., 22 табл. і 40 джерел. Актуальність теми: вирішення задачі розпізнавання іменованих сутностей в україномовних текстах. Практично у більшості готових рішень досі не існує розроблених моделей для української мови. По-друге, даний модуль дасть змогу власноруч створювати необхідні навчальні дані, специфічні для конкретної задачі (новини або художні твори, наприклад) і із необхідними типами іменованих сутностей (не лише класичні типи – Персона чи Локація, а можливо, Модель машини, Посада тощо). Мета дослідження – спрощення процесу розробки моделей машинного навчання для вирішення задачі розпізнавання іменованих сутностей. Об’єкт дослідження – проблема інтелектуальної обробки природної мови. Предмет дослідження – методи вирішення проблеми розпізнавання іменованих сутностей для інтелектуальної обробки природніх текстів. Наукова новизна: розробка інструменту для побудови власних моделей машинного навчання, які можна використати для автоматизації процесу розмітки нових текстів.Master's thesis: 92 p., 10 f., 22 tables, 40 sources. Subject relevance − solving the problem of recognizing named entities in Ukrainian-language texts. Almost all solutions still do not have developed models for the Ukrainian language. Secondly, this module will allow user to create the necessary training data specific to a particular task (news or artwork, for example) and with the necessary types of named entities (not only the classic types - Person or Location, but possibly Machine Model, Position, etc). Purpose of research is to simplify the process of developing machine learning models to solve the problem of recognizing named entities. Object of research – problem of natural language processing. Subject of research − methods for solving the problem of recognizing named entities for natural language processing. Scientific novelty: development of a tool for building your own machine learning models that can be used to automate the process of new texts annotation