Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks

Much of the focus in the area of knowledge distillation has been on distilling knowledge from a larger teacher network to a smaller student network. However, there has been little research on how the concept of distillation can be leveraged to distill the knowledge encapsulated in the training data itself into a reduced form. In this study, we explore the concept of progressive label distillation, where we leverage a series of teacher-student network pairs to progressively generate distilled training data for learning deep neural networks with greatly reduced input dimensions. To investigate the efficacy of the proposed progressive label distillation approach, we experimented with learning a deep limited vocabulary speech recognition network based on generated 500ms input utterances distilled progressively from 1000ms source training data, and demonstrated a significant increase in test accuracy of almost 78% compared to direct learning.Comment: 9 page

Differential Privacy of Aggregated DC Optimal Power Flow Data

We consider the problem of privately releasing aggregated network statistics obtained from solving a DC optimal power flow (OPF) problem. It is shown that the mechanism that determines the noise distribution parameters are linked to the topology of the power system and the monotonicity of the network. We derive a measure of "almost" monotonicity and show how it can be used in conjunction with a linear program in order to release aggregated OPF data using the differential privacy framework.Comment: Accepted by 2019 American Control Conference (ACC

A Geometric Approach of Gradient Descent Algorithms in Neural Networks

In this paper, we present an original geometric framework to analyze the convergence properties of gradient descent trajectories in the context of linear neural networks. Built upon a key invariance property induced by the network structure, we propose a conjecture called \emph{overfitting conjecture} stating that, for almost every training data, the corresponding gradient descent trajectory converges to a global minimum, for almost every initial condition. This would imply that, for linear neural networks of an arbitrary number of hidden layers, the solution achieved by simple gradient descent algorithm is equivalent to that of least square estimation. Our first result consists in establishing, in the case of linear networks of arbitrary depth, convergence of gradient descent trajectories to critical points of the loss function. Our second result is the proof of the \emph{overfitting conjecture} in the case of single-hidden-layer linear networks with an argument based on the notion of normal hyperbolicity and under a generic property on the training data (i.e., holding for almost every training data).Comment: Preprint. Work in progres

Meteorological and dynamical requirements for MST radar networks: Waves

Studies of wave motions using the MST radar have concentrated on single station time series analyses of gravity waves and tides. Since these radars collect high time resolution data they have the potential to become a significant tool for mesoscale research. In addition, radars are operated almost continuously unattended and, consequently, data sets are available for analyzing longer period wave motions such as tides and planetary scale waves. Although there is much to learn from single station data, the possibilities of new knowledge from a network of radars is exciting. The scales of wave motions in the atmosphere cover a broad range. Consequently the choice of a radar network depends to a large extent on the types of wave motions that are studied. There are many outstanding research problems that would benefit from observations from a MST radar network. In particular, there is a strong need for measurements of gravity wave parameters and equatorial wave motions. Some of the current problems in wave dynamics are discussed

Design and development of an emulated human cognition using novel 3D neural networks

This paper describes the development of an Emulated Human Cognition (EHC) which is designed and based on a replicated human brain with a right- and a left- hand lobe, one a deductive side and the other a generic one. Right-hand lobe consists of a newly designed Artificial Neural Network (ANN) with a multi-hidden layer topology. Left-hand lobe is a newly designed 3-dimensional cellular neural network. The input variables presented to the EHC are immediately analysed for it to decide which lobe should be activated. The EHC, when fully developed, has almost an unlimited memory capacity and is capable of immediate recall of any data in its almost unlimited memory locations. EHC has been used in several applications where neural networks have been used to establish relationship between two or more sets of variables. In this paper the EHC has been used to forecast demand for a given product