Understanding Gradient Descent on Edge of Stability in Deep Learning

Deep learning experiments by Cohen et al. [2021] using deterministic Gradient Descent (GD) revealed an Edge of Stability (EoS) phase when learning rate (LR) and sharpness (i.e., the largest eigenvalue of Hessian) no longer behave as in traditional optimization. Sharpness stabilizes around $2/$LR and loss goes up and down across iterations, yet still with an overall downward trend. The current paper mathematically analyzes a new mechanism of implicit regularization in the EoS phase, whereby GD updates due to non-smooth loss landscape turn out to evolve along some deterministic flow on the manifold of minimum loss. This is in contrast to many previous results about implicit bias either relying on infinitesimal updates or noise in gradient. Formally, for any smooth function $L$ with certain regularity condition, this effect is demonstrated for (1) Normalized GD, i.e., GD with a varying LR $\eta_t =\frac{\eta}{\| \nabla L(x(t)) \|}$ and loss $L$; (2) GD with constant LR and loss $\sqrt{L- \min_x L(x)}$. Both provably enter the Edge of Stability, with the associated flow on the manifold minimizing $\lambda_{1}(\nabla^2 L)$. The above theoretical results have been corroborated by an experimental study.Comment: 63 pages. This paper has been accepted for conference proceedings in the 39th International Conference on Machine Learning (ICML), 202

An Expert System Based on Least Mean Square and Neural Network for Classification of Power System Disturbances

This paper proposes a new solution method for power quality (PQ) classification using least mean square (LMS) and neural network (NN). The proposed hybrid LMS-NN method comprises of LMS based effective feature extractor and PQ classifier based on a multi layer perceptron neural network (MLP-NN). First, the LMS method is employed to estimate the efficient features such as amplitude, slope, and harmonic indication from the measured voltage signals where the developed structure is merely simple. Further, the PQ classification is executed with the aid of MLP-NN. The different voltage signals analyzed for this research work are pure sine, sag, swell, outage, harmonics, sag with harmonics, and swell with harmonics. The performance and efficiency of the presented hybrid LMS-NN classifier is assessed by testing total 1400 voltage samples which are simulated based on PQ disturbance model. The rate of average correct classification is about 96.71 for the different PQ disturbance signals under noise conditions

Fairness in systems based on multiparty interactions

In the context of the Multiparty Interaction Model, fairness is used to insure that an interaction that is enabled sufficiently often in a concurrent program will eventually be selected for execution. Unfortunately, this notion does not take conspiracies into account, i.e. situations in which an interaction never becomes enabled because of an unfortunate interleaving of independent actions; furthermore, eventual execution is usually too weak for practical purposes since this concept can only be used in the context of infinite executions. In this article, we present a new fairness notion, k-conspiracy-free fairness, that improves on others because it takes finite executions into account, alleviates conspiracies that are not inherent to a program, and k may be set a priori to control its goodness to address the above-mentioned problems.Ministerio de Ciencia y Tecnología TIC-2000-1106-C02-01Ministerio de Ciencia y Tecnología FIT-150100-2001-78Ministerio de Ciencia y Tecnología TAMANSI PCB-02-00

Bandwidth Allocation Mechanism based on Users' Web Usage Patterns for Campus Networks

Managing the bandwidth in campus networks becomes a challenge in recent years. The limited bandwidth resource and continuous growth of users make the IT managers think on the strategies concerning bandwidth allocation. This paper introduces a mechanism for allocating bandwidth based on the users’ web usage patterns. The main purpose is to set a higher bandwidth to the users who are inclined to browsing educational websites compared to those who are not. In attaining this proposed technique, some stages need to be done. These are the preprocessing of the weblogs, class labeling of the dataset, computation of the feature subspaces, training for the development of the ANN for LDA/GSVD algorithm, visualization, and bandwidth allocation. The proposed method was applied to real weblogs from university’s proxy servers. The results indicate that the proposed method is useful in classifying those users who used the internet in an educational way and those who are not. Thus, the developed ANN for LDA/GSVD algorithm outperformed the existing algorithm up to 50% which indicates that this approach is efficient. Further, based on the results, few users browsed educational contents. Through this mechanism, users will be encouraged to use the internet for educational purposes. Moreover, IT managers can make better plans to optimize the distribution of bandwidth