LSTM with Working Memory

Previous RNN architectures have largely been superseded by LSTM, or "Long Short-Term Memory". Since its introduction, there have been many variations on this simple design. However, it is still widely used and we are not aware of a gated-RNN architecture that outperforms LSTM in a broad sense while still being as simple and efficient. In this paper we propose a modified LSTM-like architecture. Our architecture is still simple and achieves better performance on the tasks that we tested on. We also introduce a new RNN performance benchmark that uses the handwritten digits and stresses several important network capabilities.Comment: Accepted at IJCNN 201

Forecasting JPFA Share Price using Long Short Term Memory Neural Network

To invest or buy and sell on the stock exchange requires understanding in the field of data analysis. The movement of the curve in the stock market is very dynamic, so it requires data modeling to predict stock prices in order to get prices with a high degree of accuracy. Machine Learning currently has a good level of accuracy in processing and predicting data. In this study, we modeled data using the Long-Short Term Memory (LSTM) algorithm to predict the stock price of a company called Japfa Comfeed. The main objective of this journal is to analyze the level of accuracy of Machine Learning algorithms in predicting stock price data and to analyze the number of epochs in forming an optimal model. The results of our research show that the LSTM algorithm has a good level of accurate prediction shown in mape values and the data model obtained on variations in epochs values. All optimization models show that the higher the epoch value, the lower the loss value. Adam's Optimization Model is the model with the highest accuracy value of 98.44%

ANALISIS PREDIKSI HARGA SAHAM SEKTOR PERBANKAN MENGGUNAKAN ALGORITMA LONG-SHORT TERMS MEMORY (LSTM)

AbstractInvesting, buying or selling activity on the stock exchange requires knowledge and skill in the field of data analysis. The movement of the curve in the stock market place is very dynamic, hence it requires data modelling to predict stock prices in order to get a price with a high degree of accuracy. Currently, machine learning has a good level of accuracy in processing and predicting data. In this work, we proposed the data modelling using the Long-Short Term Memory (LSTM) algorithm to predict stock prices. The main purpose for this research is to analyze the accuracy of the machine learning algorithm in predicting stock price data and analyzing the number of epochs in the optimal model formation. The results of our study indicate that the LSTM algorithm has an accurate level of prediction as indicated by the RMSE value and the data model obtained the variation of the epochs value.Keywords : LSTM Algorithm, Stock Price, Analysis Prediction, Machine LearningUntuk melakukan investasi atau jual beli di bursa saham memerlukan pemahaman dibidang analisis data. Pergerakan kurva pada pasar saham sangat dinamis, sehingga memerlukan pemodelan data untuk melakukan prediksi harga saham agar mendapatkan harga dengan tingkat akurasi yang tinggi. Machine Learning pada saat ini memiliki tingkat keakuratan yang baik dalam mengolah dan memprediksi data. Pada penelitian ini kami melakukan pemodelan data menggunakan algoritma Long-Short Term Memory (LSTM) untuk memprediksi harga saham. Tujuan utama pada jurnal ini adalah untuk menganalisis tingkat keakuratan algoritma Machine Learning dalam melakukan prediksi data harga saham serta melakukan analisis pada banyaknya epochs dalam pembentukan model yang optimal. Hasil penelitian kami menunjukkan bahwa algoritma LSTM memiliki tingkat prediksi yangg akurat dengan ditunjukkan pada nilai RMSE serta model data yang di dapatkan pada variasi nilai epochs.Kata Kunci : Algoritma LSTM, Harga Saham, Analisis Prediksi, Machine Learnin

Faster and Cheaper Energy Demand Forecasting at Scale

Energy demand forecasting is one of the most challenging tasks for grids operators. Many approaches have been suggested over the years to tackle it. Yet, those still remain too expensive to train in terms of both time and computational resources, hindering their adoption as customers behaviors are continuously evolving. We introduce Transplit, a new lightweight transformer-based model, which significantly decreases this cost by exploiting the seasonality property and learning typical days of power demand. We show that Transplit can be run efficiently on CPU and is several hundred times faster than state-of-the-art predictive models, while performing as well

MomentumRNN: Integrating Momentum into Recurrent Neural Networks

Designing deep neural networks is an art that often involves an expensive search over candidate architectures. To overcome this for recurrent neural nets (RNNs), we establish a connection between the hidden state dynamics in an RNN and gradient descent (GD). We then integrate momentum into this framework and propose a new family of RNNs, called {\em MomentumRNNs}. We theoretically prove and numerically demonstrate that MomentumRNNs alleviate the vanishing gradient issue in training RNNs. We study the momentum long-short term memory (MomentumLSTM) and verify its advantages in convergence speed and accuracy over its LSTM counterpart across a variety of benchmarks. We also demonstrate that MomentumRNN is applicable to many types of recurrent cells, including those in the state-of-the-art orthogonal RNNs. Finally, we show that other advanced momentum-based optimization methods, such as Adam and Nesterov accelerated gradients with a restart, can be easily incorporated into the MomentumRNN framework for designing new recurrent cells with even better performance. The code is available at https://github.com/minhtannguyen/MomentumRNN.Comment: 21 pages, 11 figures, Accepted for publication at Advances in Neural Information Processing Systems (NeurIPS) 202

SAM: a unified self-adaptive multicompartmental spiking neuron model for learning with working memory

Working memory is a fundamental feature of biological brains for perception, cognition, and learning. In addition, learning with working memory, which has been show in conventional artificial intelligence systems through recurrent neural networks, is instrumental to advanced cognitive intelligence. However, it is hard to endow a simple neuron model with working memory, and to understand the biological mechanisms that have resulted in such a powerful ability at the neuronal level. This article presents a novel self-adaptive multicompartment spiking neuron model, referred to as SAM, for spike-based learning with working memory. SAM integrates four major biological principles including sparse coding, dendritic non-linearity, intrinsic self-adaptive dynamics, and spike-driven learning. We first describe SAM’s design and explore the impacts of critical parameters on its biological dynamics. We then use SAM to build spiking networks to accomplish several different tasks including supervised learning of the MNIST dataset using sequential spatiotemporal encoding, noisy spike pattern classification, sparse coding during pattern classification, spatiotemporal feature detection, meta-learning with working memory applied to a navigation task and the MNIST classification task, and working memory for spatiotemporal learning. Our experimental results highlight the energy efficiency and robustness of SAM in these wide range of challenging tasks. The effects of SAM model variations on its working memory are also explored, hoping to offer insight into the biological mechanisms underlying working memory in the brain. The SAM model is the first attempt to integrate the capabilities of spike-driven learning and working memory in a unified single neuron with multiple timescale dynamics. The competitive performance of SAM could potentially contribute to the development of efficient adaptive neuromorphic computing systems for various applications from robotics to edge computing

Derin öğrenme tabanlı elektrikli ev aletleri veri setinin sınıflandırılması

Elektriğe bağlı olan her ev aletinin akım/gerilim karakteristiği farklı olduğundan, bu cihazların her birinin şebekeden çektiği gücün özelliği farklı olmaktadır. Bu nedenle şebekeye bağlı olan cihazın tipinin tespiti cihazın şebekeden çektiği harmoniğin tespit edilmesinde ve de düzeltilmesinde önemli rol oynamaktadır. Bu çalışma kapsamında farklı derin öğrenme teknikleri kullanılarak “ACS-F2 Elektrikli Ev Aletleri Veri Seti” üzerinde sınıflandırma gerçekleştirilmiştir. ACS-F2 veri setinde toplamda 15 farklı sınıf için 225 cihaz bulunmasına karşın, çalışma kapsamında yapılan ön işlemler ile veri setindeki sınıf sayısı 14’e indirilmiştir. Sonrasında LSTM, FeedForwardNet, çift yönlü LSTM( Bi-LSTM) ve parametreleri genetik algoritma tarafından optimize edilmiş Bi-LSTM kullanılarak sınıflandırma yapılarak sınıflandırıcının performansları karşılaştırılmıştır. Yapılan çalışma kapsamında parametreleri optimize edilmiş sınıflandırıcının diğer yöntemlerden daha başarılı sonuçlar elde ettiği gözlenmiştir.Since the current/voltage characteristics of each electrical appliance are different, the power consumption of devices are specific. Therefore, determining the type of electrical appliance connected to the network is crucial for the detection and correction of the device based harmonics. In this study, the classification of electrical appliances is carried out on the "ACS-F2 Electrical Appliances Dataset" using different deep learning algorithms. Although there are 225 devices for 15 different classes in the ACS-F2 data set, the number of classes in the data set has been reduced to 14 with the preprocessesing step. The LSTM, FeedForwardNet, Bi-LSTM and Ga+Bi-LSTM, models are then built to classifiy electrical appliances. It is observed that the GA+Bi-LSTM classifier, which has %94 classifiaction accuracy, overcomes among the models