47,494 research outputs found
Adaptive Momentum for Neural Network Optimization
In this thesis, we develop a novel and efficient algorithm for optimizing neural networks inspired by a recently proposed geodesic optimization algorithm. Our algorithm, which we call Stochastic Geodesic Optimization (SGeO), utilizes an adaptive coefficient on top of Polyaks Heavy Ball method effectively controlling the amount of weight put on the previous update to the parameters based on the change of direction in the optimization path. Experimental results on strongly convex functions with Lipschitz gradients and deep Autoencoder benchmarks show that SGeO reaches lower errors than established first-order methods and competes well with lower or similar errors to a recent second-order method called K-FAC (Kronecker-Factored Approximate Curvature). We also incorporate Nesterov style lookahead gradient into our algorithm (SGeO-N) and observe notable improvements. We believe that our research will open up new directions for high-dimensional neural network optimization where combining the efficiency of first-order methods and the effectiveness of second-order methods proves a promising avenue to explore
Multivariate time series analysis for short-term forecasting of ground level ozone (O3) in Malaysia
The declining of air quality mostly affects the elderly, children, people with asthma,
as well as a restriction on outdoor activities. Therefore, there is an importance to
provide a statistical modelling to forecast the future values of surface layer ozone (O3)
concentration. The objectives of this study are to obtain the best multivariate time
series (MTS) model and develop an online air quality forecasting system for O3
concentration in Malaysia. The implementations of MTS model improve the recent
statistical model on air quality for short-term prediction. Ten air quality monitoring
stations situated at four (4) different types of location were selected in this study. The
first type is industrial represent by Pasir Gudang, Perai, and Nilai, second type is urban
represent by Kuala Terengganu, Kota Bharu, and Alor Setar. The third is suburban
located in Banting, Kangar, and Tanjung Malim, also the only background station at
Jerantut. The hourly record data from 2010 to 2017 were used to assess the
characteristics and behaviour of O3 concentration. Meanwhile, the monthly record data
of O3, particulate matter (PM10), nitrogen dioxide (NO2), sulphur dioxide (SO2),
carbon monoxide (CO), temperature (T), wind speed (WS), and relative humidity (RH)
were used to examine the best MTS models. Three methods of MTS namely vector
autoregressive (VAR), vector moving average (VMA), and vector autoregressive
moving average (VARMA), has been applied in this study. Based on the performance
error, the most appropriate MTS model located in Pasir Gudang, Kota Bharu and
Kangar is VAR(1), Kuala Terengganu and Alor Setar for VAR(2), Perai and Nilai for
VAR(3), Tanjung Malim for VAR(4) and Banting for VAR(5). Only Jerantut obtained
the VMA(2) as the best model. The lowest root mean square error (RMSE) and
normalized absolute error is 0.0053 and <0.0001 which is for MTS model in Perai and
Kuala Terengganu, respectively. Meanwhile, for mean absolute error (MAE), the
lowest is in Banting and Jerantut at 0.0013. The online air quality forecasting system
for O3 was successfully developed based on the best MTS models to represent each
monitoring station
- …