Text prediction recurrent neural networks using long shortterm memory-dropout

"Unit short-term memory (LSTM) is a type of recurrent neural network (RNN) whose sequence-based models are being used in text generation and/or prediction tasks, question answering, and classification systems due to their ability to learn long-term dependencies. The present research integrates the LSTM network and dropout technique to generate a text from a corpus as input, a model is developed to find the best way to extract the words from the context. For training the model, the poem ""La Ciudad y los perros"" which is composed of 128,600 words is used as input data. The poem was divided into two data sets, 38.88% for training and the remaining 61.12% for testing the model. The proposed model was tested in two variants: word importance and context. The results were evaluated in terms of the semantic proximity of the generated text to the given context.

Recognizing Emotions in Video Using Multimodal DNN Feature Fusion

We present our system description of input-levelmultimodal fusion of audio, video, and text forrecognition of emotions and their intensities forthe 2018 First Grand Challenge on ComputationalModeling of Human Multimodal Language. Ourproposed approach is based on input-level featurefusion with sequence learning from BidirectionalLong-Short Term Memory (BLSTM) deep neuralnetworks (DNNs). We show that our fusion approach outperforms unimodal predictors. Our system performs 6-way simultaneous classificationand regression, allowing for overlapping emotionlabels in a video segment. This leads to an overall binary accuracy of 90%, overall 4-class accuracy of 89.2% and an overall mean-absolute-error(MAE) of 0.12. Our work shows that an early fusion technique can effectively predict the presenceof multi-label emotions as well as their coarse grained intensities. The presented multimodal approach creates a simple and robust baseline on thisnew Grand Challenge dataset. Furthermore, weprovide a detailed analysis of emotion intensitydistributions as output from our DNN, as well asa related discussion concerning the inherent difficulty of this task.<br/

A Combined CNN and LSTM Model for Arabic Sentiment Analysis

Deep neural networks have shown good data modelling capabilities when dealing with challenging and large datasets from a wide range of application areas. Convolutional Neural Networks (CNNs) offer advantages in selecting good features and Long Short-Term Memory (LSTM) networks have proven good abilities of learning sequential data. Both approaches have been reported to provide improved results in areas such image processing, voice recognition, language translation and other Natural Language Processing (NLP) tasks. Sentiment classification for short text messages from Twitter is a challenging task, and the complexity increases for Arabic language sentiment classification tasks because Arabic is a rich language in morphology. In addition, the availability of accurate pre-processing tools for Arabic is another current limitation, along with limited research available in this area. In this paper, we investigate the benefits of integrating CNNs and LSTMs and report obtained improved accuracy for Arabic sentiment analysis on different datasets. Additionally, we seek to consider the morphological diversity of particular Arabic words by using different sentiment classification levels.Comment: Authors accepted version of submission for CD-MAKE 201

Deep learning for multimedia processing-Predicting media interestingness

This thesis explores the application of a deep learning approach for the prediction of media interestingness. Two different models are investigated, one for the prediction of image and one for the prediction of video interestingness. For the prediction of image interestingness, the ResNet50 network is fine-tuned to obtain best results. First, some layers are added. Next, the model is trained and fine-tuned using data augmentation, dropout, class weights, and changing other hyper parameters. For the prediction of video interestingness, first, features are extracted with a 3D convolutional network. Next a LSTM network is trained and fine-tuned with the features. The final result is a binary label for each image/video: 1 for interesting, 0 for not interesting. Additionally, a confidence value is provided for each prediction. Finally, the Mean Average Precision (MAP) is employed as evaluation metric to estimate the quality of the final results.Esta tesis explora un enfoque con deep learning aplicado a la predicción del nivel de interés de imágenes y vídeos. Se investigan dos modelos, uno para predecir el nivel de interés de imágenes y otro para vídeos. Para la predicción del nivel de interés de imágenes, se adapta la red ResNet50 con el fin de obtener los mejores resultados. En primer lugar, se añaden capas. A continuación, se entrena y se adapta el modelo utilizando aumento de datos, dropout, ponderación de clases y cambiando otros hiperparámetros. Para la predicción del nivel de interés de vídeos, en primer lugar, se extraen características de los vídeos con una red convolucional 3D. A continuación se entrena y se adapta una red LSTM con estas características. El resultado final es una clasificación binaria para cada imagen/vídeo: 1 para "interesante", 0 para "no interesante". Además, se aporta un nivel de confianza en cada predicción. Finalmente, el promedio de la precisión media (MAP) se usa como métrica de evaluación para estimar la calidad de los resultados finales.Aquesta tèsi explora un enfocament amb deep learning aplicat a la predicció del nivell d'interès d'imatges i vídeos. S'investiguen dos models, un per a predir el nivell d'interès d'imatges i un altre per a vídeos. Per a la predicció del nivell d'interès d'imatges, s'adapta la xarxa ResNet50 amb la finalitat d'obtenir els millors resultats. En primer lloc, s'afegeixen capes. A continuació, s'entrena i s'adapta el model utilitzant augmentació de les dades, dropout, ponderació de classes i canviant hiperparàmetres. Per a la predicció del nivell d'interès de vídeos, en primer lloc, s'extreuen característiques dels videos amb una xarxa convolucional 3D. A continuació, s'entrena i s'adapta una xarxa LSTM amb aquestes característiques. El resultat final és una classificació binària de cada imatge/vídeo: 1 per a "interessant", 0 per a "no interessant". A més a més, s'aporta un nivell de confiança a cada predicció. Finalment, el promig de la precisió mitja (MAP) s'utilitza com a mètrica d'evaluació per a estimar la qualitat dels resultats finals

EXPLAINING CUSTOMER ACTIVATION WITH DEEP ATTENTION MODELS

Effectively informing consumers is a big challenge for financial service providers. Triggering involvement in the personal situation of the client is a result of sending relevant information at the right time. While general machine learning techniques are able to accurately predict the behavior of consumers, they tend to lack interpretability. This is a problem since interpretation aims at producing the information a communication department requires to be able to trigger involvement. In this paper we provide a solution for predicting and explaining customer activation as result of a series of events, by means of deep learning and attention models. The proposed solution is applied to data concerning the activity of pension fund participants and compared to standard machine learning techniques on both accuracy and interpretability. We conclude that the attention based model is as accurate as top tier machine learning algorithms in predicting customer activation, while being able to extract the key events in the communication with a single customer. This results in the ability to help understand the needs of customers on a personal level and to construct an individual marketing strategy for each customer

Quaternion-Based Graph Convolution Network for Recommendation

Graph Convolution Network (GCN) has been widely applied in recommender systems for its representation learning capability on user and item embeddings. However, GCN is vulnerable to noisy and incomplete graphs, which are common in real world, due to its recursive message propagation mechanism. In the literature, some work propose to remove the feature transformation during message propagation, but making it unable to effectively capture the graph structural features. Moreover, they model users and items in the Euclidean space, which has been demonstrated to have high distortion when modeling complex graphs, further degrading the capability to capture the graph structural features and leading to sub-optimal performance. To this end, in this paper, we propose a simple yet effective Quaternion-based Graph Convolution Network (QGCN) recommendation model. In the proposed model, we utilize the hyper-complex Quaternion space to learn user and item representations and feature transformation to improve both performance and robustness. Specifically, we first embed all users and items into the Quaternion space. Then, we introduce the quaternion embedding propagation layers with quaternion feature transformation to perform message propagation. Finally, we combine the embeddings generated at each layer with the mean pooling strategy to obtain the final embeddings for recommendation. Extensive experiments on three public benchmark datasets demonstrate that our proposed QGCN model outperforms baseline methods by a large margin.Comment: 13 pages, 7 figures, 6 tables. Submitted to ICDE 202

Recurrent neural networks and transformer-based models for multi-step prediction of agricultural commodity prices

Orientador: Prof. Dr. Ademir Aparecido Constantino.Coorientador: Prof. Dr. Rodrigo Clemente Thom de Souza.Dissertação (mestrado em Engenharia de Produção) - Universidade Estadual de Maringá, 2023This dissertation explores the use of Recurrent Neural Networks (RNNs) and Transformerbased models for multi-step prediction of agricultural commodity prices. The study begins with a systematic literature review, providing a comprehensive understanding of the field and identifying gaps in research. Two experimental papers are then presented, focusing on the application of RNNs, specifically Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term Memory Networks (BiLSTM) models, as well as a Transformer-based model for predicting agricultural commodity prices. The commodities of particular interest are soybeans, corn, cattle, and sugar, especially soybeans, the target of the analysis. The study investigates the performance of the Transformer model in forecasting agricultural commodity prices, comparing it with RNNs, both in short and long-term forecasting horizons. The findings highlight the importance of hyperparameter optimization, unbiased model evaluation, and the selection of a suitable forecast horizon that minimizes errors while providing useful information to farmers. Overall, this research contributes to advancing knowledge in the field of RNN models for predicting agricultural commodity prices. It underscores the significance of optimization techniques and the versatility of Transformer models across different domains.Essa dissertação explora o uso de Redes Neurais Recorrentes (RNRs) e modelos baseados em Transformers para a previsão de múltiplos passos dos preços de commodities agrícolas. O estudo começa com uma revisão sistemática da literatura, fornecendo uma compreensão abrangente do campo e identificando lacunas na pesquisa. Dois artigos experimentais são então apresentados, focando na aplicação de RNRs, especificamente os modelos Long Short-Term Memory (LSTM) e Bidirectional Long Short-Term Memory Networks (BiLSTM), assim como um modelo baseado em Transformer para prever preços de commodities agrícolas. As commodities de interesse particular são soja, milho, gado e açúcar, especialmente a soja, o alvo da análise. O estudo investiga a eficácia do modelo Transformer na previsão de preços de commodities agrícolas, comparando-o com as RNRs, tanto em horizontes de previsão de curto prazo quanto de longo prazo. Os resultados destacam a importância da otimização de hiperparâmetros, avaliação imparcial do modelo e seleção de um horizonte de previsão adequado que minimize erros ao mesmo tempo em que fornece informações úteis aos agricultores. No geral, essa pesquisa contribui para avançar o conhecimento no campo dos modelos de RNR para prever preços de commodities agrícolas. Ela destaca a importância das técnicas de otimização e a versatilidade dos modelos baseados em Transformers em diferentes domínios.96 f. : il. color., tabs. figs

Traffic flow forecasting with deep learning

In recent years there has been a vast increase in available data with the ad- vancement of smart cities. In the domain of Intelligent Transportation Systems (ITS) this modernisation can positively effect transportation networks, thus cut- ting down travel time, increase efficacy, and reduce environmental impact from vehicles. Norwegian Public Roads Administration (NPRA) is currently deploying a new vehicle detector system named Datainn on all public roads in Norway. Datainn sends metadata on all detected vehicles in real time. This includes information about speed, gap between vehicles, weight, and classification of vehicle type. Many machine learning approaches has been researched in literature on how to forecast traffic flow information. One such approach is that of using Artificial Neural Networks (ANNs). In this research ANN based methods have been explored. This was done by first performing a state-of-the-art Structured Literature Review (SLR) on ANN methods in literature. From the review, Stacked Sparse Autoencoder (SSAE) model was compared with recent advances of Long Short-Term Memory (LSTM) and Deep Neural Network (DNN) on four different prediction horizons. The data foundation was the new Datainn system using traffic data from a highway around Norway s capitol, Oslo. Further, the model performance was assessed with extended feature vectors including more metadata from Datainn. The results found that the LSTM model always outperformed DNN and SSAE, although in general the performance characteristics was somewhat similar. Ex- tending the feature vector with more variables had a negative effect on DNN, while resulting in better performance for Recurrent Neural Network (RNN) on long-term (60 minute) forecasting horizons. For SSAE it had a slight positive effect, but not enough get better results than RNN or DNN