Character-level Recurrent Neural Networks in Practice: Comparing Training and Sampling Schemes

Recurrent neural networks are nowadays successfully used in an abundance of applications, going from text, speech and image processing to recommender systems. Backpropagation through time is the algorithm that is commonly used to train these networks on specific tasks. Many deep learning frameworks have their own implementation of training and sampling procedures for recurrent neural networks, while there are in fact multiple other possibilities to choose from and other parameters to tune. In existing literature this is very often overlooked or ignored. In this paper we therefore give an overview of possible training and sampling schemes for character-level recurrent neural networks to solve the task of predicting the next token in a given sequence. We test these different schemes on a variety of datasets, neural network architectures and parameter settings, and formulate a number of take-home recommendations. The choice of training and sampling scheme turns out to be subject to a number of trade-offs, such as training stability, sampling time, model performance and implementation effort, but is largely independent of the data. Perhaps the most surprising result is that transferring hidden states for correctly initializing the model on subsequences often leads to unstable training behavior depending on the dataset.Comment: 23 pages, 11 figures, 4 table

PENINGKATAN FEATURE SELECTION DENGAN WINDOWED MOMENTUM UNTUK PREDIKSI KANKER PAYUDARA

Kanker payudara meningkat di setiap negara di dunia , terutama di negara-negara berkembang seperti Indonesia . Neural Network mampu memecahkan masalah dengan akurasi data dan tidak linear . Neural Network optimasi diuji minggu untuk menghasilkan yang terbaik nilai akurasi , menerapkan jaringan saraf dengan metode seleksi fitur seperti Wrapper dengan Penghapusan Mundur untuk meningkatkan akurasi yang dihasilkan oleh Neural Network. Percobaan yang dilakukan untuk mendapatkan arsitektur yang optimal dan meningkatkan nilai akurasi . Hasil dari penelitian ini adalah matriks kebingungan untuk membuktikan keakuratan Neural Network sebelum dioptimalkan oleh Backward Elimination adalah 96,42 % dan 96,71 % setelah menjadi dioptimalkan. Hal ini membuktikan estimasi uji seleksi fitur menggunakan metode berbasis jaringan saraf Backward Elimination lebih akurat dibandingkan dengan metode jaringan saraf tiruan. Windowed momentum  dapat meningkatkan waktu pengklasifikasian feature selection sehingga didapat momentum yang lebih maksimal

A comparative review of dynamic neural networks and hidden Markov model methods for mobile on-device speech recognition

The adoption of high-accuracy speech recognition algorithms without an effective evaluation of their impact on the target computational resource is impractical for mobile and embedded systems. In this paper, techniques are adopted to minimise the required computational resource for an effective mobile-based speech recognition system. A Dynamic Multi-Layer Perceptron speech recognition technique, capable of running in real time on a state-of-the-art mobile device, has been introduced. Even though a conventional hidden Markov model when applied to the same dataset slightly outperformed our approach, its processing time is much higher. The Dynamic Multi-layer Perceptron presented here has an accuracy level of 96.94% and runs significantly faster than similar techniques

MUST-CNN: A Multilayer Shift-and-Stitch Deep Convolutional Architecture for Sequence-based Protein Structure Prediction

Predicting protein properties such as solvent accessibility and secondary structure from its primary amino acid sequence is an important task in bioinformatics. Recently, a few deep learning models have surpassed the traditional window based multilayer perceptron. Taking inspiration from the image classification domain we propose a deep convolutional neural network architecture, MUST-CNN, to predict protein properties. This architecture uses a novel multilayer shift-and-stitch (MUST) technique to generate fully dense per-position predictions on protein sequences. Our model is significantly simpler than the state-of-the-art, yet achieves better results. By combining MUST and the efficient convolution operation, we can consider far more parameters while retaining very fast prediction speeds. We beat the state-of-the-art performance on two large protein property prediction datasets.Comment: 8 pages ; 3 figures ; deep learning based sequence-sequence prediction. in AAAI 201

Learning to detect dysarthria from raw speech

Speech classifiers of paralinguistic traits traditionally learn from diverse hand-crafted low-level features, by selecting the relevant information for the task at hand. We explore an alternative to this selection, by learning jointly the classifier, and the feature extraction. Recent work on speech recognition has shown improved performance over speech features by learning from the waveform. We extend this approach to paralinguistic classification and propose a neural network that can learn a filterbank, a normalization factor and a compression power from the raw speech, jointly with the rest of the architecture. We apply this model to dysarthria detection from sentence-level audio recordings. Starting from a strong attention-based baseline on which mel-filterbanks outperform standard low-level descriptors, we show that learning the filters or the normalization and compression improves over fixed features by 10% absolute accuracy. We also observe a gain over OpenSmile features by learning jointly the feature extraction, the normalization, and the compression factor with the architecture. This constitutes a first attempt at learning jointly all these operations from raw audio for a speech classification task.Comment: 5 pages, 3 figures, submitted to ICASS

Speaker Identification Using a Combination of Different Parameters as Feature Inputs to an Artificial Neural Network Classifier

This paper presents a technique using artificial neural networks (ANNs) for speaker identification that results in a better success rate compared to other techniques. The technique used in this paper uses both power spectral densities (PSDs) and linear prediction coefficients (LPCs) as feature inputs to a self organizing feature map to achieve a better identification performance. Results for speaker identification with different methods are presented and compared