Learning neural trans-dimensional random field language models with noise-contrastive estimation

Trans-dimensional random field language models (TRF LMs) where sentences are modeled as a collection of random fields, have shown close performance with LSTM LMs in speech recognition and are computationally more efficient in inference. However, the training efficiency of neural TRF LMs is not satisfactory, which limits the scalability of TRF LMs on large training corpus. In this paper, several techniques on both model formulation and parameter estimation are proposed to improve the training efficiency and the performance of neural TRF LMs. First, TRFs are reformulated in the form of exponential tilting of a reference distribution. Second, noise-contrastive estimation (NCE) is introduced to jointly estimate the model parameters and normalization constants. Third, we extend the neural TRF LMs by marrying the deep convolutional neural network (CNN) and the bidirectional LSTM into the potential function to extract the deep hierarchical features and bidirectionally sequential features. Utilizing all the above techniques enables the successful and efficient training of neural TRF LMs on a 40x larger training set with only 1/3 training time and further reduces the WER with relative reduction of 4.7% on top of a strong LSTM LM baseline.Comment: 5 pages and 2 figure

LEARNING MEDIA DEVELOPMENT TEST OF TRANS-DIMENSIONAL PICTURE OBJECT (TRANS-DIMENSIONAL OBJEK GAMBAR/TDOG) IN THE OF FIRST BUILDING CONSTRUCTION AND DRAWING COURSE (KONSTRUKSI BANGUNAN/KBM I)

The competency of KBM I course contained a concept of complicated building structures, especially for students whose background of Senior High School. The complexity of building structures caused some difficulties in understanding and drawing skills results unachieved basic competence of KBM I course. This research aims at making self-learning media by Trans Dimensi Obyek Gambar (TDOG) method based on the students’ necessities in order to be easier in understanding relationship between concept and reality, and the real object and the picture on the paper. The procedures of this research used Research and Development (R&D) procedures. The plan of research implementation was divided into two stages, those were: 1) product development, and 2) product’s effectiveness testing. In detail, product development was divided into some stages, such as: concept making, designing, materials collecting, combining, try out implementation, and distribution. The method used to analyze data was quantitative descriptive analysis technique expressed in score distribution and assessment scale category that was determined. Based on the test results, it can be concluded that: 1) media development model based on students’ necessities was developed into some steps, those were: doing preliminary research, designing software, collecting the materials, developing initial product, validating initial product, analyzing, revising initial product, conducting field try out, analyzing field try out, doing the second revision, and final product; 2) the average score of field try out results on the first stage was 3.82, on the second try out, it improved with the average score of 4.00, and on the third try out, it also improved with the average score of 4.24 of 5. It was categorized as “Very Good”. From the result of the third try out, this media was considered as appropriate to use as learning media by the students. Key Words: learning media, TDO

Bayesian orthogonal component analysis for sparse representation

This paper addresses the problem of identifying a lower dimensional space where observed data can be sparsely represented. This under-complete dictionary learning task can be formulated as a blind separation problem of sparse sources linearly mixed with an unknown orthogonal mixing matrix. This issue is formulated in a Bayesian framework. First, the unknown sparse sources are modeled as Bernoulli-Gaussian processes. To promote sparsity, a weighted mixture of an atom at zero and a Gaussian distribution is proposed as prior distribution for the unobserved sources. A non-informative prior distribution defined on an appropriate Stiefel manifold is elected for the mixing matrix. The Bayesian inference on the unknown parameters is conducted using a Markov chain Monte Carlo (MCMC) method. A partially collapsed Gibbs sampler is designed to generate samples asymptotically distributed according to the joint posterior distribution of the unknown model parameters and hyperparameters. These samples are then used to approximate the joint maximum a posteriori estimator of the sources and mixing matrix. Simulations conducted on synthetic data are reported to illustrate the performance of the method for recovering sparse representations. An application to sparse coding on under-complete dictionary is finally investigated.Comment: Revised version. Accepted to IEEE Trans. Signal Processin