Exploiting Compositionality to Explore a Large Space of Model Structures

The recent proliferation of richly structured probabilistic models raises the question of how to automatically determine an appropriate model for a dataset. We investigate this question for a space of matrix decomposition models which can express a variety of widely used models from unsupervised learning. To enable model selection, we organize these models into a context-free grammar which generates a wide variety of structures through the compositional application of a few simple rules. We use our grammar to generically and efficiently infer latent components and estimate predictive likelihood for nearly 2500 structures using a small toolbox of reusable algorithms. Using a greedy search over our grammar, we automatically choose the decomposition structure from raw data by evaluating only a small fraction of all models. The proposed method typically finds the correct structure for synthetic data and backs off gracefully to simpler models under heavy noise. It learns sensible structures for datasets as diverse as image patches, motion capture, 20 Questions, and U.S. Senate votes, all using exactly the same code.United States. Army Research Office (ARO grant W911NF-08-1-0242)American Society for Engineering Education. National Defense Science and Engineering Graduate Fellowshi

Cosine transform priors for enhanced decoding of compressed images

Abstract. Image compression methods such as JPEG use quantisation of discrete cosine transform (DCT) coefficients of image blocks to produce lossy compression. During decoding, an inverse DCT of the quantised values is used to obtain the lossy image. These methods suffer from blocky effects from the region boundaries, and can produce poor representations of regions containing sharp edges. Such problems can be obvious artefacts in compressed images but also cause significant problems for many super-resolution algorithms. Prior information about the DCT coefficients of an image and the continuity between image blocks can be used to improve the decoding using the same compressed image information. This paper analyses empirical priors for DCT coefficients, and shows how they can be combined with block edge contiguity information to produce decoding methods which reduce the blockiness of images. We show that the use of DCT priors is generic can be useful in many other circumstances.

Training Neural Networks for Financial Forecasting: Backpropagation vs Particle Swarm Optimization

Neural networks (NN) architectures can be effectively used to classify, forecast and recognize quantity of interest in, e.g., computer vision, machine translation, finance, etc. Concerning the financial framework, fore- casting procedures are often used as a part of the decision making process in both trading and portfolio strategy optimization. Unfortunately training a NN is in general a challenging task mainly because of the high number of parameters involved. In particular, a typical NN is based on a large number of layers, each of which may be composed by several neurons , moreover, for every component, normalization as well as training algorithms, have to be performed. One of the most popular method to overcome such difficulties is represented by the so called back propagation algorithm . Other possibilities are represented by genetic algorithms , and, in this family, the swarm particle optimization method seems to be rather promising. In this paper we want to compare canonical back- propagation and the swarm particle optimization algorithm in minimizing the error on surface created by financial time series, particularly concerning the task of forecast up/down movements for the assets we are interested in

Implicit surfaces for interactive animated characters

Thesis (S.M.)--Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1999.Includes bibliographical references (leaves 64-68).Implicit surface modeling in computer graphics is a powerful technique for representing smooth and organic shapes. Skeletal elements of an implicit surface blend to create a smooth, seamless skin which exhibits desired properties for animation such as squash and stretch. Because of their high computational cost to render, implicit surfaces have not been used extensively in the real-time graphics domain. This thesis discusses the problems and some solutions in the application of implicit surfaces to the domain of interactive character animation. A design process for an implicit surface-based character is proposed, from the modeling and texturing stages to animation and rendering.by Kenneth Bradley Russell.S.M

ForeNet: fourier recurrent neural networks for time series prediction.

Ying-Qian Zhang.Thesis (M.Phil.)--Chinese University of Hong Kong, 2001.Includes bibliographical references (leaves 115-124).Abstracts in English and Chinese.Abstract --- p.iAcknowledgement --- p.iiiChapter 1 --- Introduction --- p.1Chapter 1.1 --- Background --- p.1Chapter 1.2 --- Objective --- p.2Chapter 1.3 --- Contributions --- p.3Chapter 1.4 --- Thesis Overview --- p.4Chapter 2 --- Literature Review --- p.6Chapter 2.1 --- Takens' Theorem --- p.6Chapter 2.2 --- Linear Models for Prediction --- p.7Chapter 2.2.1 --- Autoregressive Model --- p.7Chapter 2.2.2 --- Moving Average Model --- p.8Chapter 2.2.3 --- Autoregressive-moving Average Model --- p.9Chapter 2.2.4 --- Fitting a Linear Model to a Given Time Series --- p.9Chapter 2.2.5 --- State-space Reconstruction --- p.10Chapter 2.3 --- Neural Network Models for Time Series Processing --- p.11Chapter 2.3.1 --- Feed-forward Neural Networks --- p.11Chapter 2.3.2 --- Recurrent Neural Networks --- p.14Chapter 2.3.3 --- Training Algorithms for Recurrent Networks --- p.18Chapter 2.4 --- Combining Neural Networks and other approximation techniques --- p.22Chapter 3 --- ForeNet: Model and Representation --- p.24Chapter 3.1 --- Fourier Recursive Prediction Equation --- p.24Chapter 3.1.1 --- Fourier Analysis of Time Series --- p.25Chapter 3.1.2 --- Recursive Form --- p.25Chapter 3.2 --- Fourier Recurrent Neural Network Model (ForeNet) --- p.27Chapter 3.2.1 --- Neural Networks Representation --- p.28Chapter 3.2.2 --- Architecture of ForeNet --- p.29Chapter 4 --- ForeNet: Implementation --- p.32Chapter 4.1 --- Improvement on ForeNet --- p.33Chapter 4.1.1 --- Number of Hidden Neurons --- p.33Chapter 4.1.2 --- Real-valued Outputs --- p.34Chapter 4.2 --- Parameters Initialization --- p.37Chapter 4.3 --- Application of ForeNet: the Process of Time Series Prediction --- p.38Chapter 4.4 --- Some Implications --- p.39Chapter 5 --- ForeNet: Initialization --- p.40Chapter 5.1 --- Unfolded Form of ForeNet --- p.40Chapter 5.2 --- Coefficients Analysis --- p.43Chapter 5.2.1 --- "Analysis of the Coefficients Set, vn " --- p.43Chapter 5.2.2 --- "Analysis of the Coefficients Set, μn(d) " --- p.44Chapter 5.3 --- Experiments of ForeNet Initialization --- p.47Chapter 5.3.1 --- Objective and Experiment Setting --- p.47Chapter 5.3.2 --- Prediction of Sunspot Series --- p.49Chapter 5.3.3 --- Prediction of Mackey-Glass Series --- p.53Chapter 5.3.4 --- Prediction of Laser Data --- p.56Chapter 5.3.5 --- Three More Series --- p.59Chapter 5.4 --- Some Implications on the Proposed Initialization Method --- p.63Chapter 6 --- ForeNet: Learning Algorithms --- p.67Chapter 6.1 --- Complex Real Time Recurrent Learning (CRTRL) --- p.68Chapter 6.2 --- Batch-mode Learning --- p.70Chapter 6.3 --- Time Complexity --- p.71Chapter 6.4 --- Property Analysis and Experimental Results --- p.72Chapter 6.4.1 --- Efficient initialization:compared with random initialization --- p.74Chapter 6.4.2 --- Complex-valued network:compared with real-valued net- work --- p.78Chapter 6.4.3 --- Simple architecture:compared with ring-structure RNN . --- p.79Chapter 6.4.4 --- Linear model: compared with nonlinear ForeNet --- p.80Chapter 6.4.5 --- Small number of hidden units --- p.88Chapter 6.5 --- Comparison with Some Other Models --- p.89Chapter 6.5.1 --- Comparison with AR model --- p.91Chapter 6.5.2 --- Comparison with TDNN Networks and FIR Networks . --- p.93Chapter 6.5.3 --- Comparison to a few more results --- p.94Chapter 6.6 --- Summarization --- p.95Chapter 7 --- Learning and Prediction: On-Line Training --- p.98Chapter 7.1 --- On-Line Learning Algorithm --- p.98Chapter 7.1.1 --- Advantages and Disadvantages --- p.98Chapter 7.1.2 --- Training Process --- p.99Chapter 7.2 --- Experiments --- p.101Chapter 7.3 --- Predicting Stock Time Series --- p.105Chapter 8 --- Discussions and Conclusions --- p.109Chapter 8.1 --- Limitations of ForeNet --- p.109Chapter 8.2 --- Advantages of ForeNet --- p.111Chapter 8.3 --- Future Works --- p.112Bibliography --- p.11

A Neural Network Linking Process

A novel method of integrating multiple neural networks into one large network via a process referred to as a neural network linking process is proposed. Neural networks are commonly trained to solve a specific problem for an encapsulated problem domain. A single network can undertake simple classification or generalisation problems. Dividing them into sub-problems, which in turn are solved by a sub-network, can disentangle more complicated classification or generalisation problems. A controller generally combines sub-network results. A controller can be, for instance, a gating network, voting system or a mathematical combiner. In each case, every sub-network is used as a separate unit and is not interconnected to any other sub-network. However, with the linking process a novel method for linking trained sub-networks into one large network by maintaining the knowledge of each individual sub-network is introduced. Furthermore, the linked network will utilize a stimulus process in order to distinguish the type of sub-problem to be solved, by largely retaining the accuracy of the sub-network, as well as being one step closer to the biological reality