S-TREE: Self-Organizing Trees for Data Clustering and Online Vector Quantization

This paper introduces S-TREE (Self-Organizing Tree), a family of models that use unsupervised learning to construct hierarchical representations of data and online tree-structured vector quantizers. The S-TREE1 model, which features a new tree-building algorithm, can be implemented with various cost functions. An alternative implementation, S-TREE2, which uses a new double-path search procedure, is also developed. S-TREE2 implements an online procedure that approximates an optimal (unstructured) clustering solution while imposing a tree-structure constraint. The performance of the S-TREE algorithms is illustrated with data clustering and vector quantization examples, including a Gauss-Markov source benchmark and an image compression application. S-TREE performance on these tasks is compared with the standard tree-structured vector quantizer (TSVQ) and the generalized Lloyd algorithm (GLA). The image reconstruction quality with S-TREE2 approaches that of GLA while taking less than 10% of computer time. S-TREE1 and S-TREE2 also compare favorably with the standard TSVQ in both the time needed to create the codebook and the quality of image reconstruction.Office of Naval Research (N00014-95-10409, N00014-95-0G57

Sparse Probabilistic Circuits via Pruning and Growing

Probabilistic circuits (PCs) are a tractable representation of probability distributions allowing for exact and efficient computation of likelihoods and marginals. There has been significant recent progress on improving the scale and expressiveness of PCs. However, PC training performance plateaus as model size increases. We discover that most capacity in existing large PC structures is wasted: fully-connected parameter layers are only sparsely used. We propose two operations: pruning and growing, that exploit the sparsity of PC structures. Specifically, the pruning operation removes unimportant sub-networks of the PC for model compression and comes with theoretical guarantees. The growing operation increases model capacity by increasing the size of the latent space. By alternatingly applying pruning and growing, we increase the capacity that is meaningfully used, allowing us to significantly scale up PC learning. Empirically, our learner achieves state-of-the-art likelihoods on MNIST-family image datasets and on Penn Tree Bank language data compared to other PC learners and less tractable deep generative models such as flow-based models and variational autoencoders (VAEs).Comment: 36th Conference on Neural Information Processing Systems (NeurIPS 2022

Character-Level Incremental Speech Recognition with Recurrent Neural Networks

In real-time speech recognition applications, the latency is an important issue. We have developed a character-level incremental speech recognition (ISR) system that responds quickly even during the speech, where the hypotheses are gradually improved while the speaking proceeds. The algorithm employs a speech-to-character unidirectional recurrent neural network (RNN), which is end-to-end trained with connectionist temporal classification (CTC), and an RNN-based character-level language model (LM). The output values of the CTC-trained RNN are character-level probabilities, which are processed by beam search decoding. The RNN LM augments the decoding by providing long-term dependency information. We propose tree-based online beam search with additional depth-pruning, which enables the system to process infinitely long input speech with low latency. This system not only responds quickly on speech but also can dictate out-of-vocabulary (OOV) words according to pronunciation. The proposed model achieves the word error rate (WER) of 8.90% on the Wall Street Journal (WSJ) Nov'92 20K evaluation set when trained on the WSJ SI-284 training set.Comment: To appear in ICASSP 201

Tree pruning/inspection robot climbing mechanism design, kinematics study and intelligent control : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Mechatronics at Massey University, Manawatu Campus, New Zealand

Forestry plays an important role in New Zealand’s economy as its third largest export earner. To achieve New Zealand Wood Council’s export target of $12 billion by 2022 in forest and improve the current situation that is the reduction of wood harvesting area, the unit value and volume of lumber must be increased. Pruning is essential and critical for obtaining high-quality timber during plantation growing. Powerful tools and robotic systems have great potential for sustainable forest management. Up to now, only a few tree-pruning robotic systems are available on the market. Unlike normal robotic manipulators or mobile robots, tree pruning robot has its unique requirements and features. The challenges include climbing pattern control, anti-free falling, and jamming on the tree trunk etc. Through the research on the available pole and tree climbing robots, this thesis presents a novel mechanism of tree climbing robotic system that could serve as a climbing platform for applications in the forest industry like tree pruning, inspection etc. that requires the installation of powerful or heavy tools. The unique features of this robotic system include the passive and active anti-falling mechanisms that prevent the robot falling to the ground under either static or dynamic situations, the capability to vertically or spirally climb up a tree trunk and the flexibility to suit different sizes of tree trunk. Furthermore, for the convenience of tree pruning and the fulfilment of robot anti-jamming feature, the robot platform while the robot climbs up should move up without tilting. An intelligent platform balance control system with real-time sensing integration was developed to overcome the climbing tilting problem. The thesis also presents the detail kinematic and dynamic study, simulation, testing and analysis. A physical testing model of this proposed robotic system was built and tested on a cylindrical rod. The mass of the prototype model is 6.8 Kg and can take 2.1 Kg load moving at the speed of 42 mm/s. The trunk diameter that the robot can climb up ranges from 120 to 160 mm. The experiment results have good matches with the simulations and analysis. This research established a basis for developing wheel-driven tree or pole climbing robots. The design and simulation method, robotic leg mechanism and the control methodologies could be easily applied for other wheeled tree/pole climbing robots. This research has produced 6 publications, two ASME journal papers and 4 IEEE international conference papers that are available on IEEE Xplore. The published content ranges from robotic mechanism design, signal processing, platform balance control, and robot climbing behavior optimization. This research also brought interesting topics for further research such as the integration with artificial intelligent module and mobile robot for remote tree/forest inspection after pruning or for pest control

Solving Multiclass Learning Problems via Error-Correcting Output Codes

Multiclass learning problems involve finding a definition for an unknown function f(x) whose range is a discrete set containing k &gt 2 values (i.e., k ``classes''). The definition is acquired by studying collections of training examples of the form [x_i, f (x_i)]. Existing approaches to multiclass learning problems include direct application of multiclass algorithms such as the decision-tree algorithms C4.5 and CART, application of binary concept learning algorithms to learn individual binary functions for each of the k classes, and application of binary concept learning algorithms with distributed output representations. This paper compares these three approaches to a new technique in which error-correcting codes are employed as a distributed output representation. We show that these output representations improve the generalization performance of both C4.5 and backpropagation on a wide range of multiclass learning tasks. We also demonstrate that this approach is robust with respect to changes in the size of the training sample, the assignment of distributed representations to particular classes, and the application of overfitting avoidance techniques such as decision-tree pruning. Finally, we show that---like the other methods---the error-correcting code technique can provide reliable class probability estimates. Taken together, these results demonstrate that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems.Comment: See http://www.jair.org/ for any accompanying file