Neural Computing in Quaternion Algebra

兵庫県立大学201

Selected aspects of complex, hypercomplex and fuzzy neural networks

This short report reviews the current state of the research and methodology on theoretical and practical aspects of Artificial Neural Networks (ANN). It was prepared to gather state-of-the-art knowledge needed to construct complex, hypercomplex and fuzzy neural networks. The report reflects the individual interests of the authors and, by now means, cannot be treated as a comprehensive review of the ANN discipline. Considering the fast development of this field, it is currently impossible to do a detailed review of a considerable number of pages. The report is an outcome of the Project 'The Strategic Research Partnership for the mathematical aspects of complex, hypercomplex and fuzzy neural networks' meeting at the University of Warmia and Mazury in Olsztyn, Poland, organized in September 2022.Comment: 46 page

Deep Quaternion Networks

The field of deep learning has seen significant advancement in recent years. However, much of the existing work has been focused on real-valued numbers. Recent work has shown that a deep learning system using the complex numbers can be deeper for a fixed parameter budget compared to its real-valued counterpart. In this work, we explore the benefits of generalizing one step further into the hyper-complex numbers, quaternions specifically, and provide the architecture components needed to build deep quaternion networks. We develop the theoretical basis by reviewing quaternion convolutions, developing a novel quaternion weight initialization scheme, and developing novel algorithms for quaternion batch-normalization. These pieces are tested in a classification model by end-to-end training on the CIFAR-10 and CIFAR-100 data sets and a segmentation model by end-to-end training on the KITTI Road Segmentation data set. These quaternion networks show improved convergence compared to real-valued and complex-valued networks, especially on the segmentation task, while having fewer parametersComment: IJCNN 2018, 8 pages, 1 figur

Meta-Heuristic Optimization Methods for Quaternion-Valued Neural Networks

In recent years, real-valued neural networks have demonstrated promising, and often striking, results across a broad range of domains. This has driven a surge of applications utilizing high-dimensional datasets. While many techniques exist to alleviate issues of high-dimensionality, they all induce a cost in terms of network size or computational runtime. This work examines the use of quaternions, a form of hypercomplex numbers, in neural networks. The constructed networks demonstrate the ability of quaternions to encode high-dimensional data in an efficient neural network structure, showing that hypercomplex neural networks reduce the number of total trainable parameters compared to their real-valued equivalents. Finally, this work introduces a novel training algorithm using a meta-heuristic approach that bypasses the need for analytic quaternion loss or activation functions. This algorithm allows for a broader range of activation functions over current quaternion networks and presents a proof-of-concept for future work

Demand Forecasting for Food Production Using Machine Learning Algorithms: A Case Study of University Refectory

Accurate food demand forecasting is one of the critical aspects of successfully managing restaurants, cafeterias, canteens, and refectories. This paper aims to develop demand forecasting models for a university refectory. Our study focused on the development of Machine Learning-based forecasting models which take into account the calendar effect and meal ingredients to predict the heavy demand for food within a limited timeframe (e.g., lunch) and without pre-booking. We have developed eighteen prediction models gathered under five main techniques. Three Artificial Neural Network models (i.e., Feed Forward, Function Fitting, and Cascade Forward), four Gauss Process Regression models (i.e., Rational Quadratic, Squared Exponential, Matern 5/2, and Exponential), six Support Vector Regression models (i.e., Linear, Quadratic, Cubic, Fine Gaussian, Medium Gaussian, and Coarse Gaussian), three Regression Tree models (i.e., Fine, Medium, and Coarse), two Ensemble Decision Tree (EDT) models (i.e., Boosted and Bagged) and one Linear Regression model were applied. When evaluated in terms of method diversity, prediction performance, and application area, to the best of our knowledge, this study offers a different contribution from previous studies. The EDT Boosted model obtained the best prediction performance (i.e., Mean Squared Error = 0,51, Mean Absolute Erro = 0,50, and R = 0,96)

Real to H-space Encoder for Speech Recognition

International audienceDeep neural networks (DNNs) and more precisely recurrent neural networks (RNNs) are at the core of modern automatic speech recognition systems, due to their efficiency to process input sequences. Recently, it has been shown that different input representations, based on multidimensional algebras, such as complex and quaternion numbers, are able to bring to neural networks a more natural, compressive and powerful representation of the input signal by outperforming common real-valued NNs. Indeed, quaternion-valued neural networks (QNNs) better learn both internal dependencies, such as the relation between the Mel-filter-bank value of a specific time frame and its time derivatives, and global dependencies, describing the relations that exist between time frames. Nonetheless, QNNs are limited to quaternion-valued input signals, and it is difficult to benefit from this powerful representation with real-valued input data. This paper proposes to tackle this weakness by introducing a real-to-quaternion encoder that allows QNNs to process any one dimensional input features, such as traditional Mel-filter-banks for automatic speech recognition

Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition

Recently, the connectionist temporal classification (CTC) model coupled with recurrent (RNN) or convolutional neural networks (CNN), made it easier to train speech recognition systems in an end-to-end fashion. However in real-valued models, time frame components such as mel-filter-bank energies and the cepstral coefficients obtained from them, together with their first and second order derivatives, are processed as individual elements, while a natural alternative is to process such components as composed entities. We propose to group such elements in the form of quaternions and to process these quaternions using the established quaternion algebra. Quaternion numbers and quaternion neural networks have shown their efficiency to process multidimensional inputs as entities, to encode internal dependencies, and to solve many tasks with less learning parameters than real-valued models. This paper proposes to integrate multiple feature views in quaternion-valued convolutional neural network (QCNN), to be used for sequence-to-sequence mapping with the CTC model. Promising results are reported using simple QCNNs in phoneme recognition experiments with the TIMIT corpus. More precisely, QCNNs obtain a lower phoneme error rate (PER) with less learning parameters than a competing model based on real-valued CNNs.Comment: Accepted at INTERSPEECH 201

Deep quaternion neural networks for spoken language understanding

International audienceThe availability of open-source software is playing a remarkable role in the popularization of speech recognition and deep learning. Kaldi, for instance, is nowadays an established framework used to develop state-of-the-art speech recognizers. PyTorch is used to build neural networks with the Python language and has recently spawn tremendous interest within the machine learning community thanks to its simplicity and flexibility. The PyTorch-Kaldi project aims to bridge the gap between these popular toolkits, trying to inherit the efficiency of Kaldi and the flexibility of PyTorch. PyTorch-Kaldi is not only a simple interface between these software, but it embeds several useful features for developing modern speech recognizers. For instance, the code is specifically designed to naturally plug-in user-defined acoustic models. As an alternative, users can exploit several pre-implemented neural networks that can be customized using intuitive configuration files. PyTorch-Kaldi supports multiple feature and label streams as well as combinations of neural networks, enabling the use of complex neural architectures. The toolkit is publicly-released along with a rich documentation and is designed to properly work locally or on HPC clusters. Experiments, that are conducted on several datasets and tasks, show that PyTorch-Kaldi can effectively be used to develop modern state-of-the-art speech recognizers

Single Exponential Smoothing-Multilayer Perceptron Untuk Peramalan Pengunjung Unik Jurnal Elektronik

Jumlah kunjungan rerata pengunjung unik per hari pada jurnal elektronik menunjukkan bahwa hasil terbitan karya ilmiah website tersebut menarik. Sehingga jumlah pengunjung unik dijadikan indikator penting dalam mengukur keberhasilan sebuah jurnal elektronik untuk memenuhi perluasan, penyebaran dan percepatan sistem akreditasi jurnal. Pengunjung Unik merupakan jumlah pengunjung per Internet Address (IP) yang mengakses sebuah jurnal elektronik dalam kurun waktu tertentu. Terdapat beberapa metode yang biasa digunakan untuk peramalan, diantaranya adalah Multilayer Perceptron (MLP). Kualitas data berpengaruh besar dalam membangun model MLP yang baik, karena sukses tidaknya permodelan pada MLP sangat dipengaruhi oleh data input. Salah satu cara untuk meningkatkan kualitas data adalah dengan melakukan smoothing pada data tersebut. Pada penelitian ini digunkan metode peramalan Multilayer Perceptron berdasarkan penelitian sebelumnya dengan kombinasi data training dan testing 80%-20% dengan asitektur 2-1-1 dan learning rate 0,4. Selanjutnya untuk meningkatkan kualitas data dilakukan smoothing dengan menerapkan metode Single Exponential Smoothing. Dari penelitian yang dilakukan diperoleh hasil terbaik menggunakan alpha 0.9 dengan hasil akurasi MSE 94.02% dan RMSE 75.54% dengan lama waktu eksekusi 580,27 detik. The number of visits by the average unique visitor per day on electronic journals shows that the published scientific papers on the website are interesting. So that the number of unique visitors is used as an important indicator in measuring the success of an electronic journal to meet the expansion, dissemination and acceleration of the journal accreditation system. Unique Visitors is the number of visitors per Internet Address (IP) who access an electronic journal within a certain period of time. There are several methods commonly used for forecasting, including the Multilayer Perceptron (MLP). Data quality has a big influence in building a good MLP model, because the success or failure of modeling in MLP is greatly influenced by the input data. One way to improve data quality is by smoothing the data. In this study, the Multilayer Perceptron forecasting method was used based on previous research with a combination of training data and testing 80% -20% with a 2-1-1 architecture and a learning rate of 0.4. Furthermore, to improve data quality, smoothing is done by applying the Single Exponential Smoothing method. From the research conducted, the best results were obtained using alpha 0.9 with MSE accuracy of 94.02% and RMSE 75.54% with a long execution time of 580.27 seconds