174 research outputs found
A Broad Class of Discrete-Time Hypercomplex-Valued Hopfield Neural Networks
In this paper, we address the stability of a broad class of discrete-time
hypercomplex-valued Hopfield-type neural networks. To ensure the neural
networks belonging to this class always settle down at a stationary state, we
introduce novel hypercomplex number systems referred to as real-part
associative hypercomplex number systems. Real-part associative hypercomplex
number systems generalize the well-known Cayley-Dickson algebras and real
Clifford algebras and include the systems of real numbers, complex numbers,
dual numbers, hyperbolic numbers, quaternions, tessarines, and octonions as
particular instances. Apart from the novel hypercomplex number systems, we
introduce a family of hypercomplex-valued activation functions called
-projection functions. Broadly speaking, a
-projection function projects the activation potential onto the
set of all possible states of a hypercomplex-valued neuron. Using the theory
presented in this paper, we confirm the stability analysis of several
discrete-time hypercomplex-valued Hopfield-type neural networks from the
literature. Moreover, we introduce and provide the stability analysis of a
general class of Hopfield-type neural networks on Cayley-Dickson algebras
Complex Neural Networks for Audio
Audio is represented in two mathematically equivalent ways: the real-valued time domain (i.e., waveform) and the complex-valued frequency domain (i.e., spectrum). There are advantages to the frequency-domain representation, e.g., the human auditory system is known to process sound in the frequency-domain. Furthermore, linear time-invariant systems are convolved with sources in the time-domain, whereas they may be factorized in the frequency-domain. Neural networks have become rather useful when applied to audio tasks such as machine listening and audio synthesis, which are related by their dependencies on high quality acoustic models. They ideally encapsulate fine-scale temporal structure, such as that encoded in the phase of frequency-domain audio, yet there are no authoritative deep learning methods for complex audio. This manuscript is dedicated to addressing the shortcoming. Chapter 2 motivates complex networks by their affinity with complex-domain audio, while Chapter 3 contributes methods for building and optimizing complex networks. We show that the naive implementation of Adam optimization is incorrect for complex random variables and show that selection of input and output representation has a significant impact on the performance of a complex network. Experimental results with novel complex neural architectures are provided in the second half of this manuscript. Chapter 4 introduces a complex model for binaural audio source localization. We show that, like humans, the complex model can generalize to different anatomical filters, which is important in the context of machine listening. The complex model\u27s performance is better than that of the real-valued models, as well as real- and complex-valued baselines. Chapter 5 proposes a two-stage method for speech enhancement. In the first stage, a complex-valued stochastic autoencoder projects complex vectors to a discrete space. In the second stage, long-term temporal dependencies are modeled in the discrete space. The autoencoder raises the performance ceiling for state of the art speech enhancement, but the dynamic enhancement model does not outperform other baselines. We discuss areas for improvement and note that the complex Adam optimizer improves training convergence over the naive implementation
An Introduction to Quaternion-Valued Recurrent Projection Neural Networks
Hypercomplex-valued neural networks, including quaternion-valued neural
networks, can treat multi-dimensional data as a single entity. In this paper,
we introduce the quaternion-valued recurrent projection neural networks
(QRPNNs). Briefly, QRPNNs are obtained by combining the non-local projection
learning with the quaternion-valued recurrent correlation neural network
(QRCNNs). We show that QRPNNs overcome the cross-talk problem of QRCNNs. Thus,
they are appropriate to implement associative memories. Furthermore,
computational experiments reveal that QRPNNs exhibit greater storage capacity
and noise tolerance than their corresponding QRCNNs.Comment: Accepted to be Published in: Proceedings of the 8th Brazilian
Conference on Intelligent Systems (BRACIS 2019), October 15-18, 2019,
Salvador, BA, Brazi
Programming multi-level quantum gates in disordered computing reservoirs via machine learning and TensorFlow
Novel machine learning computational tools open new perspectives for quantum
information systems. Here we adopt the open-source programming library
TensorFlow to design multi-level quantum gates including a computing reservoir
represented by a random unitary matrix. In optics, the reservoir is a
disordered medium or a multi-modal fiber. We show that trainable operators at
the input and the readout enable one to realize multi-level gates. We study
various qudit gates, including the scaling properties of the algorithms with
the size of the reservoir. Despite an initial low slop learning stage,
TensorFlow turns out to be an extremely versatile resource for designing gates
with complex media, including different models that use spatial light
modulators with quantized modulation levels.Comment: Added a new section and a new figure about implementation of the
gates by a single spatial light modulator. 9 pages and 4 figure
Substantiation of the backpropagation technique via the Hamilton—Pontryagin formalism for training nonconvex nonsmooth neural networks
The paper observes the similarity between the stochastic optimal control over discrete dynamical systems and the
lear ning multilayer neural networks. It focuses on contemporary deep networks with nonconvex nonsmooth loss and
activation functions. The machine learning problems are treated as nonconvex nonsmooth stochastic optimization
ones. As a model of nonsmooth nonconvex dependences, the so-called generalized differentiable functions are used.
A method for calculating the stochastic generalized gradients of a learning quality functional for such systems is
substantiated basing on the Hamilton—Pontryagin formalism. This method extends a well-known “backpropagation”
machine learning technique to nonconvex nonsmooth networks. Stochastic generalized gradient learning algorithms
are extended for training nonconvex nonsmooth neural networks.Простежується аналогія між задачами оптимального керування дискретними стохастичними динамічними системами та задачами навчання багатошарових нейронних мереж. Увага концентрується на вивченні
сучасних глибоких мереж з негладкими цільовими функціоналами і зв'язками. Показано, що задачі машинного навчання можуть трактуватися як задачі стохастичного програмування, і для їхнього аналізу застосовано теорію неопуклого негладкого стохастичного програмування. Як модель негладких неопуклих
залежностей використано так звані узагальнено диференційовані функції. Обґрунтовано метод обчислення стохастичних узагальнених градієнтів функціонала якості навчання для таких систем на основі формалізму Гамільтона—Понтрягіна. Цей метод узагальнює відомий метод “зворотного просування похибки” на задачі навчання негладких неопуклих мереж. Узагальнені (стохастичні) градієнтні алгоритми навчання
поширено на неопуклі негладкі нейронні мережі.Прослеживается аналогия между задачами оптимального управления дискретными стохастическими динамическими системами и задачами обучения многослойных нейронных сетей. Внимание концентрируется на изучении современных глубоких сетей с негладкими целевыми функционалами и связями. Показано, что задачи машинного обучения могут трактоваться как задачи стохастического программирования,
и для их анализа применена теория невыпуклого негладкого стохастического программирования. В качестве модели негладких невыпуклых зависимостей использованы так называемые обобщенно дифференцируемые функции. Обоснован метод вычисления стохастических обобщенных градиентов функционала качества обучения для таких систем на основе формализма Гамильтона—Понтрягина. Этот метод обобщает известный метод “обратного распространения ошибки” на задачи обучения негладких невыпуклых сетей. Обобщенные (стохастические) градиентные алгоритмы обучения распространены на невыпуклые негладкие нейронные сети
- …