Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives

Part 2 of this monograph builds on the introduction to tensor networks and their operations presented in Part 1. It focuses on tensor network models for super-compressed higher-order representation of data/parameters and related cost functions, while providing an outline of their applications in machine learning and data analytics. A particular emphasis is on the tensor train (TT) and Hierarchical Tucker (HT) decompositions, and their physically meaningful interpretations which reflect the scalability of the tensor network approach. Through a graphical approach, we also elucidate how, by virtue of the underlying low-rank tensor approximations and sophisticated contractions of core tensors, tensor networks have the ability to perform distributed computations on otherwise prohibitively large volumes of data/parameters, thereby alleviating or even eliminating the curse of dimensionality. The usefulness of this concept is illustrated over a number of applied areas, including generalized regression and classification (support tensor machines, canonical correlation analysis, higher order partial least squares), generalized eigenvalue decomposition, Riemannian optimization, and in the optimization of deep neural networks. Part 1 and Part 2 of this work can be used either as stand-alone separate texts, or indeed as a conjoint comprehensive review of the exciting field of low-rank tensor networks and tensor decompositions.Comment: 232 page

Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives

Part 2 of this monograph builds on the introduction to tensor networks and their operations presented in Part 1. It focuses on tensor network models for super-compressed higher-order representation of data/parameters and related cost functions, while providing an outline of their applications in machine learning and data analytics. A particular emphasis is on the tensor train (TT) and Hierarchical Tucker (HT) decompositions, and their physically meaningful interpretations which reflect the scalability of the tensor network approach. Through a graphical approach, we also elucidate how, by virtue of the underlying low-rank tensor approximations and sophisticated contractions of core tensors, tensor networks have the ability to perform distributed computations on otherwise prohibitively large volumes of data/parameters, thereby alleviating or even eliminating the curse of dimensionality. The usefulness of this concept is illustrated over a number of applied areas, including generalized regression and classification (support tensor machines, canonical correlation analysis, higher order partial least squares), generalized eigenvalue decomposition, Riemannian optimization, and in the optimization of deep neural networks. Part 1 and Part 2 of this work can be used either as stand-alone separate texts, or indeed as a conjoint comprehensive review of the exciting field of low-rank tensor networks and tensor decompositions.Comment: 232 page

Enhancing Deep Learning Models through Tensorization: A Comprehensive Survey and Framework

The burgeoning growth of public domain data and the increasing complexity of deep learning model architectures have underscored the need for more efficient data representation and analysis techniques. This paper is motivated by the work of (Helal, 2023) and aims to present a comprehensive overview of tensorization. This transformative approach bridges the gap between the inherently multidimensional nature of data and the simplified 2-dimensional matrices commonly used in linear algebra-based machine learning algorithms. This paper explores the steps involved in tensorization, multidimensional data sources, various multiway analysis methods employed, and the benefits of these approaches. A small example of Blind Source Separation (BSS) is presented comparing 2-dimensional algorithms and a multiway algorithm in Python. Results indicate that multiway analysis is more expressive. Contrary to the intuition of the dimensionality curse, utilising multidimensional datasets in their native form and applying multiway analysis methods grounded in multilinear algebra reveal a profound capacity to capture intricate interrelationships among various dimensions while, surprisingly, reducing the number of model parameters and accelerating processing. A survey of the multi-away analysis methods and integration with various Deep Neural Networks models is presented using case studies in different application domains.Comment: 34 pages, 8 figures, 4 table

Simulation and Theory of Large-Scale Cortical Networks

Cerebral cortex is composed of intricate networks of neurons. These neuronal networks are strongly interconnected: every neuron receives, on average, input from thousands or more presynaptic neurons. In fact, to support such a number of connections, a majority of the volume in the cortical gray matter is filled by axons and dendrites. Besides the networks, neurons themselves are also highly complex. They possess an elaborate spatial structure and support various types of active processes and nonlinearities. In the face of such complexity, it seems necessary to abstract away some of the details and to investigate simplified models. In this thesis, such simplified models of neuronal networks are examined on varying levels of abstraction. Neurons are modeled as point neurons, both rate-based and spike-based, and networks are modeled as block-structured random networks. Crucially, on this level of abstraction, the models are still amenable to analytical treatment using the framework of dynamical mean-field theory. The main focus of this thesis is to leverage the analytical tractability of random networks of point neurons in order to relate the network structure, and the neuron parameters, to the dynamics of the neurons—in physics parlance, to bridge across the scales from neurons to networks. More concretely, four different models are investigated: 1) fully connected feedforward networks and vanilla recurrent networks of rate neurons; 2) block-structured networks of rate neurons in continuous time; 3) block-structured networks of spiking neurons; and 4) a multi-scale, data-based network of spiking neurons. We consider the first class of models in the light of Bayesian supervised learning and compute their kernel in the infinite-size limit. In the second class of models, we connect dynamical mean-field theory with large-deviation theory, calculate beyond mean-field fluctuations, and perform parameter inference. For the third class of models, we develop a theory for the autocorrelation time of the neurons. Lastly, we consolidate data across multiple modalities into a layer- and population-resolved model of human cortex and compare its activity with cortical recordings. In two detours from the investigation of these four network models, we examine the distribution of neuron densities in cerebral cortex and present a software toolbox for mean-field analyses of spiking networks

Genetic determination and layout rules of visual cortical architecture

The functional architecture of the primary visual cortex is set up by neurons that preferentially respond to visual stimuli with contours of a specific orientation in visual space. In primates and placental carnivores, orientation preference is arranged into continuous and roughly repetitive (iso-) orientation domains. Exceptions are pinwheels that are surrounded by all orientation preferences. The configuration of pinwheels adheres to quantitative species-invariant statistics, the common design. This common design most likely evolved independently at least twice in the course of the past 65 million years, which might indicate a functionally advantageous trait. The possible acquisition of environment-dependent functional traits by genes, the Baldwin effect, makes it conceivable that visual cortical architecture is partially or redundantly encoded by genetic information. In this conception, genetic mechanisms support the emergence of visual cortical architecture or even establish it under unfavorable environments. In this dissertation, I examine the capability of genetic mechanisms for encoding visual cortical architecture and mathematically dissect the pinwheel configuration under measurement noise as well as in different geometries. First, I theoretically explore possible roles of genetic mechanisms in visual cortical development that were previously excluded from theoretical research, mostly because the information capacity of the genome appeared too small to contain a blueprint for wiring up the cortex. For the first time, I provide a biologically plausible scheme for quantitatively encoding functional visual cortical architecture by genetic information that circumvents the alleged information bottleneck. Key ingredients for this mechanism are active transport and trans-neuronal signaling as well as joined dynamics of morphogens and connectome. This theory provides predictions for experimental tests and thus may help to clarify the relative importance of genes and environments on complex human traits. Second, I disentangle the link between orientation domain ensembles and the species-invariant pinwheel statistics of the common design. This examination highlights informative measures of pinwheel configurations for model benchmarking. Third, I mathematically investigate the susceptibility of the pinwheel configuration to measurement noise. The results give rise to an extrapolation method of pinwheel densities to the zero noise limit and provide an approximated analytical expression for confidence regions of pinwheel centers. Thus, the work facilitates high-precision measurements and enhances benchmarking for devising more accurate models of visual cortical development. Finally, I shed light on genuine three-dimensional properties of functional visual cortical architectures. I devise maximum entropy models of three-dimensional functional visual cortical architectures in different geometries. This theory enables the examination of possible evolutionary transitions between different functional architectures for which intermediate organizations might still exist

Computational Methods in Science and Engineering : Proceedings of the Workshop SimLabs@KIT, November 29 - 30, 2010, Karlsruhe, Germany

In this proceedings volume we provide a compilation of article contributions equally covering applications from different research fields and ranging from capacity up to capability computing. Besides classical computing aspects such as parallelization, the focus of these proceedings is on multi-scale approaches and methods for tackling algorithm and data complexity. Also practical aspects regarding the usage of the HPC infrastructure and available tools and software at the SCC are presented

Altered brain criticality in schizophrenia: new insights from magnetoencephalography

Schizophrenia has a complex etiology and symptomatology that is difficult to untangle. After decades of research, important advancements toward a central biomarker are still lacking. One of the missing pieces is a better understanding of how non-linear neural dynamics are altered in this patient population. In this study, the resting-state neuromagnetic signals of schizophrenia patients and healthy controls were analyzed in the framework of criticality. When biological systems like the brain are in a state of criticality, they are thought to be functioning at maximum efficiency (e.g., optimal communication and storage of information) and with maximum adaptability to incoming information. Here, we assessed the self-similarity and multifractality of resting-state brain signals recorded with magnetoencephalography in patients with schizophrenia patients and in matched controls. Schizophrenia patients had similar, although attenuated, patterns of self-similarity and multifractality values. Statistical tests showed that patients had higher values of self-similarity than controls in fronto-temporal regions, indicative of more regularity and memory in the signal. In contrast, patients had less multifractality than controls in the parietal and occipital regions, indicative of less diverse singularities and reduced variability in the signal. In addition, supervised machine-learning, based on logistic regression, successfully discriminated the two groups using measures of self-similarity and multifractality as features. Our results provide new insights into the baseline cognitive functioning of schizophrenia patients by identifying key alterations of criticality properties in their resting-state brain data

Advanced robust non-invasive foetal heart detection techniques during active labour using one pair of transabdominal electrodes

The thesis proposes and evaluates three state-of-the-art signal processing techniques to detect fetal heartbeats within each maternal cardiac cycle, during labour contractions, using only a pair of transabdominal electrodes. The first and second techniques are, namely, the structured third- order cumulant-slice-template matching and the bispectral-contours-template matching for fetal QRS identification, respectively. The third technique is based on the modified and appropriately weighted spectral multiple signal classification (MUSIC) with incorporated covariance matrix for uterine contraction noise-like interfering signals also contaminated with noise. Essentially, two modifications to the standard MUSIC have been developed in order to enhance the performance of the spectral estimator in our applied work. The first modification involves the introduction of an optimised weighting function to the segmented ECG covariance matrix, and is chiefly aimed at enhancing the fetal QRS major spectral peak which occurs at around 30 Hz against the mother QRS major spectral peak usually occurring around 17 Hz and all other noise contributions. Additional optional pseudo-bispectral enhancement to sharpen the maternal and fetal spectral peaks, in particular when the mother and fetal R-waves are temporally coincident, have been achieved. The second modification to the spectral MUSIC is the removal of the unjustified assumption that only white Gaussian noise is present and the incorporation of the actual measured labour uterine contraction covariance matrix in reconfigured subspace analysis. This inevitably leads to the generalised eigenvectors - eigenvalues decomposition modern signal processing. This is now coined the modified, interference incorporated pseudo-spectral MUSIC. The above mentioned first and second techniques are higher-order statistics-based (HOS) and hybrid involving both signal processing and NN classifiers. The third technique is second-order statistics-based (SOS). In all techniques, the removal of signal non-linearity with the aid of non-linear Volterra synthesisers plays a crucial part in the fetal detection integrity. Accurately assessed fetal heart classification rates as high as 95% have been achieved during labour, thus helping to provide non-invasive transparency to fetal intrapartum welfare. Performance analysis and evaluation processes involved more than 30 critical cases classified as “fetal under stress in labour” recorded in a London hospital database and used both transbadominal ECG electrodes and fetal scalp electrodes. The latter facilitates detection of the instantaneous fetal heart rate which is then used as the Reference Fetal Heart Rate in the assessment of the classification rate of each of the above mentioned techniques. It will be shown that the fetal heartbeats are completely masked by uterine activity and noise artefacts in all the recorded transabdominal maternal ECG signals. The fetal scalp electrode was, therefore, deemed necessary to provide the highest accurate measure of fetal heart functionality (from the hospital viewpoint), and in the assessment of the three non-invasive techniques presented in this thesis. The techniques may also be used during gestation and as early as 10 weeks