Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks

Since the recognition in the early nineties of the vanishing/exploding (V/E) gradient issue plaguing the training of neural networks (NNs), significant efforts have been exerted to overcome this obstacle. However, a clear solution to the V/E issue remained elusive so far. In this manuscript a new architecture of NN is proposed, designed to mathematically prevent the V/E issue to occur. The pursuit of approximate dynamical isometry, i.e. parameter configurations where the singular values of the input-output Jacobian are tightly distributed around 1, leads to the derivation of a NN's architecture that shares common traits with the popular Residual Network model. Instead of skipping connections between layers, the idea is to filter the previous activations orthogonally and add them to the nonlinear activations of the next layer, realising a convex combination between them. Remarkably, the impossibility for the gradient updates to either vanish or explode is demonstrated with analytical bounds that hold even in the infinite depth case. The effectiveness of this method is empirically proved by means of training via backpropagation an extremely deep multilayer perceptron of 50k layers, and an Elman NN to learn long-term dependencies in the input of 10k time steps in the past. Compared with other architectures specifically devised to deal with the V/E problem, e.g. LSTMs for recurrent NNs, the proposed model is way simpler yet more effective. Surprisingly, a single layer vanilla RNN can be enhanced to reach state of the art performance, while converging super fast; for instance on the psMNIST task, it is possible to get test accuracy of over 94% in the first epoch, and over 98% after just 10 epochs

Transitions in echo index and dependence on input repetitions

The echo index counts the number of simultaneously stable asymptotic responses of a nonautonomous (i.e. input-driven) dynamical system. It generalizes the well-known echo state property for recurrent neural networks - this corresponds to the echo index being equal to one. In this paper, we investigate how the echo index depends on parameters that govern typical responses to a finite-state ergodic external input that forces the dynamics. We consider the echo index for a nonautonomous system that switches between a finite set of maps, where we assume that each map possesses a finite set of hyperbolic equilibrium attractors. We find the minimum and maximum repetitions of each map are crucial for the resulting echo index. Casting our theoretical findings in the RNN computing framework, we obtain that for small amplitude forcing the echo index corresponds to the number of attractors for the input-free system, while for large amplitude forcing, the echo index reduces to one. The intermediate regime is the most interesting; in this region the echo index depends not just on the amplitude of forcing but also on more subtle properties of the input

Edge of stability echo state networks

Echo State Networks (ESNs) are time-series processing models working under the Echo State Property (ESP) principle. The ESP is a notion of stability that imposes an asymptotic fading of the memory of the input. On the other hand, the resulting inherent architectural bias of ESNs may lead to an excessive loss of information, which in turn harms the performance in certain tasks with long short-term memory requirements. With the goal of bringing together the fading memory property and the ability to retain as much memory as possible, in this paper we introduce a new ESN architecture, called the Edge of Stability Echo State Network (ES$^2$N). The introduced ES$^2$N model is based on defining the reservoir layer as a convex combination of a nonlinear reservoir (as in the standard ESN), and a linear reservoir that implements an orthogonal transformation. We provide a thorough mathematical analysis of the introduced model, proving that the whole eigenspectrum of the Jacobian of the ES$^2$N map can be contained in an annular neighbourhood of a complex circle of controllable radius, and exploit this property to demonstrate that the ES$^2$N's forward dynamics evolves close to the edge-of-chaos regime by design. Remarkably, our experimental analysis shows that the newly introduced reservoir model is able to reach the theoretical maximum short-term memory capacity. At the same time, in comparison to standard ESN, ES$^2$N is shown to offer an excellent trade-off between memory and nonlinearity, as well as a significant improvement of performance in autoregressive nonlinear modeling

Interpreting recurrent neural networks behaviour via excitable network attractors

Introduction: Machine learning provides fundamental tools both for scientific research and for the development of technologies with significant impact on society. It provides methods that facilitate the discovery of regularities in data and that give predictions without explicit knowledge of the rules governing a system. However, a price is paid for exploiting such flexibility: machine learning methods are typically black-boxes where it is difficult to fully understand what the machine is doing or how it is operating. This poses constraints on the applicability and explainability of such methods. Methods: Our research aims to open the black-box of recurrent neural networks, an important family of neural networks used for processing sequential data. We propose a novel methodology that provides a mechanistic interpretation of behaviour when solving a computational task. Our methodology uses mathematical constructs called excitable network attractors, which are invariant sets in phase space composed of stable attractors and excitable connections between them. Results and Discussion: As the behaviour of recurrent neural networks depends both on training and on inputs to the system, we introduce an algorithm to extract network attractors directly from the trajectory of a neural network while solving tasks. Simulations conducted on a controlled benchmark task confirm the relevance of these attractors for interpreting the behaviour of recurrent neural networks, at least for tasks that involve learning a finite number of stable states and transitions between them.Comment: revised versio

Cross frequency coupling in next generation inhibitory neural mass models

Coupling among neural rhythms is one of the most important mechanisms at the basis of cognitive processes in the brain. In this study we consider a neural mass model, rigorously obtained from the microscopic dynamics of an inhibitory spiking network with exponential synapses, able to autonomously generate collective oscillations (COs). These oscillations emerge via a super-critical Hopf bifurcation, and their frequencies are controlled by the synaptic time scale, the synaptic coupling and the excitability of the neural population. Furthermore, we show that two inhibitory populations in a master-slave configuration with different synaptic time scales can display various collective dynamical regimes: namely, damped oscillations towards a stable focus, periodic and quasi-periodic oscillations, and chaos. Finally, when bidirectionally coupled the two inhibitory populations can exhibit different types of theta-gamma cross-frequency couplings (CFCs): namely, phase-phase and phase-amplitude CFC. The coupling between theta and gamma COs is enhanced in presence of a external theta forcing, reminiscent of the type of modulation induced in Hippocampal and Cortex circuits via optogenetic drive.Comment: 14 pages, 10 figure

Oxidative Stress in the Healthy and Wounded Hepatocyte: A Cellular Organelles Perspective

Accurate control of the cell redox state is mandatory for maintaining the structural integrity and physiological functions. This control is achieved both by a fine-tuned balance between prooxidant and anti-oxidant molecules and by spatial and temporal confinement of the oxidative species. The diverse cellular compartments each, although structurally and functionally related, actively maintain their own redox balance, which is necessary to fulfill specialized tasks. Many fundamental cellular processes such as insulin signaling, cell proliferation and differentiation and cell migration and adhesion, rely on localized changes in the redox state of signal transducers, which is mainly mediated by hydrogen peroxide (H2O2). Therefore, oxidative stress can also occur long before direct structural damage to cellular components, by disruption of the redox circuits that regulate the cellular organelles homeostasis. The hepatocyte is a systemic hub integrating the whole body metabolic demand, iron homeostasis and detoxification processes, all of which are redox-regulated processes. Imbalance of the hepatocyte’s organelles redox homeostasis underlies virtually any liver disease and is a field of intense research activity. This review recapitulates the evolving concept of oxidative stress in the diverse cellular compartments, highlighting the principle mechanisms of oxidative stress occurring in the healthy and wounded hepatocyte

Sentinel node mapping in endometrial cancer: Tips and tricks to improve bilateral detection rate. The sentitricks study, a monocentric experience.

Abstract Objective The objective of the study is to show some small tricks for bilateral sentinel lymph node (SLN) uptake in endometrial cancer. Materials and methods Each step of the sentinel lymph node technique was analyzed. The cervix was exposed through the use of vaginal valves and by Martin pliers stapling of the anterior cervical lip. Fifty mg Indocyanine Green (ICG) powder was diluted with 10 ml of physiological solution. The spinal needle was marked at 15 mm with a steri-strip. After 20 min from the administration, in case of no LNS identification, an additional 1 ml in the non-detected side was administered in the superficial cervical area. All cervical injections were made by a single (BR) surgeon experienced in oncological gynecology. Results Fifty patients undergoing sentinel lymph node research for endometrial cancer. The uptake of at least one side of the sentinel node was 98% (49 cases). Forty-six (92%) patients had bilateral lymph node uptake and 3 patients (6%) had unilateral uptake. Only one patient with pelvic and metastatic aortic lymph nodes had no sentinel nodal uptake. Conclusions Little tricks can increase the bilateral uptake of the SLN up to 92%. The reinjection could be a key element for the success of the SLN technique. Experienced surgeons could certainly play a fundamental role in raising bilateral SLN detection. Further prospective randomized studies are needed to achieve the best SLN infiltration strategy

Antidiabetic thiazolidinediones induce ductal differentiation but not apoptosis in pancreatic cancer cells

AIM: Thiazolidinediones (TZD) are a new class of oral antidiabetic drugs that have been shown to inhibit growth of same epithelial cancer cells. Although TZD were found to be ligands for peroxisome proliferator-activated receptor gamma (PPARgamma), the mechanism by which TZD exert their anticancer effect is presently unclear. In this study, we analyzed the mechanism by which TZD inhibit growth of human pancreatic carcinoma cell lines in order to evaluate the potential therapeutic use of these drugs in pancreatic adenocarcinoma. METHODS: The effects of TZD in pancreatic cancer cells were assessed in anchorage-independent growth assay. Expression of PPARgamma was measured by reverse-transcription polymerase chain reaction and confirmed by Western blot analysis. PPARgamma activity was evaluated by transient reporter gene assay. Flow cytometry and DNA fragmentation assay were used to determine the effect of TZD on cell cycle progression and apoptosis respectively. The effect of TZD on ductal differentiation markers was performed by Western blot. RESULTS: Exposure to TZD inhibited colony formation in a PPARgamma-dependent manner. Growth inhibition was linked to G1 phase cell cycle arrest through induction of the ductal differentiation program without any increase of the apoptotic rate. CONCLUSION: TZD treatment in pancr

8-Oxo-7,8-dihydro-2'-deoxyguanosine and other lesions along the coding strand of the exon 5 of the tumour suppressor gene P53 in a breast cancer case-control study.

The next-generation sequencing studies of breast cancer have reported that the tumour suppressor P53 (TP53) gene is mutated in more than 40% of the tumours. We studied the levels of oxidative lesions, including 8-oxo-7,8-dihydro-2′-deoxyguanosine (8-oxodG), along the coding strand of the exon 5 in breast cancer patients as well as in a reactive oxygen species (ROS)-attacked breast cancer cell line using the ligation-mediated polymerase chain reaction technique. We detected a significant ‘in vitro’ generation of 8-oxodG between the codons 163 and 175, corresponding to a TP53 region with high mutation prevalence, after treatment with xanthine plus xanthine oxidase, a ROS-generating system. Then, we evaluated the occurrence of oxidative lesions in the DNA-binding domain of the TP53 in the core needle biopsies of 113 of women undergoing breast investigation for diagnostic purpose. An increment of oxidative damage at the −G− residues into the codons 163 and 175 was found in the cancer cases as compared to the controls. We found significant associations with the pathological stage and the histological grade of tumours. As the major news of this study, this largest analysis of genomic footprinting of oxidative lesions at the TP53 sequence level to date provided a first roadmap describing the signatures of oxidative lesions in human breast cancer. Our results provide evidence that the generation of oxidative lesions at single nucleotide resolution is not an event highly stochastic, but causes a characteristic pattern of DNA lesions at the site of mutations in the TP53, suggesting causal relationship between oxidative DNA adducts and breast cancer