DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion

Real-world data generation often involves complex inter-dependencies among instances, violating the IID-data hypothesis of standard learning paradigms and posing a challenge for uncovering the geometric structures for learning desired instance representations. To this end, we introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states that progressively incorporate other instances' information by their interactions. The diffusion process is constrained by descent criteria w.r.t.~a principled energy function that characterizes the global consistency of instance representations over latent structures. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs, which gives rise to a new class of neural encoders, dubbed as DIFFormer (diffusion-based Transformers), with two instantiations: a simple version with linear complexity for prohibitive instance numbers, and an advanced version for learning complex structures. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks, such as node classification on large graphs, semi-supervised image/text classification, and spatial-temporal dynamics prediction.Comment: Accepted by International Conference on Learning Representations (ICLR 2023

Space adaptive and hierarchical Bayesian variational models for image restoration

The main contribution of this thesis is the proposal of novel space-variant regularization or penalty terms motivated by a strong statistical rational. In light of the connection between the classical variational framework and the Bayesian formulation, we will focus on the design of highly flexible priors characterized by a large number of unknown parameters. The latter will be automatically estimated by setting up a hierarchical modeling framework, i.e. introducing informative or non-informative hyperpriors depending on the information at hand on the parameters. More specifically, in the first part of the thesis we will focus on the restoration of natural images, by introducing highly parametrized distribution to model the local behavior of the gradients in the image. The resulting regularizers hold the potential to adapt to the local smoothness, directionality and sparsity in the data. The estimation of the unknown parameters will be addressed by means of non-informative hyperpriors, namely uniform distributions over the parameter domain, thus leading to the classical Maximum Likelihood approach. In the second part of the thesis, we will address the problem of designing suitable penalty terms for the recovery of sparse signals. The space-variance in the proposed penalties, corresponding to a family of informative hyperpriors, namely generalized gamma hyperpriors, will follow directly from the assumption of the independence of the components in the signal. The study of the properties of the resulting energy functionals will thus lead to the introduction of two hybrid algorithms, aimed at combining the strong sparsity promotion characterizing non-convex penalty terms with the desirable guarantees of convex optimization

Inferential stability in systems biology

The modern biological sciences are fraught with statistical difficulties. Biomolecular stochasticity, experimental noise, and the “large p, small n” problem all contribute to the challenge of data analysis. Nevertheless, we routinely seek to draw robust, meaningful conclusions from observations. In this thesis, we explore methods for assessing the effects of data variability upon downstream inference, in an attempt to quantify and promote the stability of the inferences we make. We start with a review of existing methods for addressing this problem, focusing upon the bootstrap and similar methods. The key requirement for all such approaches is a statistical model that approximates the data generating process. We move on to consider biomarker discovery problems. We present a novel algorithm for proposing putative biomarkers on the strength of both their predictive ability and the stability with which they are selected. In a simulation study, we find our approach to perform favourably in comparison to strategies that select on the basis of predictive performance alone. We then consider the real problem of identifying protein peak biomarkers for HAM/TSP, an inflammatory condition of the central nervous system caused by HTLV-1 infection. We apply our algorithm to a set of SELDI mass spectral data, and identify a number of putative biomarkers. Additional experimental work, together with known results from the literature, provides corroborating evidence for the validity of these putative biomarkers. Having focused on static observations, we then make the natural progression to time course data sets. We propose a (Bayesian) bootstrap approach for such data, and then apply our method in the context of gene network inference and the estimation of parameters in ordinary differential equation models. We find that the inferred gene networks are relatively unstable, and demonstrate the importance of finding distributions of ODE parameter estimates, rather than single point estimates

Output Feedback Controller for Operation of Spark Ignition Engines at Lean Conditions Using Neural Networks

Spark ignition (SI) engines operating at very lean conditions demonstrate significant nonlinear behavior by exhibiting cycle-to-cycle bifurcation of heat release. Past literature suggests that operating an engine under such lean conditions can significantly reduce NO emissions by as much as 30% and improve fuel efficiency by as much as 5%-10%. At lean conditions, the heat release per engine cycle is not close to constant, as it is when these engines operate under stoichiometric conditions where the equivalence ratio is 1.0. A neural network controller employing output feedback has shown ability in simulation to reduce the nonlinear cyclic dispersion observed under lean operating conditions. This neural network (NN) output controller consists of three NNs: a) an NN observer to estimate the states of the engine such as total fuel and air; b) a second NN for generating virtual input; and c) a third NN for generating actual control input. The uniform ultimate boundedness of all closed-loop signals is demonstrated by using the Lyapunov analysis without using the separation principle. Persistency of the excitation condition, the certainty equivalence principle, and the linearity in the unknown parameter assumptions are also relaxed. The controller is implemented for a research engine as a program running on an embeddable PC that communicates with the engine through a custom hardware interface, and the results are similar to those observed in simulation. Experimental results at an equivalence ratio of 0.77 show a drop in NO emissions by around 98% from stoichiometric levels with an improvement of fuel efficiency by 5%. A 30% drop in unburned hydrocarbons from uncontrolled case is observed at this equivalence ratio of 0.77. Similar performance was observed with the controller on a different engine

Neuroimaging, Networking and Systems Biology: The New Way to Investigate Pathologies with Neurological System Implications. The example of Tourette Syndrome as a Pilot Study

Purpose: Recently, many academic research groups have focused their attention on changes in human brain networks related to several kinds of pathologies and diseases, generating the new discipline termed “Network Medicine”. Purpose of this paper is to investigate the ability of the Network Medicine to give deeper insights in the functionality of brain activity. Material and Methods: In the proposed study of Tourette syndrome, we have investigated with the functional magnetic resonance imaging the possibility that the mechanisms associated with the monitoring and internal control of movements were compromised in individuals with Tourette syndrome; we enrolled 20 Tourette Syndrome patients in comparison with a healthy Controls group of 15 subjects matching for age and sex distribution. We proposed, for the fMRI analysis, a novel task based on the execution of switching between complex movements on demand. Results: The elementary activation model found that the effort related to the task in comparing Tourettic vs Controls mainly concerns the areas of the Gyrus of the Cingulum, the precuneus and the thalamic area of the ventral-lateral nucleus. In particular, the BA11 plays an essential role in the Tourette Patients related to the continue tentative to correct the TIC. Considering the status of the pilot study of this work, we remark the power of proposed methods to investigate the complex interaction of the brain networks. Conclusion: Alteration in brain activity for a population of Tourette Syndrome patients is evaluable by the use of complex indexes, results confirm the literature about this pathology and these medical physics methods can be applied to all neurological diseases investigation by opportune task-driven experiments or by resting state fc-MRI experiments

Hybrid structural health monitoring using data-driven modal analysis and model-based Bayesian inference.

Civil infrastructures that are valuable assets for the public and owners must be adequately and periodically maintained to guarantee safety, continuous service, and avoid economic losses. Vibration-based structural health monitoring (VBSHM) has been a significant tool to assess the structural performance of civil infrastructures over the last decades. Challenges in VBSHM exist in two aspects: operational modal analysis (OMA) and Finite element model updating (FEMU). The former aims to extract natural frequency, damping ratio, and mode shapes using vibrational data under normal operation; the latter focuses on minimizing the discrepancies between measurements and model prediction. The main impediments to real-world application of VBSHM include 1) uncertainties are inevitably involved due to measurement noise and modeling error; 2) computational burden in analyzing massive data and high-fidelity model; 3) updating structural coupled parameters, e.g., mass and stiffness. Bayesian model updating approach (BMUA) is an advanced FEMU technique to update structural parameters using modal data and account for underlying uncertainties. However, traditional BMUA generally assumes mass is precisely known and only updating stiffness to circumvent the coupling effect of mass and stiffness. Simultaneously updating mass and stiffness is necessary to fully understand the structural integrity, especially when the mass has a relatively large variation. To tackle these challenges, this dissertation proposed a hybrid framework using data-driven and model-based approaches in two sequential phases: automated OMA and a BMUA with added mass/stiffness. Automated stochastic subspace identification (SSI) and Bayesian modal identification are firstly developed to acquire modal properties. Following by a novel BMUA, new eigen-equations based on two sets of modal data from the original and modified system with added mass or stiffness are derived to address the coupling effect of structural parameters, e.g., mass and stiffness. To avoid multi-dimensional integrals, an asymptotic optimization method and Differential Evolutionary Adaptive Metropolis (DREAM) sampling algorithm are employed for Bayesian inference. To alleviate computational burden, variance-based global sensitivity analysis to reduce model dimensionality and Kriging model to substitute time-consuming FEM are integrated into BMUA. The proposed VBSHM are verified and illustrated using numerical, laboratory and field test data, achieving following goals: 1) properly treating parameter uncertainties; 2) substantially reducing the computational cost; 3) simultaneously updating structural parameters with addressing the coupling effect; 4) performing the probabilistic damage identification at an accurate level