A Short Tutorial on The Weisfeiler-Lehman Test And Its Variants

Graph neural networks are designed to learn functions on graphs. Typically, the relevant target functions are invariant with respect to actions by permutations. Therefore the design of some graph neural network architectures has been inspired by graph-isomorphism algorithms. The classical Weisfeiler-Lehman algorithm (WL) -- a graph-isomorphism test based on color refinement -- became relevant to the study of graph neural networks. The WL test can be generalized to a hierarchy of higher-order tests, known as $k$-WL. This hierarchy has been used to characterize the expressive power of graph neural networks, and to inspire the design of graph neural network architectures. A few variants of the WL hierarchy appear in the literature. The goal of this short note is pedagogical and practical: We explain the differences between the WL and folklore-WL formulations, with pointers to existing discussions in the literature. We illuminate the differences between the formulations by visualizing an example

Approximately Equivariant Graph Networks

Graph neural networks (GNNs) are commonly described as being permutation equivariant with respect to node relabeling in the graph. This symmetry of GNNs is often compared to the translation equivariance symmetry of Euclidean convolution neural networks (CNNs). However, these two symmetries are fundamentally different: The translation equivariance of CNNs corresponds to symmetries of the fixed domain acting on the image signal (sometimes known as active symmetries), whereas in GNNs any permutation acts on both the graph signals and the graph domain (sometimes described as passive symmetries). In this work, we focus on the active symmetries of GNNs, by considering a learning setting where signals are supported on a fixed graph. In this case, the natural symmetries of GNNs are the automorphisms of the graph. Since real-world graphs tend to be asymmetric, we relax the notion of symmetries by formalizing approximate symmetries via graph coarsening. We present a bias-variance formula that quantifies the tradeoff between the loss in expressivity and the gain in the regularity of the learned estimator, depending on the chosen symmetry group. To illustrate our approach, we conduct extensive experiments on image inpainting, traffic flow prediction, and human pose estimation with different choices of symmetries. We show theoretically and empirically that the best generalization performance can be achieved by choosing a suitably larger group than the graph automorphism group, but smaller than the full permutation group

A Spectral Analysis of Graph Neural Networks on Dense and Sparse Graphs

In this work we propose a random graph model that can produce graphs at different levels of sparsity. We analyze how sparsity affects the graph spectra, and thus the performance of graph neural networks (GNNs) in node classification on dense and sparse graphs. We compare GNNs with spectral methods known to provide consistent estimators for community detection on dense graphs, a closely related task. We show that GNNs can outperform spectral methods on sparse graphs, and illustrate these results with numerical examples on both synthetic and real graphs.Comment: Extended version of ICASSP 2024 submissio

Fine-grained Expressivity of Graph Neural Networks

Numerous recent works have analyzed the expressive power of message-passing graph neural networks (MPNNs), primarily utilizing combinatorial techniques such as the $1$-dimensional Weisfeiler-Leman test ($1$-WL) for the graph isomorphism problem. However, the graph isomorphism objective is inherently binary, not giving insights into the degree of similarity between two given graphs. This work resolves this issue by considering continuous extensions of both $1$-WL and MPNNs to graphons. Concretely, we show that the continuous variant of $1$-WL delivers an accurate topological characterization of the expressive power of MPNNs on graphons, revealing which graphs these networks can distinguish and the level of difficulty in separating them. We identify the finest topology where MPNNs separate points and prove a universal approximation theorem. Consequently, we provide a theoretical framework for graph and graphon similarity combining various topological variants of classical characterizations of the $1$-WL. In particular, we characterize the expressive power of MPNNs in terms of the tree distance, which is a graph distance based on the concepts of fractional isomorphisms, and substructure counts via tree homomorphisms, showing that these concepts have the same expressive power as the $1$-WL and MPNNs on graphons. Empirically, we validate our theoretical findings by showing that randomly initialized MPNNs, without training, exhibit competitive performance compared to their trained counterparts. Moreover, we evaluate different MPNN architectures based on their ability to preserve graph distances, highlighting the significance of our continuous $1$-WL test in understanding MPNNs' expressivity

From Local to Global: Spectral-Inspired Graph Neural Networks

Graph Neural Networks (GNNs) are powerful deep learning methods for Non-Euclidean data. Popular GNNs are message-passing algorithms (MPNNs) that aggregate and combine signals in a local graph neighborhood. However, shallow MPNNs tend to miss long-range signals and perform poorly on some heterophilous graphs, while deep MPNNs can suffer from issues like over-smoothing or over-squashing. To mitigate such issues, existing works typically borrow normalization techniques from training neural networks on Euclidean data or modify the graph structures. Yet these approaches are not well-understood theoretically and could increase the overall computational complexity. In this work, we draw inspirations from spectral graph embedding and propose $\texttt{PowerEmbed}$ -- a simple layer-wise normalization technique to boost MPNNs. We show $\texttt{PowerEmbed}$ can provably express the top-$k$ leading eigenvectors of the graph operator, which prevents over-smoothing and is agnostic to the graph topology; meanwhile, it produces a list of representations ranging from local features to global signals, which avoids over-squashing. We apply $\texttt{PowerEmbed}$ in a wide range of simulated and real graphs and demonstrate its competitive performance, particularly for heterophilous graphs.Comment: Accepted for publication at the NeurIPS 2022 GLFrontiers Worksho

Deep Learning is Provably Robust to Symmetric Label Noise

Deep neural networks (DNNs) are capable of perfectly fitting the training data, including memorizing noisy data. It is commonly believed that memorization hurts generalization. Therefore, many recent works propose mitigation strategies to avoid noisy data or correct memorization. In this work, we step back and ask the question: Can deep learning be robust against massive label noise without any mitigation? We provide an affirmative answer for the case of symmetric label noise: We find that certain DNNs, including under-parameterized and over-parameterized models, can tolerate massive symmetric label noise up to the information-theoretic threshold. By appealing to classical statistical theory and universal consistency of DNNs, we prove that for multiclass classification, $L_1$-consistent DNN classifiers trained under symmetric label noise can achieve Bayes optimality asymptotically if the label noise probability is less than $\frac{K-1}{K}$, where $K \ge 2$ is the number of classes. Our results show that for symmetric label noise, no mitigation is necessary for $L_1$-consistent estimators. We conjecture that for general label noise, mitigation strategies that make use of the noisy data will outperform those that ignore the noisy data

TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities

Recently, the success of pre-training in text domain has been fully extended to vision, audio, and cross-modal scenarios. The proposed pre-training models of different modalities are showing a rising trend of homogeneity in their model structures, which brings the opportunity to implement different pre-training models within a uniform framework. In this paper, we present TencentPretrain, a toolkit supporting pre-training models of different modalities. The core feature of TencentPretrain is the modular design. The toolkit uniformly divides pre-training models into 5 components: embedding, encoder, target embedding, decoder, and target. As almost all of common modules are provided in each component, users can choose the desired modules from different components to build a complete pre-training model. The modular design enables users to efficiently reproduce existing pre-training models or build brand-new one. We test the toolkit on text, vision, and audio benchmarks and show that it can match the performance of the original implementations

Short-term outcomes of robot-assisted versus video-assisted thoracoscopic surgery for non-small cell lung cancer patients with neoadjuvant immunochemotherapy: a single-center retrospective study

BackgroundNeoadjuvant immunochemotherapy has been increasingly applied to treat non-small cell lung cancer (NSCLC). However, the comparison between robotic-assisted thoracoscopic surgery (RATS) and video-assisted thoracoscopic surgery (VATS) in the feasibility and oncological efficacy following neoadjuvant immunochemotherapy is scarce. This study aims to assess the superiorities of RATS over (VATS) concerning short-term outcomes in treating NSCLC patients with neoadjuvant immunochemotherapy.MethodsNSCLC patients receiving RATS or VATS lobectomy following neoadjuvant immunochemotherapy at Shanghai Chest Hospital from 2019 to 2022 were retrospectively identified. Baseline clinical characteristics, perioperative outcomes, and survival profiles were analyzed.ResultsForty-six NSCLC patients with neoadjuvant immunochemotherapy were included and divided into the RATS (n=15) and VATS (n=31) groups. The baseline clinical characteristics and induction-related adverse events were comparable between the two groups (all p>0.050). The 30-day mortality in the RATS and VATS groups were 0% and 3.23%, respectively (p=1.000). Patients undergoing RATS were associated with reduced surgical-related intensive unit care (ICU) stay than those receiving VATS (0.0 [0.0-0.0] vs. 0.0 [0.0-1.0] days, p=0.026). Moreover, RATS assessed more N1 LNs (6.27 ± 1.94 vs 4.90 ± 1.92, p=0.042) and LN stations (3.07 ± 1.03 vs 2.52 ± 0.57, p=0.038) compared with VATS. By comparison, no difference was found in surgical outcomes, pathological results, and postoperative complications between the RATS and VATS groups (all p>0.050). Finally, RATS and VATS achieved comparable one-year recurrence-free survival (82.96% vs. 85.23%, p=0.821) and the timing of central nervous system, LN, and bone recurrences (all p>0.050).ConclusionRATS is safe and feasible for NSCLC patients with neoadjuvant immunochemotherapy, reducing surgical-related ICU stay, assessing increased N1 LNs and stations, and achieving similar survival profiles to VATS

Moisture and temperature influences on nonlinear vegetation trends in Serengeti National Park

While long-term vegetation greening trends have appeared across large land areas over the late 20th century, uncertainty remains in identifying and attributing finer-scale vegetation changes and trends, particularly across protected areas. Serengeti National Park (SNP) is a critical East African protected area, where seasonal vegetation cycles support vast populations of grazing herbivores and a host of ecosystem dynamics. Previous work has shown how non-climate drivers (e.g. land use) shape the SNP ecosystem, but it is still unclear to what extent changing climate conditions influence SNP vegetation, particularly at finer spatial and temporal scales. We fill this research gap by evaluating long-term (1982–2016) changes in SNP leaf area index (LAI) in relation to both temperature and moisture availability using Ensemble Empirical Mode Decomposition and Principal Component Analysis with regression techniques. We find that SNP LAI trends are nonlinear, display high sub-seasonal variation, and are influenced by lagged changes in both moisture and temperature variables and their interactions. LAI during the long rains (e.g. March) exhibits a greening-to-browning trend reversal starting in the early 2000s, partly due to antecedent precipitation declines. In contrast, LAI during the short rains (e.g. November, December) displays browning-to-greening alongside increasing moisture availability. Rising temperature trends also have important, secondary interactions with moisture variables to shape these SNP vegetation trends. Our findings show complex vegetation-climate interactions occurring at important temporal and spatial scales of the SNP, and our rigorous statistical approaches detect these complex climate-vegetation trends and interactions, while guarding against spurious vegetation signals