Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A survey

Dynamic networks are used in a wide range of fields, including social network analysis, recommender systems, and epidemiology. Representing complex networks as structures changing over time allow network models to leverage not only structural but also temporal patterns. However, as dynamic network literature stems from diverse fields and makes use of inconsistent terminology, it is challenging to navigate. Meanwhile, graph neural networks (GNNs) have gained a lot of attention in recent years for their ability to perform well on a range of network science tasks, such as link prediction and node classification. Despite the popularity of graph neural networks and the proven benefits of dynamic network models, there has been little focus on graph neural networks for dynamic networks. To address the challenges resulting from the fact that this research crosses diverse fields as well as to survey dynamic graph neural networks, this work is split into two main parts. First, to address the ambiguity of the dynamic network terminology we establish a foundation of dynamic networks with consistent, detailed terminology and notation. Second, we present a comprehensive survey of dynamic graph neural network models using the proposed terminologyComment: 28 pages, 9 figures, 8 table

Heterogeneous Feature Representation for Digital Twin-Oriented Complex Networked Systems

Building models of Complex Networked Systems (CNS) that can accurately represent reality forms an important research area. To be able to reflect real world systems, the modelling needs to consider not only the intensity of interactions between the entities but also features of all the elements of the system. This study aims to improve the expressive power of node features in Digital Twin-Oriented Complex Networked Systems (DT-CNSs) with heterogeneous feature representation principles. This involves representing features with crisp feature values and fuzzy sets, each describing the objective and the subjective inductions of the nodes' features and feature differences. Our empirical analysis builds DT-CNSs to recreate realistic physical contact networks in different countries from real node feature distributions based on various representation principles and an optimised feature preference. We also investigate their respective disaster resilience to an epidemic outbreak starting from the most popular node. The results suggest that the increasing flexibility of feature representation with fuzzy sets improves the expressive power and enables more accurate modelling. In addition, the heterogeneous features influence the network structure and the speed of the epidemic outbreak, requiring various mitigation policies targeted at different people

A Network Science perspective of Graph Convolutional Networks: A survey

The mining and exploitation of graph structural information have been the focal points in the study of complex networks. Traditional structural measures in Network Science focus on the analysis and modelling of complex networks from the perspective of network structure, such as the centrality measures, the clustering coefficient, and motifs and graphlets, and they have become basic tools for studying and understanding graphs. In comparison, graph neural networks, especially graph convolutional networks (GCNs), are particularly effective at integrating node features into graph structures via neighbourhood aggregation and message passing, and have been shown to significantly improve the performances in a variety of learning tasks. These two classes of methods are, however, typically treated separately with limited references to each other. In this work, aiming to establish relationships between them, we provide a network science perspective of GCNs. Our novel taxonomy classifies GCNs from three structural information angles, i.e., the layer-wise message aggregation scope, the message content, and the overall learning scope. Moreover, as a prerequisite for reviewing GCNs via a network science perspective, we also summarise traditional structural measures and propose a new taxonomy for them. Finally and most importantly, we draw connections between traditional structural approaches and graph convolutional networks, and discuss potential directions for future research

Digital Twin-Oriented Complex Networked Systems based on Heterogeneous node features and interaction rules

This study proposes an extendable modelling framework for Digital Twin-Oriented Complex Networked Systems (DT-CNSs) with a goal of generating networks that faithfully represent real systems. Modelling process focuses on (i) features of nodes and (ii) interaction rules for creating connections that are built based on individual node's preferences. We conduct experiments on simulation-based DT-CNSs that incorporate various features and rules about network growth and different transmissibilities related to an epidemic spread on these networks. We present a case study on disaster resilience of social networks given an epidemic outbreak by investigating the infection occurrence within specific time and social distance. The experimental results show how different levels of the structural and dynamics complexities, concerned with feature diversity and flexibility of interaction rules respectively, influence network growth and epidemic spread. The analysis revealed that, to achieve maximum disaster resilience, mitigation policies should be targeted at nodes with preferred features as they have higher infection risks and should be the focus of the epidemic control

Changes of memory B- and T-cell subsets in lupus nephritis patients

Introduction. Renal involvement in systemic lupus erythematosus (SLE) is associated with production of antibodies to double stranded DNA, deposition of immune complexes and organ damage. These processes have been linked with abnormalities in B- and T-cell memory compartments. The aim of the study was to analyze subsets of peripheral memory B-cells and T-cells in lupus nephritis (LN) patients. Material and methods. We used multicolor flow cytometry to analyze major memory subsets of peripheral blood B-cells (defined by CD27, IgD and CD21) and T-cells (CD45RA, CD45RO, CCR7) in 32 patients with active or inactive LN, and 23 control subjects. Results. Lupus nephritis patients were characterized by increased percentage of immature/early-transitional B-cells (CD27-IgD+CD21-), higher frequency of activated switched memory (SM, CD27+IgD-CD21-) and exhausted memory B-cells (CD27-IgD-), and decrease in non-switched memory (NSM, CD27+IgD+) B-cells. CD21low subsets (immature and activated B-cells) were particularly expanded in patients with active disease. In both groups of LN patients we observed decline in the absolute count of NSM B-cells. It was paralleled by lymphopenia in naïve CD4+ T-cell compartment and increase in the frequency of effector memory T-cells, and these changes were more pronounced in active LN. Conclusions. B-cell memory compartment in LN is deficient in NSM cells and during active disease it is further skewed towards SM and exhausted memory phenotypes, most likely as a cause of chronic antigenic stimulation. Parallel changes in T-helper cell subsets suggest a similar mechanism of SLE-related lymphopenia for both B-cell and T-cell compartment

AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation

Automated machine learning pipeline (ML) composition and optimisation aim at automating the process of finding the most promising ML pipelines within allocated resources (i.e., time, CPU and memory). Existing methods, such as Bayesian-based and genetic-based optimisation, which are implemented in Auto-Weka, Auto-sklearn and TPOT, evaluate pipelines by executing them. Therefore, the pipeline composition and optimisation of these methods frequently require a tremendous amount of time that prevents them from exploring complex pipelines to find better predictive models. To further explore this research challenge, we have conducted experiments showing that many of the generated pipelines are invalid in the first place, and attempting to execute them is a waste of time and resources. To address this issue, we propose a novel method to evaluate the validity of ML pipelines, without their execution, using a surrogate model (AVATAR). The AVATAR generates a knowledge base by automatically learning the capabilities and effects of ML algorithms on datasets' characteristics. This knowledge base is used for a simplified mapping from an original ML pipeline to a surrogate model which is a Petri net based pipeline. Instead of executing the original ML pipeline to evaluate its validity, the AVATAR evaluates its surrogate model constructed by capabilities and effects of the ML pipeline components and input/output simplified mappings. Evaluating this surrogate model is less resource-intensive than the execution of the original pipeline. As a result, the AVATAR enables the pipeline composition and optimisation methods to evaluate more pipelines by quickly rejecting invalid pipelines. We integrate the AVATAR into the sequential model-based algorithm configuration (SMAC). Our experiments show that when SMAC employs AVATAR, it finds better solutions than on its own.Comment: arXiv admin note: substantial text overlap with arXiv:2001.1115

NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

Neural architecture search (NAS) has attracted a lot of attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms to some extent incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date NAS algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally cost friendly environment. All codes are publicly available at: https://xuanyidong.com/assets/projects/NATS-Bench.Comment: Accepted to IEEE TPAMI 2021, an extended version of NAS-Bench-201 (ICLR 2020) [arXiv:2001.00326

The Technological Emergence of AutoML: A Survey of Performant Software and Applications in the Context of Industry

With most technical fields, there exists a delay between fundamental academic research and practical industrial uptake. Whilst some sciences have robust and well-established processes for commercialisation, such as the pharmaceutical practice of regimented drug trials, other fields face transitory periods in which fundamental academic advancements diffuse gradually into the space of commerce and industry. For the still relatively young field of Automated/Autonomous Machine Learning (AutoML/AutonoML), that transitory period is under way, spurred on by a burgeoning interest from broader society. Yet, to date, little research has been undertaken to assess the current state of this dissemination and its uptake. Thus, this review makes two primary contributions to knowledge around this topic. Firstly, it provides the most up-to-date and comprehensive survey of existing AutoML tools, both open-source and commercial. Secondly, it motivates and outlines a framework for assessing whether an AutoML solution designed for real-world application is 'performant'; this framework extends beyond the limitations of typical academic criteria, considering a variety of stakeholder needs and the human-computer interactions required to service them. Thus, additionally supported by an extensive assessment and comparison of academic and commercial case-studies, this review evaluates mainstream engagement with AutoML in the early 2020s, identifying obstacles and opportunities for accelerating future uptake

Imbalance between Th17 and regulatory T-cells in systemic lupus erythematosus

Impaired function of regulatory T-cells (Treg) leads to a failure in immune tolerance and triggers autoimmunity. We analyzed whether the deficiency in Treg in systemic lupus erythematosus (SLE) is accompanied by an increase in effector T-cell responses. We studied the frequencies of IL-17A (Th17) and IFNg (Th1) producing CD4+ T-cells by flow cytometric detection of intracellular cytokines in PMA/ionomycin stimulated blood lymphocytes from seven patients with active SLE, eight with SLE in remission, and 11 healthy controls. Circulating Treg were evaluated as CD4+CD25+ lymphocytes expressing FoxP3. There was no difference in the percentage of Treg cells between the groups, but their absolute counts were decreased in active SLE (5 [1&#8211;7] cells/&mu;L) compared to inactive SLE (11 [6&#8211;15]; p = 0.05) and healthy controls (16 [10&#8211;20]; p < 0.01). Both the frequency and numbers of Th1 cells were decreased in SLE compared to controls. No difference was observed in the number of Th17 cells, which resulted in a decreased Th1/Th17 ratio. In parallel, a higher Treg/Th17 ratio in healthy controls (2.2 [1.8&#8211;3.6]) compared to active SLE (1.1 [1.0&#8211;2.1]; p < 0.05) was observed. There was a correlation between the number of Treg cells and disease activity status (SLEDAI, r = &#8211;0.59). SLE patients in the active phase of the disease are characterized by a deficiency in Treg cells and decreased Treg/Th17 ratio. This suggests that the imbalance between major T-cells subsets might be responsible for an increased proinflammatory response in the exacerbation of SLE. (Folia Histochemica et Cytobiologica 2011; Vol. 49, No. 4, pp. 646&#8211;653

On accuracy of PDF divergence estimators and their applicability to representative data sampling

Generalisation error estimation is an important issue in machine learning. Cross-validation traditionally used for this purpose requires building multiple models and repeating the whole procedure many times in order to produce reliable error estimates. It is however possible to accurately estimate the error using only a single model, if the training and test data are chosen appropriately. This paper investigates the possibility of using various probability density function divergence measures for the purpose of representative data sampling. As it turned out, the first difficulty one needs to deal with is estimation of the divergence itself. In contrast to other publications on this subject, the experimental results provided in this study show that in many cases it is not possible unless samples consisting of thousands of instances are used. Exhaustive experiments on the divergence guided representative data sampling have been performed using 26 publicly available benchmark datasets and 70 PDF divergence estimators, and their results have been analysed and discussed