You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets

Recent works have impressively demonstrated that there exists a subnetwork in randomly initialized convolutional neural networks (CNNs) that can match the performance of the fully trained dense networks at initialization, without any optimization of the weights of the network (i.e., untrained networks). However, the presence of such untrained subnetworks in graph neural networks (GNNs) still remains mysterious. In this paper we carry out the first-of-its-kind exploration of discovering matching untrained GNNs. With sparsity as the core tool, we can find untrained sparse subnetworks at the initialization, that can match the performance of fully trained dense GNNs. Besides this already encouraging finding of comparable performance, we show that the found untrained subnetworks can substantially mitigate the GNN over-smoothing problem, hence becoming a powerful tool to enable deeper GNNs without bells and whistles. We also observe that such sparse untrained subnetworks have appealing performance in out-of-distribution detection and robustness of input perturbations. We evaluate our method across widely-used GNN architectures on various popular datasets including the Open Graph Benchmark (OGB)

Artificial neural networks training acceleration through network science strategies

The development of deep learning has led to a dramatic increase in the number of applications of artificial intelligence. However, the training of deeper neural networks for stable and accurate models translates into artificial neural networks (ANNs) that become unmanageable as the number of features increases. This work extends our earlier study where we explored the acceleration effects obtained by enforcing, in turn, scale freeness, small worldness, and sparsity during the ANN training process. The efficiency of that approach was confirmed by recent studies (conducted independently) where a million-node ANN was trained on non-specialized laptops. Encouraged by those results, our study is now focused on some tunable parameters, to pursue a further acceleration effect. We show that, although optimal parameter tuning is unfeasible, due to the high non-linearity of ANN problems, we can actually come up with a set of useful guidelines that lead to speed-ups in practical cases. We find that significant reductions in execution time can generally be achieved by setting the revised fraction parameter (ζ) to relatively low values

Worldwide trends in underweight and obesity from 1990 to 2022: a pooled analysis of 3663 population-representative studies with 222 million children, adolescents, and adults

Background Underweight and obesity are associated with adverse health outcomes throughout the life course. We estimated the individual and combined prevalence of underweight or thinness and obesity, and their changes, from 1990 to 2022 for adults and school-aged children and adolescents in 200 countries and territories. Methods We used data from 3663 population-based studies with 222 million participants that measured height and weight in representative samples of the general population. We used a Bayesian hierarchical model to estimate trends in the prevalence of different BMI categories, separately for adults (age ≥20 years) and school-aged children and adolescents (age 5–19 years), from 1990 to 2022 for 200 countries and territories. For adults, we report the individual and combined prevalence of underweight (BMI 2 SD above the median). Findings From 1990 to 2022, the combined prevalence of underweight and obesity in adults decreased in 11 countries (6%) for women and 17 (9%) for men with a posterior probability of at least 0·80 that the observed changes were true decreases. The combined prevalence increased in 162 countries (81%) for women and 140 countries (70%) for men with a posterior probability of at least 0·80. In 2022, the combined prevalence of underweight and obesity was highest in island nations in the Caribbean and Polynesia and Micronesia, and countries in the Middle East and north Africa. Obesity prevalence was higher than underweight with posterior probability of at least 0·80 in 177 countries (89%) for women and 145 (73%) for men in 2022, whereas the converse was true in 16 countries (8%) for women, and 39 (20%) for men. From 1990 to 2022, the combined prevalence of thinness and obesity decreased among girls in five countries (3%) and among boys in 15 countries (8%) with a posterior probability of at least 0·80, and increased among girls in 140 countries (70%) and boys in 137 countries (69%) with a posterior probability of at least 0·80. The countries with highest combined prevalence of thinness and obesity in school-aged children and adolescents in 2022 were in Polynesia and Micronesia and the Caribbean for both sexes, and Chile and Qatar for boys. Combined prevalence was also high in some countries in south Asia, such as India and Pakistan, where thinness remained prevalent despite having declined. In 2022, obesity in school-aged children and adolescents was more prevalent than thinness with a posterior probability of at least 0·80 among girls in 133 countries (67%) and boys in 125 countries (63%), whereas the converse was true in 35 countries (18%) and 42 countries (21%), respectively. In almost all countries for both adults and school-aged children and adolescents, the increases in double burden were driven by increases in obesity, and decreases in double burden by declining https://researchonline.ljmu.ac.uk/images/research_banner_face_lab_290.jpgunderweight or thinness. Interpretation The combined burden of underweight and obesity has increased in most countries, driven by an increase in obesity, while underweight and thinness remain prevalent in south Asia and parts of Africa. A healthy nutrition transition that enhances access to nutritious foods is needed to address the remaining burden of underweight while curbing and reversing the increase in obesity

One-shot learning using Mixture of Variational Autoencoders:A generalization learning approach

\u3cp\u3eDeep learning, even if it is very successful nowadays, traditionally needs very large amounts of labeled data to perform excellent on the classification task. In an attempt to solve this problem, the one-shot learning paradigm, which makes use of just one labeled sample per class and prior knowledge, becomes increasingly important. In this paper, we propose a new one-shot learning method, dubbed MoVAE (Mixture of Variational AutoEncoders), to perform classification. Complementary to prior studies, MoVAE represents a shift of paradigm in comparison with the usual one-shot learning methods, as it does not use any prior knowledge. Instead, it starts from zero knowledge and one labeled sample per class. Afterward, by using unlabeled data and the generalization learning concept (in a way, more as humans do), it is capable to gradually improve by itself its performance. Even more, if there are no unlabeled data available MoVAE can still perform well in one-shot learning classification. We demonstrate empirically the efficiency of our proposed approach on three datasets, i.e. the handwritten digits (MNIST), fashion products (Fashion-MNIST), and handwritten characters (Omniglot), showing that MoVAE outperforms state-of-the-art oneshot learning algorithms.\u3c/p\u3

On the synergy of network science and artificial intelligence

Traditionally science is done using the reductionism paradigm. Artificial intelligence does not make an exception and it follows the same strategy. At the same time, network science tries to study complex systems as a whole. This Ph.D. research takes an alternative approach to the reductionism strategy, and tries to advance both fields, i.e. artificial intelligence and network science, by searching for the synergy between them, while not ignoring any other source of inspiration, e.g. neuroscience

Synopsis of the PhD thesis : network computations in artificial intelligence

\u3cp\u3eTraditionally science is done using the reductionism paradigm. Artificial intelligence does not make an exception and it follows the same strategy. At the same time, network science tries to study complex systems as a whole. This synopsis presents my PhD thesis which takes an alternative approach to the reductionism strategy, with the aim to advance both fields, advocating that major breakthroughs can be made when these two are combined. The thesis illustrates this bidirectional relation by: (1) proposing a new method which uses artificial intelligence to improve network science algorithms (i.e. a new centrality metric which computes fully decentralized the nodes and links importance, on the polylogarithmic scale with respect to the number of nodes in the network); and (2) proposing two methods which take inspiration from network science to improve artificial intelligence algorithms (e.g. quadratic acceleration in terms of memory requirements and computational speed of artificial neural network fully connected layers during both, training and inference).\u3c/p\u3

Evolving and understanding sparse deep neural networks using cosine similarity

Training sparse neural networks with adaptive connectivity is an active research topic. Such networks require less storage and have lower computational complexity compared to their dense counterparts. The Sparse Evolutionary Training (SET) procedure uses weights magnitude to evolve efficiently the topology of a sparse network to fit the dataset, while enabling it to have quadratically less parameters than its dense counterpart. To this end, we propose a novel approach that evolves a sparse network topology based on the behavior of neurons in the network. More exactly, the cosine similarities between the activations of any two neurons are used to determine which connections are added to or removed from the network. By integrating our approach within the SET procedure, we propose 5 new algorithms to train sparse neural networks. We argue that our approach has low additional computational complexity and we draw a parallel to Hebbian learning. Experiments are performed on 8 datasets taken from various domains to demonstrate the general applicability of our approach. Even without optimizing hyperparameters for specific datasets, the experiments show that our proposed training algorithms usually outperform SET and state-of-the-art dense neural network techniques. The last but not the least, we show that the evolved connectivity patterns of the input neurons reflect their impact on the classification task

Big IoT data mining for real-time energy disaggregation in buildings

In the smart grid context, the identification and prediction of building energy flexibility is a challenging open question, thus paving the way for new optimized behaviors from the demand side. At the same time, the latest smart meters developments allow us to monitor in real-time the power consumption level of the home appliances, aiming at a very accurate energy disaggregation. However, due to practical constraints is infeasible in the near future to attach smart meter devices on all home appliances, which is the problem addressed herein. We propose a hybrid approach, which combines sparse smart meters with machine learning methods. Using a subset of buildings equipped with subset of smart meters we can create a database on which we train two deep learning models, i.e. Factored Four-Way Conditional Restricted Boltzmann Machines (FFW-CRBMs) and Disjunctive FFW-CRBM. We show how our method may be used to accurately predict and identify the energy flexibility of buildings unequipped with smart meters, starting from their aggregated energy values. The proposed approach was validated on a real database, namely the Reference Energy Disaggregation Dataset. The results show that for the flexibility prediction problem solved here, Disjunctive FFW-CRBM outperforms the FFW-CRBMs approach, where for classification task their capabilities are comparable