Investigation of How Neural Networks Learn From the Experiences of Peers Through Periodic Weight Averaging

We investigate a method, weighted average model fusion, that enables neural networks to learn from the experiences of other networks, as well as from their own experiences. This method is inspired by the the Social natural of humans, which has been shown to be one of the biggest factors in the development of our cognitive abilities. Modern machine learning has focuses predominantly on learning from direct training, and has largely ignored learning through Social engagement with peers, neural networks will the same topology. In order to explore learning through engagement with peers, we have created a way for neural networks to teach each other. Our method allows neural networks to exchange knowledge by combining their weights. It calculates a pairwise weighted average of the weights of two neural networks, and then replaces the existing weight with the new value. We find that weighted average model fusion successfully enables neural networks to learn from the experiences of their peers and combine it with the knowledge that is gained from its own individual experiences. Additionally, we explore the effects that several meta-parameters have on model fusion to provide deeper insights into how the behaves in a variety of scenarios

Proceedings of the ECCS 2005 satellite workshop: embracing complexity in design - Paris 17 November 2005

Embracing complexity in design is one of the critical issues and challenges of the 21st century. As the realization grows that design activities and artefacts display properties associated with complex adaptive systems, so grows the need to use complexity concepts and methods to understand these properties and inform the design of better artifacts. It is a great challenge because complexity science represents an epistemological and methodological swift that promises a holistic approach in the understanding and operational support of design. But design is also a major contributor in complexity research. Design science is concerned with problems that are fundamental in the sciences in general and complexity sciences in particular. For instance, design has been perceived and studied as a ubiquitous activity inherent in every human activity, as the art of generating hypotheses, as a type of experiment, or as a creative co-evolutionary process. Design science and its established approaches and practices can be a great source for advancement and innovation in complexity science. These proceedings are the result of a workshop organized as part of the activities of a UK government AHRB/EPSRC funded research cluster called Embracing Complexity in Design (www.complexityanddesign.net) and the European Conference in Complex Systems (complexsystems.lri.fr). Embracing complexity in design is one of the critical issues and challenges of the 21st century. As the realization grows that design activities and artefacts display properties associated with complex adaptive systems, so grows the need to use complexity concepts and methods to understand these properties and inform the design of better artifacts. It is a great challenge because complexity science represents an epistemological and methodological swift that promises a holistic approach in the understanding and operational support of design. But design is also a major contributor in complexity research. Design science is concerned with problems that are fundamental in the sciences in general and complexity sciences in particular. For instance, design has been perceived and studied as a ubiquitous activity inherent in every human activity, as the art of generating hypotheses, as a type of experiment, or as a creative co-evolutionary process. Design science and its established approaches and practices can be a great source for advancement and innovation in complexity science. These proceedings are the result of a workshop organized as part of the activities of a UK government AHRB/EPSRC funded research cluster called Embracing Complexity in Design (www.complexityanddesign.net) and the European Conference in Complex Systems (complexsystems.lri.fr)

Model Fusion via Optimal Transport

Combining different models is a widely used paradigm in machine learning applications. While the most common approach is to form an ensemble of models and average their individual predictions, this approach is often rendered infeasible by given resource constraints in terms of memory and computation, which grow linearly with the number of models. We present a layer-wise model fusion algorithm for neural networks that utilizes optimal transport to (soft-) align neurons across the models before averaging their associated parameters. We show that this can successfully yield "one-shot" knowledge transfer (i.e, without requiring any retraining) between neural networks trained on heterogeneous non-i.i.d. data. In both i.i.d. and non-i.i.d. settings , we illustrate that our approach significantly outperforms vanilla averaging, as well as how it can serve as an efficient replacement for the ensemble with moderate fine-tuning, for standard convolutional networks (like VGG11), residual networks (like ResNet18), and multi-layer perceptrons on CIFAR10, CIFAR100, and MNIST. Finally, our approach also provides a principled way to combine the parameters of neural networks with different widths, and we explore its application for model compression. The code is available at the following link, https://github.com/sidak/otfusion.Comment: NeurIPS 2020 conference proceedings (early version featured in the Optimal Transport & Machine Learning workshop, NeurIPS 2019

Efficient Asynchronous GCN Training on a GPU Cluster

A common assumption in traditional synchronous parallel training of Graph Convolutional Networks (GCNs) using multiple GPUs is that load is perfectly balanced among all GPUs. However, this assumption does not hold in a real-world scenario where there can be imbalances in workloads among GPUs for various reasons. In a synchronous parallel implementation, a straggler in the system can limit the overall speed-up of parallel training. To address these issues, this research investigates approaches for asynchronous decentralized parallel training for GCNs. The techniques investigated are based on graph clustering and gossiping. The research specifically adapts the approach of Cluster-GCN, which uses graph partitioning for SGD-based training, and combines with a novel gossip algorithm specifically designed for a GPU cluster to periodically exchange gradients among randomly chosen partners. In addition, it incorporates a work-pool mechanism for load balancing among GPUs. The gossip algorithm is proven to be deadlock free. The implementation is done on a GPU cluster with 8 Tesla V100 GPUs per compute node, and PyTorch and DGL as the software platforms. Experiments are conducted for different benchmark datasets. The results demonstrate superior performance, at the compromise of minor accuracy loss in some runs, as compared to traditional synchronous training which uses all-reduce to synchronously accumulate parallel training results

Adolescent Development in Context: Social, Psychological, and Neurological Foundations

This project was funded by KU Libraries’ Parent’s Campaign with support from the David Shulenburger Office of Scholarly Communication & Copyright and the Open Educational Resources Working Group in the University of Kansas Libraries.Increasingly, there is a tendency to characterize the teenage years as a time of general moral degeneration and deviance. This is unfortunate because the teenage years represent a key developmental period of the typical human lifespan, and from an evolutionary point of view, the actual characteristics that define adolescence represent critical learning opportunities. The increased sensitivity to social influences, identity formation, and social-emotional skills are just a few of such opportunities that require appropriate environments and contexts for optimal, healthy outcomes. Research in the field of adolescent development has not been immune to the negative stereotypes surrounding adolescence, and it is common to see researchers, either implicitly or explicitly, refer to adolescence as a high-risk, anomalous developmental stage that must be controlled, managed, or simply endured until adult-level abilities emerge spontaneously as a result of having survived an intrinsically tumultuous developmental time. More enlightened views of adolescence recognize that all biological adaptations have a cause and a purpose, and that the purpose of adolescence can be discerned from understanding the complex evolutionary history of humans as a group-based, family-based, highly social, sometimes competitive, abstract-thinking species. Understanding the biological foundations of adolescence is meaningless if one does not also consider how biology and environment interact. In humans, these interactions are highly complex and involve not only immediate physical realities, but also social, cultural, and historical realities that create complex contexts and webs of interactions. Therefore, this textbook seeks to reconcile the biological and neurological foundations of human development with the psychological and sociological mechanisms that formed and continue to influence human developmental trajectories. To this end, we have divided the textbook into three main sections. The first, Foundations of Adolescent Development, introduces the historical science of studying adolescence and the biological foundations of puberty. The second section, Contexts of Adolescent Development, considers the primary contextual factors that influence developmental outcomes during adolescence. These include work and employment, peers, in-school and out-of-school contexts, leisure time, and the family. The final section, Milestones of Adolescent Development, addresses the primary psychological milestones that represent healthy adjustment to adult roles and responsibilities in society. The domains of these milestones include cognition and decision-making; identity, meaning, and purpose, moral development, and sexuality. From an educational point of view, the objective of this textbook is to provide a resource that is capable of fostering advanced conceptual change and learning in the field of adolescent development in order to go beyond stereotypical portrayals of adolescence as a pathological condition. Organized in a manner designed to scaffold increasingly complex ideas, the textbook redefines adolescence a sensitive period of development characterized by phylogenetically derived experience-expectant states and complex interactions of biological, psychological, and social factors. The textbook draws from the latest advances in neuroscience and psychology to construct a practical framework for use in a wide range of academic and professional contexts, and it presents historical as well as contemporary research to accomplish a radical redefining of an often misunderstood and maligned developmental period

Riding the (brain) waves! Using neural oscillations to inform bilingualism research

The study of the brains’ oscillatory activity has been a standard technique to gain insights into human neurocognition for a relatively long time. However, as a complementary analysis to ERPs, only very recently has it been utilized to study bilingualism and its neural underpinnings. Here, we provide a theoretical and methodological starter for scientists in the (psycho)linguistics and neurocognition of bilingualism field(s) to understand the bases and applications of this analytical tool. Towards this goal, we provide a description of the characteristics of the human neural (and its oscillatory) signal, followed by an in-depth description of various types of EEG oscillatory analyses, supplemented by figures and relevant examples. We then utilize the scant, yet emergent, literature on neural oscillations and bilingualism to highlight the potential of how analyzing neural oscillations can advance our understanding of the (psycho)linguistic and neurocognitive understanding of bilingualism

Quality of service routing for real-time traffic

Imperial Users onl

Digital Food Marketing to Children and Adolescents: Problematic Practices and Policy Interventions

Examines trends in digital marketing to youth that uses "immersive" techniques, social media, behavioral profiling, location targeting and mobile marketing, and neuroscience methods. Recommends principles for regulating inappropriate advertising to youth