Computing graph neural networks: A survey from algorithms to accelerators

Graph Neural Networks (GNNs) have exploded onto the machine learning scene in recent years owing to their capability to model and learn from graph-structured data. Such an ability has strong implications in a wide variety of fields whose data are inherently relational, for which conventional neural networks do not perform well. Indeed, as recent reviews can attest, research in the area of GNNs has grown rapidly and has lead to the development of a variety of GNN algorithm variants as well as to the exploration of ground-breaking applications in chemistry, neurology, electronics, or communication networks, among others. At the current stage research, however, the efficient processing of GNNs is still an open challenge for several reasons. Besides of their novelty, GNNs are hard to compute due to their dependence on the input graph, their combination of dense and very sparse operations, or the need to scale to huge graphs in some applications. In this context, this article aims to make two main contributions. On the one hand, a review of the field of GNNs is presented from the perspective of computing. This includes a brief tutorial on the GNN fundamentals, an overview of the evolution of the field in the last decade, and a summary of operations carried out in the multiple phases of different GNN algorithm variants. On the other hand, an in-depth analysis of current software and hardware acceleration schemes is provided, from which a hardware-software, graph-aware, and communication-centric vision for GNN accelerators is distilled.This work is possible thanks to funding from the European Union’s Horizon 2020 research and innovation programme under Grant No. 863337 (WiPLASH project) and the Spanish Ministry of Economy and Competitiveness under contract TEC2017-90034-C2-1-R (ALLIANCE project) that receives funding from FEDER.Peer ReviewedPostprint (published version

Graphene-based wireless agile interconnects for massive heterogeneous multi-chip processors

The main design principles in computer architecture have recently shifted from a monolithic scaling-driven approach to the development of heterogeneous architectures that tightly co-integrate multiple specialized processor and memory chiplets. In such data-hungry multi-chip architectures, current Networks-in-Package (NiPs) may not be enough to cater to their heterogeneous and fast-changing communication demands. This position article makes the case for wireless in-package networking as the enabler of efficient and versatile wired-wireless interconnect fabrics for massive heterogeneous processors. To that end, the use of graphene-based antennas and transceivers with unique frequency-beam reconfigurability in the terahertz band is proposed. The feasibility of such a wireless vision and the main research challenges toward its realization are analyzed from the technological, communications, and computer architecture perspectives.This publication is part of the Spanish I+D+i project TRAINER-A (ref. PID2020-118011GB-C21), funded by MCIN/AEI/10.13039/501100011033. This work has been also supported by the European Commission under H2020 grants WiPLASH (GA 863337), 2D-EPL (GA 952792), and Graphene Flagship (GA 881603); the FLAGERA framework under grant TUGRACO (HA 3022/9-1, LE 2440/3-1), the European Research Council under grants WINC (GA 101042080), COMPUSAPIEN (GA 725657), and PROJESTOR (GA 682675), the German Ministry of Education and Research under grant GIMMIK (03XP0210) and the and the German Research Foundation under grant HIPEDI (WA 4139/1-1).Peer ReviewedArticle signat per 21 autors/es: Sergi Abadal, Robert Guirado, Hamidreza Taghvaee, and Akshay Jain are with the Universitat Politècnica de Catalunya, Spain; Elana Pereira de Santana and Peter Haring Bolívar are with the University of Siegen, Germany; Mohamed Saeed, Renato Negra, Kun-Ta Wang, and Max C. Lemme are with RWTH Aachen University, Germany. Zhenxing Wang, Kun-Ta Wang, and Max C. Lemme are also with AMO GmbH, Germany; Joshua Klein, Marina Zapater, Alexandre Levisse, and David Atienza are with the Swiss Federal Institute of Technology, Switzerland. Marina Zapater is also with the University of Applied Sciences and Arts Western Switzerland; Davide Rossi and Francesco Conti are with the University of Bologna,Italy; Martino Dazzi, Geethan Karunaratne, Irem Boybat, and Abu Sebastian are with IBM Research Europe, SwitzerlandPostprint (author's final draft

Understanding the design-space of sparse/dense multiphase GNN dataflows on spatial accelerators

Graph Neural Networks (GNNs) have garnered a lot of recent interest because of their success in learning representations from graph-structured data across several critical applications in cloud and HPC. Owing to their unique compute and memory characteristics that come from an interplay between dense and sparse phases of computations, the emergence of recon-figurable dataflow (aka spatial) accelerators offers promise for acceleration by mapping optimized dataflows (i.e., computation order and parallelism) for both phases. The goal of this work is to characterize and understand the design-space of dataflow choices for running GNNs on spatial accelerators in order for mappers or design-space exploration tools to optimize the dataflow based on the workload. Specifically, we propose a taxonomy to describe all possible choices for mapping the dense and sparse phases of GNN inference, spatially and temporally over a spatial accelerator, capturing both the intra-phase dataflow and the inter-phase (pipelined) dataflow. Using this taxonomy, we do deep-dives into the cost and benefits of several dataflows and perform case studies on implications of hardware parameters for dataflows and value of flexibility to support pipelined execution.Parts of this work were supported through a fellowship by NEC Laboratories Europe, Project grant PID2020-112827GB-I00 funded by MCIN/AEI/ 10.13039/501100011033, RTI2018-098156-B-C53 (MCIU/AEI/FEDER,UE) and grant 20749/FPI/18 from Fundación Séneca.Peer ReviewedPostprint (author's final draft

Dataflow-architecture co-design for 2.5D DNN accelerators using wireless network-on-package

Deep neural network (DNN) models continue to grow in size and complexity, demanding higher computational power to enable real-time inference. To efficiently deliver such computational demands, hardware accelerators are being developed and deployed across scales. This naturally requires an efficient scale-out mechanism for increasing compute density as required by the application. 2.5D integration over interposer has emerged as a promising solution, but as we show in this work, the limited interposer bandwidth and multiple hops in the Network-on-Package (NoP) can diminish the benefits of the approach. To cope with this challenge, we propose WIENNA, a wireless NoP-based 2.5D DNN accelerator. In WIENNA, the wireless NoP connects an array of DNN accelerator chiplets to the global buffer chiplet, providing high-bandwidth multicasting capabilities. Here, we also identify the dataflow style that most efficienty exploits the wireless NoP's high-bandwidth multicasting capability on each layer. With modest area and power overheads, WIENNA achieves 2.2X-5.1X higher throughput and 38.2% lower energy than an interposer-based NoP design.This work was supported by the European Commission under grant 863337 and NSF under Award OAC-1909900.Peer ReviewedPostprint (author's final draft

Architecting more than Moore: wireless plasticity for massive heterogeneous computer architectures (WiPLASH)

This paper presents the research directions pursued by the WiPLASH European project, pioneering on-chip wireless communications as a disruptive enabler towards next-generation computing systems for artificial intelligence (AI). We illustrate the holistic approach driving our research efforts, which encompass expertises and abstraction levels ranging from physical design of embedded graphene antennas to system-level evaluation of wirelessly-communicating heterogeneous systems.This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 863337 (WiPLASH).Peer ReviewedArticle signat per 22 autors/es: - Joshua Klein, Alexandre Levisse, Giovanni Ansaloni, David Atienza (EPFL, Lausanne, Switzerland) - Marina Zapater (HEIG-VD/HES-SO Yverdon-les-Bains, Switzerland / EPFL, Lausanne, Switzerland) - Martino Dazzi, Geethan Karunaratne, Irem Boybat, Abu Sebastian (IBM Research Europe, Zurich, Switzerland) - Davide Rossi, Francesco Conti (Università di Bologna, Bologna, Italy) - Elana Pereira de Santana, Peter Haring Bolívar (University of Siegen, Siegen, Germany) - Mohamed Saeed, Renato Negra (RWTH, Aachen, Germany) - Zhenxing Wang, Kun-Ta Wang (AMO GmbH, Aachen, Germany) - Max C. Lemme (RWTH, Aachen, Germany / AMO GmbH, Aachen, Germany) - Akshay Jain, Robert Guirado, Hamidreza Taghvaee, Sergi Abadal (Universitat Politècnica de Catalunya, Barcelona, Spain)Postprint (author's final draft