Improving scalability of large-scale distributed Spiking Neural Network simulations on High Performance Computing systems using novel architecture-aware streaming hypergraph partitioning

After theory and experimentation, modelling and simulation is regarded as the third pillar of science, helping scientists to further their understanding of a complex system. In recent years there has been a growing scientific focus on computational neuroscience as a means to understand the brain and its functions, with large international projects (Human Brain Project, Brain Activity Map, MindScope and \textit{China Brain Project}) aiming to further our knowledge of high level cognitive functions. They are a testament to the enormous interest, difficulty and importance of solving the mysteries of the brain. Spiking Neural Network (SNN) simulations are widely used in the domain to facilitate experimentation. Scaling SNN simulations to large networks usually results in more-than-linear increase in computational complexity. The computing resources required at the brain scale simulation far surpass the capabilities of personal computers today. If those demands are to be met, distributed computation models need to be adopted, since there is a slow down of improvements in individual processors speed due to physical limitations on heat dissipation. This is a significant change that requires careful management of the workload in many levels: partition of work, communication and workload balancing, efficient inter-process communication and efficient use of available memory. If large scale neuronal network models are to be run successfully, simulators must consider these, and offer a viable solution to the challenges they pose. Large scale SNN simulations evidence most of the issues of general HPC systems evident in large distributed computation. Commonly used distribution of workload algorithms (round robin, random and manual allocation) do not take into consideration connectivity locality, which is natural in biological networks, which can lead to increased communication requirements when distributing the simulation in multiple computing nodes. State-of-the-art SNN simulations use dense communication collectives to distribute spike data. The common method of point to point communication in distributed computation is through dense patterns. Sparse communication collectives have been suggested to incur in lower overheads when the application's pattern of communication is sparse. In this work we characterise the bottlenecks on communication-bound SNN simulations and identify communication balance and sparsity as the main contributors to scalability. We propose hypergraph partitioning to distribute neurons along computing nodes to minimise communication (increasing sparsity). A hypergraph is a generalisation of graphs, where a (hyper)edge can link 2 or more vertices at once. Coupled with a novel use of sparse-aware communication collective, computational efficiency increases by up to 40.8 percent points and simulation time reduces by up to 73\%, compared to the common round-robin allocation in neuronal simulators. HPC systems have, by design, highly hierarchical communication network links, with qualitative differences in communication speed and latency between computing nodes. This can create a mismatch between the distributed simulation communication patterns and the physical capabilities of the hardware. If large distributed simulations are to take full advantage of these systems, the communication properties of the HPC need to be taken into consideration when allocating workload to route frequent, heavy communication through fast network links. Strategies that consider the heterogeneous physical communication capabilities are called architecture-aware. After demonstrating that hypergraph partitioning leads to more efficient workload allocation in SNN simulations, this thesis proposes a novel sequential hypergraph partitioning algorithm that incorporates network bandwidth via profiling. This leads to a significant reduction in execution time (up to 14x speedup in synthetic benchmark simulations compared to architecture-agnostic partitioners). The motivating context of this work is large scale brain simulations, however in the era of social media, large graphs and hypergraphs are increasingly relevant in many other scientific applications. A common feature of such graphs is that they are too big for a single machine to cope, both in terms of performance and memory requirements. State-of-the-art multilevel partitioning has been shown to struggle to scale to large graphs in distributed memory, not just because they take a long time to process, but also because they require full knowledge of the graph (not possible in dynamic graphs) and to fit the graph entirely in memory (not possible for very large graphs). To address those limitations we propose a parallel implementation of our architecture-aware streaming hypergraph partitioning algorithm (HyperPRAW) to model distributed applications. Results demonstrate that HyperPRAW produces consistent speedup over previous streaming approaches that only consider hyperedge overlap (up to 5.2x speedup). Compared to multilevel global partitioner in dense hypergraphs (those with high average cardinality), HyperPRAW is able to produce workload allocations that result in speeding up runtime in a synthetic simulation benchmark (up to 4.3x). HyperPRAW has the potential to scale to very large hypergraphs as it only requires local information to make allocation decisions, with an order of magnitude less memory footprint than global partitioners. The combined contributions of this thesis lead to a novel, parallel, scalable, streaming hypergraph partitioning algorithm (HyperPRAW) that can be used to help scale large distributed simulations in HPC systems. HyperPRAW helps tackle three of the main scalability challenges: it produces highly balanced distributed computation and communication, minimising idle time between computing nodes; it reduces the communication overhead by placing frequently communicating simulation elements close to each other (where the communication cost is minimal); and it provides a solution with a reasonable memory footprint that allows tackling larger problems than state-of-the-art alternatives such as global multilevel partitioning

Right Research

"Educational institutions play an instrumental role in social and political change, and are responsible for the environmental and social ethics of their institutional practices. The essays in this volume critically examine scholarly research practices in the age of the Anthropocene, and ask what accountability educators and researchers have in ‘righting’ their relationship to the environment. The volume further calls attention to the geographical, financial, legal and political barriers that might limit scholarly dialogue by excluding researchers from participating in traditional modes of scholarly conversation. As such, Right Research is a bold invitation to the academic community to rigorous self-reflection on what their research looks like, how it is conducted, and how it might be developed so as to increase accessibility and sustainability, and decrease carbon footprint. The volume follows a three-part structure that bridges conceptual and practical concerns: the first section challenges our assumptions about how sustainability is defined, measured and practiced; the second section showcases artist-researchers whose work engages with the impact of humans on our environment; while the third section investigates how academic spaces can model eco-conscious behaviour. This timely volume responds to an increased demand for environmentally sustainable research, and is outstanding not only in its interdisciplinarity, but its embrace of non-traditional formats, spanning academic articles, creative acts, personal reflections and dialogues. Right Research will be a valuable resource for educators and researchers interested in developing and hybridizing their scholarly communication formats in the face of the current climate crisis.

Right Research

"Educational institutions play an instrumental role in social and political change, and are responsible for the environmental and social ethics of their institutional practices. The essays in this volume critically examine scholarly research practices in the age of the Anthropocene, and ask what accountability educators and researchers have in ‘righting’ their relationship to the environment. The volume further calls attention to the geographical, financial, legal and political barriers that might limit scholarly dialogue by excluding researchers from participating in traditional modes of scholarly conversation. As such, Right Research is a bold invitation to the academic community to rigorous self-reflection on what their research looks like, how it is conducted, and how it might be developed so as to increase accessibility and sustainability, and decrease carbon footprint. The volume follows a three-part structure that bridges conceptual and practical concerns: the first section challenges our assumptions about how sustainability is defined, measured and practiced; the second section showcases artist-researchers whose work engages with the impact of humans on our environment; while the third section investigates how academic spaces can model eco-conscious behaviour. This timely volume responds to an increased demand for environmentally sustainable research, and is outstanding not only in its interdisciplinarity, but its embrace of non-traditional formats, spanning academic articles, creative acts, personal reflections and dialogues. Right Research will be a valuable resource for educators and researchers interested in developing and hybridizing their scholarly communication formats in the face of the current climate crisis.

Right Research: Modelling Sustainable Research Practices in the Anthropocene

The year 2020 started with a massive bushfire crisis in south eastern Australia, resulting in disruption to many communities, the loss of lives and businesses, an estimated loss of a billion animals and the dirtiest air on the planet in the cities of Sydney, Newcastle and Canberra. With record-high temperatures and a punishing draught lasting several years, the Australian bush was primed to explode into flames. With lightning strikes in national parks, the spontaneous eruptions of bushfire spread from the north coast to the south and inland towards the alpine regions of New South Wales and Victoria. With the very hot year of 2019 affecting other parts of the planet in 2020, the Antarctic Peninsula reached a record 65 degrees Fahrenheit. The chapter that follows reflects the new progressive politics of climate change that emerged in 2019 with large mass demonstrations taking place in Australia and around the world and examines the critical role of universities in the mitigation of climate catastrophe. The following interventions are variably focused on the concept of ‘Living Labs’ where thinking is developed within a problem-solving ethos. The three contributions here offer ways to think about sustainability with specific reference to waste recovery, environmental awareness in urban settings and the contribution that a ‘repair’ mentality can make to a shared and re-cycled economy. With a clear-eyed recommendation that mitigation of climate change starts locally, the premise of the paper is that people can work with what is available as local solutions to specific problems. The impact of this approach can be essential to people who sense the impending catastrophe and who may have experienced the crisis directly through compromises in their health outcomes, the experience of trauma and the loss of property and livelihoods, though through no fault of their own. The links through the Western Sydney University campus, common ground to the authors to both its small bushland outpost and further to the local community it serves, suggest that the boundaries of the campus are permeable – and that Living Labs are both a means and metaphor for thinking about how the campus opens learning and knowledge creation about sustainability for its students, staff and community constituents