Efficient Intra-Rack Resource Disaggregation for HPC Using Co-Packaged DWDM Photonics

The diversity of workload requirements and increasing hardware heterogeneity in emerging high performance computing (HPC) systems motivate resource disaggregation. Resource disaggregation allows compute and memory resources to be allocated individually as required to each workload. However, it is unclear how to efficiently realize this capability and cost-effectively meet the stringent bandwidth and latency requirements of HPC applications. To that end, we describe how modern photonics can be co-designed with modern HPC racks to implement flexible intra-rack resource disaggregation and fully meet the bit error rate (BER) and high escape bandwidth of all chip types in modern HPC racks. Our photonic-based disaggregated rack provides an average application speedup of 11% (46% maximum) for 25 CPU and 61% for 24 GPU benchmarks compared to a similar system that instead uses modern electronic switches for disaggregation. Using observed resource usage from a production system, we estimate that an iso-performance intra-rack disaggregated HPC system using photonics would require 4x fewer memory modules and 2x fewer NICs than a non-disaggregated baseline.Comment: 15 pages, 12 figures, 4 tables. Published in IEEE Cluster 202

Survey of Photonic and Plasmonic Interconnect Technologies for Intra-Datacenter and High-Performance Computing Communications

Large scale data centers (DC) and high performance computing (HPC) systems require more and more computing power at higher energy efficiency. They are already consuming megawatts of power, and a linear extrapolation of trends reveals that they may eventually lead to unrealistic power consumption scenarios in order to satisfy future requirements (e.g., Exascale computing). Conventional complementary metal oxide semiconductor (CMOS)-based electronic interconnects are not expected to keep up with the envisioned future board-to-board and chip-to-chip (within multi-chip-modules) interconnect requirements because of bandwidth-density and power-consumption limitations. However, low-power and high-speed optics-based interconnects are emerging as alternatives for DC and HPC communications; they offer unique opportunities for continued energy-efficiency and bandwidth-density improvements, although cost is a challenge at the shortest length scales. Plasmonics-based interconnects on the other hand, due to their extremely small size, offer another interesting solution for further scaling operational speed and energy efficiency. At the device-level, CMOS compatibility is also an important issue, since ultimately photonics or plasmonics will have to be co-integrated with electronics. In this paper, we survey the available literature and compare the aforementioned interconnect technologies, with respect to their suitability for high-speed and energy-efficient on-chip and offchip communications. This paper refers to relatively short links with potential applications in the following interconnect distance hierarchy: local group of racks, board to board, module to module, chip to chip, and on chip connections. We compare different interconnect device modules, including low-energy output devices (such as lasers, modulators, and LEDs), photodetectors, passive devices (i.e., waveguides and couplers) and electrical circuitry (such as laserdiode drivers, modulator drivers, transimpedance, and limiting amplifiers). We show that photonic technologies have the potential to meet the requirements for selected HPC and DC applications in a shorter term. We also present that plasmonic interconnect modules could offer ultra-compact active areas, leading to high integration bandwidth densities, and low device capacitances allowing for ultra-high bandwidth operation that would satisfy the application requirements further into the future

VCSEL and Integration Techniques for Wavelength-Multiplexed Optical Interconnects

GaAs-based vertical-cavity surface-emitting lasers (VCSELs) are dominating short-reach optical interconnects (OIs) due to their high modulation speed, low power consumption, circular output beam and low fabrication cost. Such OIs provide the high bandwidth connectivity needed for interconnecting servers and switches in data centers. With the rapidly increasing use of Internet-based applications and services, higher bandwidth connectivity and higher aggregate capacity VCSEL-based OIs are needed. Until now, this has been achieved mostly through an increase of the lane rate by higher speed VCSELs and higher order modulation formats. Furthermore, spatial-division-multiplexing has proven effective for increasing the aggregate capacity. Much higher capacity can be achieved by multiple wavelengths per fiber, known as wavelength-divisionmultiplexing (WDM). Moreover, smaller footprint and higher bandwidth density WDM transceivers can be built using monolithic multi-wavelength VCSEL arrays with densely spaced VCSELs. This requires a VCSEL technology where the wavelength of individual VCSELs can be precisely set in a post-epitaxial growth fabrication process and a photonic integrated circuit (PIC) for multiplexing and fiber coupling. Flip-chip integration over grating couplers (GCs) is considered for interfacing VCSELs with waveguides on the PIC. In this thesis, an intra-cavity phase tuning technique is demonstrated for setting the resonance wavelength of VCSELs in a monolithic array with an accuracy in spacing of <1 nm. Uniform performance over the array is achieved by spectral matching and balancing of mirror reflectances, optical confinement factor and optical gain. Single transverse and polarization mode VCSELs, as required for flip-chip integration over GCs, with a record output power of 6 mW are also demonstrated.Finally, an investigation of angled flip-chip integration of a VCSEL over a GC on a silicon photonic integrated circuit (Si-PIC) is presented. Dependencies of coupling efficiency and optical feedback on flip-chip angle and size of the VCSEL die are studied using numerical FDTD simulations. Moreover, flip-chip integration of a VCSEL over a GC on a Si-PIC is experimentally demonstrated. The insertion loss from the VCSEL at the input GC to a singlemode fiber, multimode fiber or flip-chip integrated photodetector over the output GC was measured and quantified. The latter forms an on-PIC optical link

2011 Strategic roadmap for Australian research infrastructure

The 2011 Roadmap articulates the priority research infrastructure areas of a national scale (capability areas) to further develop Australia’s research capacity and improve innovation and research outcomes over the next five to ten years. The capability areas have been identified through considered analysis of input provided by stakeholders, in conjunction with specialist advice from Expert Working Groups   It is intended the Strategic Framework will provide a high-level policy framework, which will include principles to guide the development of policy advice and the design of programs related to the funding of research infrastructure by the Australian Government. Roadmapping has been identified in the Strategic Framework Discussion Paper as the most appropriate prioritisation mechanism for national, collaborative research infrastructure. The strategic identification of Capability areas through a consultative roadmapping process was also validated in the report of the 2010 NCRIS Evaluation. The 2011 Roadmap is primarily concerned with medium to large-scale research infrastructure. However, any landmark infrastructure (typically involving an investment in excess of $100 million over five years from the Australian Government) requirements identified in this process will be noted. NRIC has also developed a ‘Process to identify and prioritise Australian Government landmark research infrastructure investments’ which is currently under consideration by the government as part of broader deliberations relating to research infrastructure. NRIC will have strategic oversight of the development of the 2011 Roadmap as part of its overall policy view of research infrastructure

LLM: Realizing Low-Latency Memory by Exploiting Embedded Silicon Photonics for Irregular Workloads

As emerging workloads exhibit irregular memory access patterns with poor data reuse and locality, they would benefit from a DRAM that achieves low latency without sacrificing bandwidth and energy efficiency. We propose LLM (Low Latency Memory), a codesign of the DRAM microarchitecture, the memory controller and the LLC/DRAM interconnect by leveraging embedded silicon photonics in 2.5D/3D integrated system on chip. LLM relies on Wavelength Division Multiplexing (WDM)-based photonic interconnects to reduce the contention throughout the memory subsystem. LLM also increases the bank-level parallelism, eliminates bus conflicts by using dedicated optical data paths, and reduces the access energy per bit with shorter global bitlines and smaller row buffers. We evaluate the design space of LLM for a variety of synthetic benchmarks and representative graph workloads on a full-system simulator (gem5). LLM exhibits low memory access latency for traffics with both regular and irregular access patterns. For irregular traffic, LLM achieves high bandwidth utilization (over 80% peak throughput compared to 20% of HBM2.0). For real workloads, LLM achieves 3 × and 1.8 × lower execution time compared to HBM2.0 and a state-of-the-art memory system with high memory level parallelism, respectively. This study also demonstrates that by reducing queuing on the data path, LLM can achieve on average 3.4 × lower memory latency variation compared to HBM2.0

Helmholtz Portfolio Theme Large-Scale Data Management and Analysis (LSDMA)

The Helmholtz Association funded the "Large-Scale Data Management and Analysis" portfolio theme from 2012-2016. Four Helmholtz centres, six universities and another research institution in Germany joined to enable data-intensive science by optimising data life cycles in selected scientific communities. In our Data Life cycle Labs, data experts performed joint R&D together with scientific communities. The Data Services Integration Team focused on generic solutions applied by several communities