Towards Thermally-Aware Design of 3D MPSoCs with Inter-Tier Cooling

Abstract—New tendencies envisage 3D Multi-Processor System-On-Chip (MPSoC) design as a promising solution to keep increasing the performance of the next-generation highperformance computing (HPC) systems. However, as the power density of HPC systems increases with the arrival of 3D MPSoCs, supplying electrical power to the computing equipment and constantly removing the generated heat is rapidly becoming the dominant cost in any HPC facility. Thus, both power and thermal/cooling implications play a major role in the design of new HPC systems, given the energy constraints in our society. Therefore, EPFL, IBM and ETHZ have been working within the CMOSAIC Nano-Tera.ch program project in the last three years on the development of a holistic thermally-aware design. This paper presents the exploration in CMOSAIC of novel cooling technologies, as well as suitable thermal modeling and system-level design methods, which are all necessary to develop 3D MPSoCs with inter-tier liquid cooling systems. As a result, we develop energy-efficient run-time thermal control strategies to achieve energy-efficient cooling mechanisms to compress almost 1 Tera nano sized functional units into one cubic centimeter with a 10 to 100 fold higher connectivity than otherwise possible. The proposed thermally-aware design paradigm includes exploring the synergies of hardware-, software- and mechanical-based thermal control techniques as a fundamental step to design 3D MPSoCs for HPC systems. More precisely, we target the use of inter-tier coolants ranging from liquid water and twophase refrigerants to novel engineered environmentally friendly nano-fluids, as well as using specifically designed micro-channel arrangements, in combination with the use of dynamic thermal management at system-level to tune the flow rate of the coolant in each micro-channel to achieve thermally-balanced 3D-ICs. Our management strategy prevents the system from surpassing the given threshold temperature while achieving up to 67% reduction in cooling energy and up to 30% reduction in system-level energy in comparison to setting the flow rate at the maximum value to handle the worst-case temperature

Unified Modeling Suite for Two-Phase Flow, Convective Boiling and Condensation in Macro-and Micro-Channels

This paper was presented at the 4th Micro and Nano Flows Conference (MNF2014), which was held at University College, London, UK. The conference was organised by Brunel University and supported by the Italian Union of Thermofluiddynamics, IPEM, the Process Intensification Network, the Institution of Mechanical Engineers, the Heat Transfer Society, HEXAG - the Heat Exchange Action Group, and the Energy Institute, ASME Press, LCN London Centre for Nanotechnology, UCL University College London, UCL Engineering, the International NanoScience Community, www.nanopaprika.eu.The present paper focuses on the unified modeling suite for annular flow that the authors have and continue to develop. Annular flow is of fundamental importance to the thermal design and simulation of microevaporators and micro-condensers for compact two-phase cooling systems of high heat flux components for the thermal management of computer chips, power electronics, laser diodes and high energy physics particle detectors. First, the unified suite of methods is presented, illustrating in particular the most recent updates. Then, results for convective evaporation of refrigerants in non-circular multi-microchannel configurations for microelectronics cooling are presented and discussed. The annular flow suite includes models to predict the void fraction, the entrained liquid fraction, the wall shear stress and pressure gradient, and a turbulence model for momentum and heat transport inside the annular liquid film. The turbulence model, in particular, allows prediction of the local average liquid film thicknesses and the local heat transfer coefficients during convective evaporation and condensation. The benefit of a unified modeling suite is that all the included prediction methods are consistently formulated and are proven to work well together, and provide a platform for continued advancement based on the other models in the suite

Integrated Microfluidic Power Generation and Cooling for Bright Silicon MPSoCs

The soaring demand for computing power in our digital information age has produced as collateral undesirable effect a surge in power consumption and heat density for computing servers. Accordingly, 30-40% of the energy consumed in state-of-the-art servers is dissipated in cooling. The remaining energy is used for computation, and causes the temperature ramp-up to operating conditions that already preclude operating all the cores at maximum performance levels, in order to prevent system overheating and failures. This situation is set to worsen as shipments of high-end (i.e., even denser) many-core servers are increasing at a 25% compound annual growth rate. Thus, state-of-the-art worst-case power and cooling delivery solutions on servers are reaching their limits and it will no longer be possible to power up simultaneously all the available on-chip cores (situation known as the existence of "dark silicon"); hence, drastically limiting the benefits of technology scaling. This presentation aims to completely revise the prevailing worst-case power and cooling provisioning paradigm for servers by championing a disruptive approach to computing server architecture design that prevents dark silicon. This proposed approach integrates a flexible heterogeneous many-core architecture template with an on-chip microfluidic fuel cell network for joint cooling delivery and power supply (i.e., local power generation and delivery), as well as a holistic power-temperature model predictive controller exploiting the server software stack, in order to achieve scalable and energy-minimal server architectures. Thanks to the disruptive system-level many-core architecture with microfluidic power and cooling delivery, as well as the complementary temperature control, we can envision the removal of the current limits of power delivery and heat dissipation in server designs, subsequently avoiding dark silicon in future servers and enabling new perspectives in future energy-proportional server designs

Thermal Balancing of Liquid-Cooled 3D-MPSoCs Using Channel Modulation

While possessing the potential to replace conventional air-cooled heat sinks, inter-tier microchannel liquid cooling of 3D ICs also creates the problem of increased thermal gradients from the fluid inlet to outlet ports [1, 2]. These cooling-induced thermal gradients can be high enough to create undesirable stress in the ICs, undermining the structural reliability and lifetimes. In this paper, we present a novel design-time solution for the thermal gradient problem in liquid-cooled 3D Multi-Processor System-on-Chip (MPSoC) architectures. The proposed method is based on channel width modulation and provides the designers with an additional dimension in the design-space exploration. We formulate the channel width modulation as an optimal control design problem to minimize the temperature gradients in the 3D IC while meeting the design constraints. The proposed thermal balancing technique uses an analytical model for forced convective heat transfer in microchannels, and has been applied to a two tier 3D-MPSoC. The results show that the proposed approach can reduce thermal gradients by up to 31% when applied to realistic 3D-MPSoC architectures, while maintaining pressure drops in the microchannels well below their safe limits of operation

Two-Phase Flow Boiling in a Single Layer of Future High-Performance 3D Stacked Computer Chips

The present study focuses on an experimental investigation of two-phase flow boiling in a silicon multi-microchannel evaporator, which emulates a single layer of a 3D stacked computer chip. The micro-evaporator is comprised of 67 parallel channels, each having a 100 x 100 μm2 cross-section area, and separated by 50 μm-wide fins. Two aluminium micro-heaters were sputtered onto the backside of the test section to provide two 0.5 cm2 heated areas in order to simulate the power dissipated by active component in 3D CMOS chips. The experiments were performed with a second identical test section having 50 μm-wide, 100 μm-deep, and 100 μm-long restrictions (micro-orifices) at the inlet of each channel to stabilize the two-phase flow. The goal of this experimental campaign was to perform simultaneous high-speed flow visualization and infra-red measurements of the two-phase flow and heat transfer dynamics across the entire micro-evaporator area. Refrigerants R245fa, R236fa and R1234ze(E) were chosen as the working fluids. The micro-orifices successfully suppressed back flow, eliminated flow instabilities, provided a good flow distribution, and started the boiling process with some flashed vapor. Thermal performance was found to be uniform widthwise using these orifices

PowerCool: Simulation of Integrated Microfluidic Power Generation in Bright Silicon MPSoCs

Integrated microfluidic power generation and power delivery promises to be a disruptive packaging technology with the potential to combat dark silicon. It essentially consists of integrated microchannel-based electrochemical “flow cells” in a 2D/3D multiprocessor system-on-chip (MPSoC), that generate electricity to power up the entire or part of the chip, while also simultaneously acting as a high-efficiency microfluidic heat sink. Further development of this technology requires efficient modeling tools that would assess the efficacy of such solutions and help perform early-stage design space exploration. In this paper, we propose a compact mathematical model, called Power- Cool, that performs electro-chemical modeling and simulation of integrated microfluidic power generation in MPSoCs. The accuracy of the model has been validated against fine-grained multiphysics simulations of flow cells in the COMSOL software that is unsuitable for EDA because of large simulation times. PowerCool model is demonstrated to be up to 425x times faster than COMSOL simulations while incurring a worst-case error of only 5%. Furthermore, the PowerCool model has been used to study and assess the efficacy of this technology for a test MPSoC

Temperature-Aware Design and Management for 3D Multi-Core Architectures

Vertically-integrated 3D multiprocessors systems-on-chip (3D MPSoCs) provide the means to continue integrating more functionality within a unit area while enhancing manufacturing yields and runtime performance. However, 3D MPSoCs incur amplified thermal challenges that undermine the corresponding reliability. To address these issues, several advanced cooling technologies, alongside temperature-aware design-time optimizations and run-time management schemes have been proposed. In this monograph, we provide an overall survey on the recent advances in temperature-aware 3D MPSoC considerations. We explore the recent advanced cooling strategies, thermal modeling frameworks, design-time optimizations and run-time thermal management schemes that are primarily targeted for 3D MPSoCs. Our aim of proposing this survey is to provide a global perspective, highlighting the advancements and drawbacks on the recent state-of-the-ar

Thermal and Visual Operational Characteristics of Multi-Microchannel Evaporators using Refrigerants

Two-phase flow operational stability of refrigerants R245fa, R236fa, and R1234ze(E) in 100 x 100 μm2 multi-microchannels for cooling of future high-performance 3D stacked architectures with interlayer cooling has been addressed in the current experimental investigation. Without any inlet restrictions in the micro-evaporator, significant flow instabilities, back flow, and flow maldistribution led to high-amplitude and high-frequency temperature and pressure oscillations. Such undesired phenomena were inhibited by placing rectangular restrictions (micro-orifices) at the inlet of each channel, thus ensuring a wide range of stable two-phase flow operating conditions. The effects of different orifice expansion ratios and fluids on the performance of the evaporator were studied by using suitably designed modular test sections. Simultaneous high-speed video and infra-red camera visualizations of the two-phase flow and heat transfer dynamics across the micro-evaporator area allowed the various different operating regimes to be identified and then represented by the two-phase flow operational maps. Two-phase flow flashed by the micro-orifices was identified as the optimal operating condition, while dissipating high heat fluxes and keeping the junction chip temperature below a typical CPU operating condition. In the present study, a novel in-situ pixel by pixel technique was developed to calibrate the raw infra-red images, thus converting them into two-dimensional temperature fields of 10’000 pixels over the test section surface. A comprehensive analysis of those temperature maps supported by the flow visualization videos confirmed that the two-phase flow patterns appearing in the channel and the transitions between them have a remarkable influence on the heat transfer coefficients, which were determined taking into account 3D heat spreading. The inlet and the outlet restriction pressure losses were quantified in order to accurately simulate the hydraulic performance of microchannel evaporators and provide more reliable heat transfer data. The heat transfer coefficients were determined with a very fine resolution that enabled the trends in the heat transfer coefficients along the channel length in the neighborhood of the flow transition from coalescing bubble to the annular flow to be studied in detail. It was shown that the heat transfer coefficient does not change sharply at the transition zone, but rather has a smooth change in trend. The characteristic U-shape of the heat transfer coefficient trend was observed, where the descending branch of the curve corresponds to the coalescing elongated bubble flow regime, while ascending one represents the increasing heat transfer coefficient in the annular flow regime. A good agreement of the experimental heat transfer coefficients and one existing flow pattern-based prediction method was found and a new vapor quality buffer is proposed as an update of this model

A 3D ALE Finite Element Method for Two-Phase Flows with Phase Change

A new numerical method is proposed to study two-phase flow and heat transfer for interlayer cooling of the new generation of multi-stacked computer chips. The fluid flow equations are developed in 3-dimensions based on the Arbitrary Lagrangian-Eulerian formulation (ALE) and the Finite Element Method (FEM), creating a new two-phase method with an improved model for the liquid-gas interface. A new adaptive mesh update procedure is also proposed for effective management of the mesh at the two-phase interface to remove, add and repair surface elements, since the computational mesh nodes move according to the flow. The Lagrangian description explicitly defines the two-phase interface position by a set of interconnected nodes which ensures a sharp representation of the boundary, including the role of the surface tension. The new methodology for computing the curvature leads to accurate results with moderate programming effort and computational cost. Static and dynamic tests have been carried out to validate the method and so far all the obtained results have compared well to analytical solutions and experimental results found in the literature, demonstrating that the new proposed methodology to simulate two-phase flows provides good accuracy to describe the interfacial forces and bubble dynamics. The new code was then used to simulate elengated bubble flows in square microchannels, being considered for two-phase interlayer cooling in future 3D-IC compute chips