POWAR: Power-Aware Routing in HPC Networks with On/Off Links

[EN] In order to save energy in HPC interconnection networks, one usual proposal is to switch idle links into a low-power mode after a certain time without any transmission, as IEEE Energy Efficient Ethernet standard proposes. Extending the low-power mode mechanism, we propose POWer-Aware Routing (POWAR), a simple power-aware routing and selection function for fat-tree and torus networks. POWAR adapts the amount of network links that can be used, taking into account the network load, and obtaining great energy savings in the network (55%-65%) and the entire system (9%-10%) with negligible performance overhead.This work has been supported by the Spanish MINECO and European Commission (FEDER funds) under project TIN2015-66972-C5-1-R. Francisco J. Andujar has been partially funded by the Spanish MICINN and by the ERDF program of the European Union: PCAS Project (TIN2017-88614-R), CAPAP-H6 (TIN2016-81840-REDT), and Junta de Castilla y Leon FEDER Grant VA082P17 (PROPHET Project).Andújar-Muñoz, FJ.; Coll, S.; Alonso Díaz, M.; López Rodríguez, PJ.; Martínez-Rubio, J. (2019). POWAR: Power-Aware Routing in HPC Networks with On/Off Links. ACM Transactions on Architecture and Code Optimization. 15(4):1-22. https://doi.org/10.1145/3293445S122154Abts, D., Marty, M. R., Wells, P. M., Klausler, P., & Liu, H. (2010). Energy proportional datacenter networks. Proceedings of the 37th annual international symposium on Computer architecture - ISCA ’10. doi:10.1145/1815961.1816004Adiga, N. R., Blumrich, M. A., Chen, D., Coteus, P., Gara, A., Giampapa, M. E., … Vranas, P. (2005). Blue Gene/L torus interconnection network. IBM Journal of Research and Development, 49(2.3), 265-276. doi:10.1147/rd.492.0265M. Alonso S. Coll J. M. Martínez V. Santonja and P. López. 2015. Power consumption management in fat-tree interconnection networks. Parallel Comput. 48 C (Oct. 2015) 59--80. 10.1016/j.parco.2015.03.007 M. Alonso S. Coll J. M. Martínez V. Santonja and P. López. 2015. Power consumption management in fat-tree interconnection networks. Parallel Comput. 48 C (Oct. 2015) 59--80. 10.1016/j.parco.2015.03.007Marina Alonso, Coll, S., Martínez, J.-M., Santonja, V., López, P., & Duato, J. (2010). Power saving in regular interconnection networks. Parallel Computing, 36(12), 696-712. doi:10.1016/j.parco.2010.08.003Bob Alverson Edwin Froese Larry Kaplan and Duncan Roweth. 2012. Cray XC series network. Cray Inc. White Paper WP-Aries01-1112 (2012). Bob Alverson Edwin Froese Larry Kaplan and Duncan Roweth. 2012. Cray XC series network. Cray Inc. White Paper WP-Aries01-1112 (2012).Anderson, T. E., Owicki, S. S., Saxe, J. B., & Thacker, C. P. (1993). High-speed switch scheduling for local-area networks. ACM Transactions on Computer Systems, 11(4), 319-352. doi:10.1145/161541.161736Andujar, F. J., Villar, J. A., Sanchez, J. L., Alfaro, F. J., & Escudero-Sahuquillo, J. (2015). VEF Traces: A Framework for Modelling MPI Traffic in Interconnection Network Simulators. 2015 IEEE International Conference on Cluster Computing. doi:10.1109/cluster.2015.141Barroso, L. A., & Hölzle, U. (2007). The Case for Energy-Proportional Computing. Computer, 40(12), 33-37. doi:10.1109/mc.2007.443Camacho, J., & Flich, J. (2011). HPC-Mesh: A Homogeneous Parallel Concentrated Mesh for Fault-Tolerance and Energy Savings. 2011 ACM/IEEE Seventh Symposium on Architectures for Networking and Communications Systems. doi:10.1109/ancs.2011.17Chen, D., Parker, J. J., Eisley, N. A., Heidelberger, P., Senger, R. M., Sugawara, Y., … Steinmacher-Burow, B. (2011). The IBM Blue Gene/Q interconnection network and message unit. Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC ’11. doi:10.1145/2063384.2063419Chen, L., & Pinkston, T. M. (2012). NoRD: Node-Router Decoupling for Effective Power-gating of On-Chip Routers. 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture. doi:10.1109/micro.2012.33Christensen, K., Reviriego, P., Nordman, B., Bennett, M., Mostowfi, M., & Maestro, J. (2010). IEEE 802.3az: the road to energy efficient ethernet. IEEE Communications Magazine, 48(11), 50-56. doi:10.1109/mcom.2010.5621967Dally, & Seitz. (1987). Deadlock-Free Message Routing in Multiprocessor Interconnection Networks. IEEE Transactions on Computers, C-36(5), 547-553. doi:10.1109/tc.1987.1676939Das, R., Narayanasamy, S., Satpathy, S. K., & Dreslinski, R. G. (2013). Catnap. Proceedings of the 40th Annual International Symposium on Computer Architecture - ISCA ’13. doi:10.1145/2485922.2485950Derradji, S., Palfer-Sollier, T., Panziera, J.-P., Poudes, A., & Atos, F. W. (2015). The BXI Interconnect Architecture. 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects. doi:10.1109/hoti.2015.15Jack Dongarra Hans W. Meuer and Erich Strohmaier. 2018. TOP500 Supercomputer Sites. Retrieved from https://www.top500.org. Jack Dongarra Hans W. Meuer and Erich Strohmaier. 2018. TOP500 Supercomputer Sites. Retrieved from https://www.top500.org.Duato, J. (1993). A new theory of deadlock-free adaptive routing in wormhole networks. IEEE Transactions on Parallel and Distributed Systems, 4(12), 1320-1331. doi:10.1109/71.250114José Duato Sudhakar Yalamanchili and Lionel Ni. 2003. Interconnection Networks. An Engineering Approach. Morgan Kaufmann Publishers Inc. San Francisco CA. José Duato Sudhakar Yalamanchili and Lionel Ni. 2003. Interconnection Networks. An Engineering Approach. Morgan Kaufmann Publishers Inc. San Francisco CA.GALGO 2017. GALGO—Albacete Research Institute of Informatics Supercomputer Center homepage. Retrieved from http://www.i3a.uclm.es/galgo. GALGO 2017. GALGO—Albacete Research Institute of Informatics Supercomputer Center homepage. Retrieved from http://www.i3a.uclm.es/galgo.Greenberg, A., Hamilton, J., Maltz, D. A., & Patel, P. (2008). The cost of a cloud. ACM SIGCOMM Computer Communication Review, 39(1), 68-73. doi:10.1145/1496091.1496103HPCC {n.d.}. HPC Challenge Benchmark. Retrieved from http://icl.cs.utk.edu/hpcc/index.html. HPCC {n.d.}. HPC Challenge Benchmark. Retrieved from http://icl.cs.utk.edu/hpcc/index.html.Hluchyj, M. G., & Karol, M. J. (1988). Queueing in high-performance packet switching. IEEE Journal on Selected Areas in Communications, 6(9), 1587-1597. doi:10.1109/49.12886Koibuchi, M., Otsuka, T., Hiroki Matsutani, & Amano, H. (2009). An on/off link activation method for low-power ethernet in PC clusters. 2009 IEEE International Symposium on Parallel & Distributed Processing. doi:10.1109/ipdps.2009.5161069Phillips, J. C., Braun, R., Wang, W., Gumbart, J., Tajkhorshid, E., Villa, E., … Schulten, K. (2005). Scalable molecular dynamics with NAMD. Journal of Computational Chemistry, 26(16), 1781-1802. doi:10.1002/jcc.20289Pronk, S., Páll, S., Schulz, R., Larsson, P., Bjelkmar, P., Apostolov, R., … Lindahl, E. (2013). GROMACS 4.5: a high-throughput and highly parallel open source molecular simulation toolkit. Bioinformatics, 29(7), 845-854. doi:10.1093/bioinformatics/btt055Reviriego, P., Hernandez, J., Larrabeiti, D., & Maestro, J. (2009). Performance evaluation of energy efficient ethernet. IEEE Communications Letters, 13(9), 697-699. doi:10.1109/lcomm.2009.090880K. P. Saravanan and P. Carpenter. 2018. PerfBound: Conserving energy with bounded overheads in on/off-based HPC interconnects. IEEE Trans. Comput. (2018) 1--1. 10.1109/TC.2018.2790394 K. P. Saravanan and P. Carpenter. 2018. PerfBound: Conserving energy with bounded overheads in on/off-based HPC interconnects. IEEE Trans. Comput. (2018) 1--1. 10.1109/TC.2018.2790394Saravanan, K. P., Carpenter, P. M., & Ramirez, A. (2013). Power/performance evaluation of energy efficient Ethernet (EEE) for High Performance Computing. 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). doi:10.1109/ispass.2013.6557171Soteriou, V., & Li-Shiuan Peh. (s. f.). Dynamic power management for power optimization of interconnection networks using on/off links. 11th Symposium on High Performance Interconnects, 2003. Proceedings. doi:10.1109/conect.2003.1231472Totoni, E., Jain, N., & Kale, L. V. (2013). Toward Runtime Power Management of Exascale Networks by on/off Control of Links. 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum. doi:10.1109/ipdpsw.2013.191VEF 2017. VEF traces homepage. Retrieved from http://www.i3a.info/VEFtraces. VEF 2017. VEF traces homepage. Retrieved from http://www.i3a.info/VEFtraces

Migration energy aware reconfigurations of virtual network function instances in NFV architectures

Network function virtualization (NFV) is a new network architecture framework that implements network functions in software running on a pool of shared commodity servers. NFV can provide the infrastructure flexibility and agility needed to successfully compete in today's evolving communications landscape. Any service is represented by a service function chain (SFC) that is a set of VNFs to be executed according to a given order. The running of VNFs needs the instantiation of VNF instances (VNFIs) that are software modules executed on virtual machines. This paper deals with the migration problem of the VNFIs needed in the low traffic periods to turn OFF servers and consequently to save energy consumption. Though the consolidation allows for energy saving, it has also negative effects as the quality of service degradation or the energy consumption needed for moving the memories associated to the VNFI to be migrated. We focus on cold migration in which virtual machines are redundant and suspended before performing migration. We propose a migration policy that determines when and where to migrate VNFI in response to changes to SFC request intensity. The objective is to minimize the total energy consumption given by the sum of the consolidation and migration energies. We formulate the energy aware VNFI migration problem and after proving that it is NP-hard, we propose a heuristic based on the Viterbi algorithm able to determine the migration policy with low computational complexity. The results obtained by the proposed heuristic show how the introduced policy allows for a reduction of the migration energy and consequently lower total energy consumption with respect to the traditional policies. The energy saving can be on the order of 40% with respect to a policy in which migration is not performed

(EMC)-M-3: Improving Energy Efficiency via Elastic Multi-Controller SDN in Data Center Networks

Energy consumed by network constitutes a significant portion of the total power budget in modern data centers. Thus, it is critical to understand the energy consumption and improve the power efficiency of data center networks (DCNs). In doing so, one straightforward and effective way is to make the size of DCNs elastic along with traffic demands, i.e., turning off unnecessary network components to reduce the energy consumption. Today, software defined networking (SDN), as one of the most promising solutions for data center management, provides a paradigm to elastically control the resources of DCNs. However, to the best of our knowledge, the features of SDN have not been fully leveraged to improve the power saving, especially for large-scale multi-controller DCNs. To address this problem, we propose (EMC)-M-3, a mechanism to improve DCN\u27s energy efficiency via the elastic multi-controller SDN. In (EMC)-M-3, the energy optimizations for both forwarding and control plane are considered by utilizing SDN\u27s fine-grained routing and dynamic control mapping. In particular, the flow network theory and the bin-packing heuristic are used to deal with the forwarding plane and control plane, respectively. Our simulation results show that E3MC can achieve more efficient power management, especially in highly structured topologies such as Fat-Tree and BCube, by saving up to 50% of network energy, at an acceptable level of computation cost

Congestion control, energy efficiency and virtual machine placement for data centers

Data centers, facilities with communications network equipment and servers for data processing and/or storage, are prevalent and essential to provide a myriad of services and applications for various private, non-profit, and government systems, and they also form the foundation of cloud computing, which is transforming the technological landscape of the Internet. With rapid deployment of modern high-speed low-latency large-scale data centers, many issues have emerged in data centers, such as data center architecture design, congestion control, energy efficiency, virtual machine placement, and load balancing. The objective of this thesis is multi-fold. First, an enhanced Quantized Congestion Notification (QCN) congestion notification algorithm, called fair QCN (FQCN), is proposed to improve rate allocation fairness of multiple flows sharing one bottleneck link in data center networks. Detailed analysis on FQCN and simulation results is provided to validate the fair share rate allocation while maintaining the queue length stability. Furthermore, the effects of congestion notification algorithms, including QCN, AF-QCN and FQCN, are investigated with respect to TCP throughput collapse. The results show that FQCN can significantly enhance TCP throughput performance, and achieve better TCP throughput than QCN and AF-QCN in a TCP Incast setting. Second, a unified congestion detection, notification and control system for data center networks is designed to efficiently resolve network congestion in a uniform solution and to ensure convergence to statistical fairness with “no state” switches simultaneously. The architecture of the proposed system is described in detail and the FQCN algorithm is implemented in the proposed framework. The simulation results of the FQCN algorithm implemented in the proposed framework validate the robustness and efficiency of the proposed congestion control system. Third, a two-level power optimization model, namely, Hierarchical EneRgy Optimization (HERO), is established to reduce the power consumption of data center networks by switching off network switches and links while still guaranteeing full connectivity and maximizing link utilization. The power-saving performance of the proposed HERO model is evaluated by simulations with different traffic patterns. The simulation results have shown that HERO can reduce the power consumption of data center networks effectively with reduced complexity. Last, several heterogeneity aware dominant resource assistant heuristic algorithms, namely, dominant residual resource aware first-fit decreasing (DRR-FFD), individual DRR-FFD (iDRR-FFD) and dominant residual resource based bin fill (DRR-BinFill), are proposed for virtual machine (VM) consolidation. The proposed heuristic algorithms exploit the heterogeneity of the VMs’ requirements for different resources by capturing the differences among VMs’ demands, and the heterogeneity of the physical machines’ resource capacities by capturing the differences among physical machines’ resources. The performance of the proposed heuristic algorithms is evaluated with different classes of synthetic workloads under different VM requirement heterogeneity conditions, and the simulation results demonstrate that the proposed heuristics achieve quite similar consolidation performance as dimension-aware heuristics with almost the same computational cost as those of the single dimensional heuristics

Load-optimization in reconfigurable data-center networks: algorithms and complexity of flow routing

Emerging reconfigurable data centers introduce unprecedented flexibility in how the physical layer can be programmed to adapt to current traffic demands. These reconfigurable topologies are commonly hybrid, consisting of static and reconfigurable links, enabled by e.g., an Optical Circuit Switch (OCS) connected to top-of-rack switches in Clos networks. Even though prior work has showcased the practical benefits of hybrid networks, several crucial performance aspects are not well understood. For example, many systems enforce artificial segregation of the hybrid network parts, leaving money on the table. In this article, we study the algorithmic problem of how to jointly optimize topology and routing in reconfigurable data centers, in order to optimize a most fundamental metric, maximum link load. The complexity of reconfiguration mechanisms in this space is unexplored at large, especially for the following cross-layer network-design problem: given a hybrid network and a traffic matrix, jointly design the physical layer and the flow routing in order to minimize the maximum link load. We chart the corresponding algorithmic landscape in our work, investigating both un-/splittable flows and (non-)segregated routing policies. A topological complexity classification of the problem reveals NP-hardness in general for network topologies that are trees of depth at least two, in contrast to the tractability on trees of depth one. We moreover prove that the problem is not submodular for all these routing policies, even in multi-layer trees. However, networks that can be abstracted by a single packet switch (e.g., nonblocking Fat-Tree topologies) can be optimized efficiently, and we present optimal polynomial-time algorithms accordingly. We complement our theoretical results with trace-driven simulation studies, where our algorithms can significantly improve the network load in comparison to the state-of-the-art