HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing

The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires a higher level of scalability and dependability than that provided by current file and storage systems. The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to real application performance improvements. The project involves the following components: 1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns. 2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability 3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching. 4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching. 5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies

Agrégation et routage efficace de données dans les réseaux de capteurs sans fils

Wireless Sensor Networks (WSNs) have gained much attention in a large range of technical fields such as industrial, military, environmental monitoring etc. Sensors are powered by batteries, which are not easy to replace in harsh environments. The energy stored by each sensor is the greatest impediment for increasing WSN lifetime. Since data transmission consumes more energy, our major concern is how to efficiently transmit the data from all sensors towards a sink. We suggest three tree-based data aggregation algorithms: Depth-First Search Aggregation (DFSA), Flooding Aggregation (FA) and Well-Connected Dominating Set Aggregation (WCDSA) to reduce the number of transmissions from each sensor towards the sink. Tree-based data aggregation suffers from increased data delivery time because the parents must wait for the data from their leaves. Some parents might have many leaves, making it very expensive for a parent to store all incoming data in its buffer. We need to determine the aggregation time each parent in the tree has to spend in aggregating and processing the data from its leaves. We propose an Efficient Tree-based Aggregation and Processing Time (ETAPT) algorithm using Appropriate Data Aggregation and Processing Time (ADAPT) metric. Given the maximum acceptable latency, ETAPT's algorithm takes into account the position of parents, their number of leaves and the depth of the tree, in order to compute an optimal ADAPT time. At any time, the amount of data aggregated by parents may become greater than the amount of data that can be forwarded. We propose the introduction into the network of many data aggregators called Mini-Sinks (MSs). MSs are mobile and move according to a random mobility model inside the sensor field to maintain the fully-connected network in order to aggregate the data based on the controlled Multipath Energy Conserving Routing Protocol (MECRP). Sensors may use many radio interfaces sharing a single wireless channel, which they may use to communicate with several neighbours. Two sensors operating on the same wireless channel may interfere with each other during the transmission of data. We need to know which channel to use in the presence of multiple channels for a given transmission. We propose a distributed Well-Connected Dominating Set Channel Assignment (WCDS-CA) approach, in which the number of channels that are needed over all sensor nodes in the network in such a way that adjacent sensor nodes are assigned to distinct channels.Les Réseaux de Capteurs Sans Fils (RCSFs) ont pris beaucoup d'importance dans plusieurs domaines tels que l'industrie, l'armée, la pollution atmosphérique etc. Les capteurs sont alimentés par des batteries qui ne sont pas faciles à remplacer surtout dans les environnements peu accessibles. L'énergie de chaque capteur est considérée comme la source première d'augmentation de la durée de vie des RCSFs. Puisque la transmission de données est plus coûteuse en consommation d'énergie, notre préoccupation première est de proposer une technique efficace de transmission des données de tous les capteurs vers le sink tout en réduisant la consommation en énergie. Nous suggérons trois trois algorithmes d'agrégation de données basé sur la construction d'arbres : Depth-First Search Aggregation (DFSA), Flooding Aggregation (FA) et Well-Connected Dominating Set Aggregation (WCDSA) qui permettront de réduire le nombre de transmissions de chaque capteur vers le sink. L'agrégation des données basée sur la construction d'arbres souffre du délai de délivrance de données parce que les parents doivent attendre de recevoir les données de leurs feuilles. Certains parents pourraient avoir beaucoup de feuilles, et il serait alors assez coûteux pour un parent de stocker toutes les données entrantes dans sa mémoire. Ainsi, nous devons déterminer le temps que chaque parent doit mettre pour agréger et traiter les données de ses feuilles. Nous proposons un algorithme, Efficient Tree-based Aggregation and Processing Time (ETAPT) qui utilise la métrique Appropriate Data Aggregation and Processing Time (ADAPT). Etant donné la durée maximale acceptable, l'algorithme ETAPT prend en compte la position des parents, le nombre de feuilles et la profondeur de l'arbre pour calculer l'ADAPT optimal. A n'importe quel moment pendant l'agrégation des données par les parents, il peut arriver que la quantité de données collectées soit très grande et dépasse la quantité de stockage maximale de données que peut contenir leurs mémoires. Nous proposons l'introduction dans le réseau de plusieurs collecteurs de données appelés Mini-Sinks (MSs). Ces MSs sont mobiles et se déplacent selon un modèle de mobilité aléatoire dans le réseau pour maintenir la connexité afin d'assurer la collecte contrôlée des données basée sur le protocole de routage Mulipath Energy Conserving Routing Protocol (MECRP). Les capteurs peuvent être équipés de plusieurs interfaces radios partageant un seul canal sans fil avec lequel ils peuvent communiquer avec plusieurs voisins. La transmission des données à travers une liaison de communication entre deux parents peut interférer avec les transmissions d'autres liaisons si elles transmettent à travers le même canal. Nous avons besoin de savoir quel canal utiliser en présence de plusieurs canaux pour une transmission donnée. Nous proposons une méthode distribuée appelée: Well Connected Dominating Set Channel Assignement (WCDS-CA), pour calculer le nombre de canaux qui seront alloués à tous les capteurs de telle sorte que les capteurs adjacents se voient attribués des canaux différent

Congestion control, energy efficiency and virtual machine placement for data centers

Data centers, facilities with communications network equipment and servers for data processing and/or storage, are prevalent and essential to provide a myriad of services and applications for various private, non-profit, and government systems, and they also form the foundation of cloud computing, which is transforming the technological landscape of the Internet. With rapid deployment of modern high-speed low-latency large-scale data centers, many issues have emerged in data centers, such as data center architecture design, congestion control, energy efficiency, virtual machine placement, and load balancing. The objective of this thesis is multi-fold. First, an enhanced Quantized Congestion Notification (QCN) congestion notification algorithm, called fair QCN (FQCN), is proposed to improve rate allocation fairness of multiple flows sharing one bottleneck link in data center networks. Detailed analysis on FQCN and simulation results is provided to validate the fair share rate allocation while maintaining the queue length stability. Furthermore, the effects of congestion notification algorithms, including QCN, AF-QCN and FQCN, are investigated with respect to TCP throughput collapse. The results show that FQCN can significantly enhance TCP throughput performance, and achieve better TCP throughput than QCN and AF-QCN in a TCP Incast setting. Second, a unified congestion detection, notification and control system for data center networks is designed to efficiently resolve network congestion in a uniform solution and to ensure convergence to statistical fairness with “no state” switches simultaneously. The architecture of the proposed system is described in detail and the FQCN algorithm is implemented in the proposed framework. The simulation results of the FQCN algorithm implemented in the proposed framework validate the robustness and efficiency of the proposed congestion control system. Third, a two-level power optimization model, namely, Hierarchical EneRgy Optimization (HERO), is established to reduce the power consumption of data center networks by switching off network switches and links while still guaranteeing full connectivity and maximizing link utilization. The power-saving performance of the proposed HERO model is evaluated by simulations with different traffic patterns. The simulation results have shown that HERO can reduce the power consumption of data center networks effectively with reduced complexity. Last, several heterogeneity aware dominant resource assistant heuristic algorithms, namely, dominant residual resource aware first-fit decreasing (DRR-FFD), individual DRR-FFD (iDRR-FFD) and dominant residual resource based bin fill (DRR-BinFill), are proposed for virtual machine (VM) consolidation. The proposed heuristic algorithms exploit the heterogeneity of the VMs’ requirements for different resources by capturing the differences among VMs’ demands, and the heterogeneity of the physical machines’ resource capacities by capturing the differences among physical machines’ resources. The performance of the proposed heuristic algorithms is evaluated with different classes of synthetic workloads under different VM requirement heterogeneity conditions, and the simulation results demonstrate that the proposed heuristics achieve quite similar consolidation performance as dimension-aware heuristics with almost the same computational cost as those of the single dimensional heuristics

GPUの性能分析モデリンクと通信レイテンシの隠蔽に基づく性能の最適化方法に関する研究

学位の種別:課程博士University of Tokyo(東京大学

Telecommunication Economics

This book constitutes a collaborative and selected documentation of the scientific outcome of the European COST Action IS0605 Econ@Tel "A Telecommunications Economics COST Network" which run from October 2007 to October 2011. Involving experts from around 20 European countries, the goal of Econ@Tel was to develop a strategic research and training network among key people and organizations in order to enhance Europe's competence in the field of telecommunications economics. Reflecting the organization of the COST Action IS0605 Econ@Tel in working groups the following four major research areas are addressed: - evolution and regulation of communication ecosystems; - social and policy implications of communication technologies; - economics and governance of future networks; - future networks management architectures and mechanisms

Telecommunication Economics

This book constitutes a collaborative and selected documentation of the scientific outcome of the European COST Action IS0605 Econ@Tel "A Telecommunications Economics COST Network" which run from October 2007 to October 2011. Involving experts from around 20 European countries, the goal of Econ@Tel was to develop a strategic research and training network among key people and organizations in order to enhance Europe's competence in the field of telecommunications economics. Reflecting the organization of the COST Action IS0605 Econ@Tel in working groups the following four major research areas are addressed: - evolution and regulation of communication ecosystems; - social and policy implications of communication technologies; - economics and governance of future networks; - future networks management architectures and mechanisms

Flexible Application-Layer Multicast in Heterogeneous Networks

This work develops a set of peer-to-peer-based protocols and extensions in order to provide Internet-wide group communication. The focus is put to the question how different access technologies can be integrated in order to face the growing traffic load problem. Thereby, protocols are developed that allow autonomous adaptation to the current network situation on the one hand and the integration of WiFi domains where applicable on the other hand

Esquema de controlo para redes multicast baseadas com classes

Doutoramento em Engenharia ElectrotécnicaThe expectations of citizens from the Information Technologies (ITs) are increasing as the ITs have become integral part of our society, serving all kinds of activities whether professional, leisure, safety-critical applications or business. Hence, the limitations of the traditional network designs to provide innovative and enhanced services and applications motivated a consensus to integrate all services over packet switching infrastructures, using the Internet Protocol, so as to leverage flexible control and economical benefits in the Next Generation Networks (NGNs). However, the Internet is not capable of treating services differently while each service has its own requirements (e.g., Quality of Service - QoS). Therefore, the need for more evolved forms of communications has driven to radical changes of architectural and layering designs which demand appropriate solutions for service admission and network resources control. This Thesis addresses QoS and network control issues, aiming to improve overall control performance in current and future networks which classify services into classes. The Thesis is divided into three parts. In the first part, we propose two resource over-reservation algorithms, a Class-based bandwidth Over-Reservation (COR) and an Enhanced COR (ECOR). The over-reservation means reserving more bandwidth than a Class of Service (CoS) needs, so the QoS reservation signalling rate is reduced. COR and ECOR allow for dynamically defining over-reservation parameters for CoSs based on network interfaces resource conditions; they aim to reduce QoS signalling and related overhead without incurring CoS starvation or waste of bandwidth. ECOR differs from COR by allowing for optimizing control overhead minimization. Further, we propose a centralized control mechanism called Advanced Centralization Architecture (ACA), that uses a single state-full Control Decision Point (CDP) which maintains a good view of its underlying network topology and the related links resource statistics on real-time basis to control the overall network. It is very important to mention that, in this Thesis, we use multicast trees as the basis for session transport, not only for group communication purposes, but mainly to pin packets of a session mapped to a tree to follow the desired tree. Our simulation results prove a drastic reduction of QoS control signalling and the related overhead without QoS violation or waste of resources. Besides, we provide a generic-purpose analytical model to assess the impact of various parameters (e.g., link capacity, session dynamics, etc.) that generally challenge resource overprovisioning control. In the second part of this Thesis, we propose a decentralization control mechanism called Advanced Class-based resource OverpRovisioning (ACOR), that aims to achieve better scalability than the ACA approach. ACOR enables multiple CDPs, distributed at network edge, to cooperate and exchange appropriate control data (e.g., trees and bandwidth usage information) such that each CDP is able to maintain a good knowledge of the network topology and the related links resource statistics on real-time basis. From scalability perspective, ACOR cooperation is selective, meaning that control information is exchanged dynamically among only the CDPs which are concerned (correlated). Moreover, the synchronization is carried out through our proposed concept of Virtual Over-Provisioned Resource (VOPR), which is a share of over-reservations of each interface to each tree that uses the interface. Thus, each CDP can process several session requests over a tree without requiring synchronization between the correlated CDPs as long as the VOPR of the tree is not exhausted. Analytical and simulation results demonstrate that aggregate over-reservation control in decentralized scenarios keep low signalling without QoS violations or waste of resources. We also introduced a control signalling protocol called ACOR Protocol (ACOR-P) to support the centralization and decentralization designs in this Thesis. Further, we propose an Extended ACOR (E-ACOR) which aggregates the VOPR of all trees that originate at the same CDP, and more session requests can be processed without synchronization when compared with ACOR. In addition, E-ACOR introduces a mechanism to efficiently track network congestion information to prevent unnecessary synchronization during congestion time when VOPRs would exhaust upon every session request. The performance evaluation through analytical and simulation results proves the superiority of E-ACOR in minimizing overall control signalling overhead while keeping all advantages of ACOR, that is, without incurring QoS violations or waste of resources. The last part of this Thesis includes the Survivable ACOR (SACOR) proposal to support stable operations of the QoS and network control mechanisms in case of failures and recoveries (e.g., of links and nodes). The performance results show flexible survivability characterized by fast convergence time and differentiation of traffic re-routing under efficient resource utilization i.e. without wasting bandwidth. In summary, the QoS and architectural control mechanisms proposed in this Thesis provide efficient and scalable support for network control key sub-systems (e.g., QoS and resource control, traffic engineering, multicasting, etc.), and thus allow for optimizing network overall control performance.À medida que as Tecnologias de Informação (TIs) se tornaram parte integrante da nossa sociedade, a expectativa dos cidadãos relativamente ao uso desses serviços também demonstrou um aumento, seja no âmbito das atividades profissionais, de lazer, aplicações de segurança crítica ou negócios. Portanto, as limitações dos projetos de rede tradicionais quanto ao fornecimento de serviços inovadores e aplicações avançadas motivaram um consenso quanto à integração de todos os serviços e infra-estruturas de comutação de pacotes, utilizando o IP, de modo a extrair benefícios económicos e um controlo mais flexível nas Redes de Nova Geração (RNG). Entretanto, tendo em vista que a Internet não apresenta capacidade de diferenciação de serviços, e sabendo que cada serviço apresenta as suas necessidades próprias, como por exemplo, a Qualidade de Serviço - QoS, a necessidade de formas mais evoluídas de comunicação tem-se tornado cada vez mais visível, levando a mudanças radicais na arquitectura das redes, que exigem soluções adequadas para a admissão de serviços e controlo de recursos de rede. Sendo assim, este trabalho aborda questões de controlo de QoS e rede com o objetivo de melhorar o desempenho do controlo de recursos total em redes atuais e futuras, através da análise dos serviços de acordo com as suas classes de serviço. Esta Tese encontra-se dividida em três partes. Na primeira parte são propostos dois algoritmos de sobre-reserva, o Class-based bandwidth Over-Reservation (COR) e uma extensão melhorada do COR denominado de Enhanced COR (ECOR). A sobre-reserva significa a reserva de uma largura de banda maior para o serviço em questão do que uma classe de serviço (CoS) necessita e, portanto, a quantidade de sinalização para reserva de recursos é reduzida. COR e ECOR consideram uma definição dinâmica de sobre-reserva de parâmetros para CoSs com base nas condições da rede, com vista à redução da sobrecarga de sinalização em QoS sem que ocorra desperdício de largura de banda. O ECOR, por sua vez, difere do COR por permitir a otimização com minimização de controlo de overhead. Além disso, nesta Tese é proposto também um mecanismo de controlo centralizado chamado Advanced Centralization Architecture (ACA) , usando um único Ponto de Controlo de Decisão (CDP) que mantém uma visão ampla da topologia de rede e de análise dos recursos ocupados em tempo real como base de controlo para a rede global. Nesta Tese são utilizadas árvores multicast como base para o transporte de sessão, não só para fins de comunicação em grupo, mas principalmente para que os pacotes que pertençam a uma sessão que é mapeada numa determinada árvore sigam o seu caminho. Os resultados obtidos nas simulações dos mecanismos mostram uma redução significativa da sobrecarga da sinalização de controlo, sem a violação dos requisitos de QoS ou desperdício de recursos. Além disso, foi proposto um modelo analítico no sentido de avaliar o impacto provocado por diversos parâmetros (como por exemplo, a capacidade da ligação, a dinâmica das sessões, etc), no sobre-provisionamento dos recursos. Na segunda parte desta tese propôe-se um mecanismo para controlo descentralizado de recursos denominado de Advanced Class-based resource OverprRovisioning (ACOR), que permite obter uma melhor escalabilidade do que o obtido pelo ACA. O ACOR permite que os pontos de decisão e controlo da rede, os CDPs, sejam distribuídos na periferia da rede, cooperem entre si, através da troca de dados e controlo adequados (por exemplo, localização das árvores e informações sobre o uso da largura de banda), de tal forma que cada CDP seja capaz de manter um bom conhecimento da topologia da rede, bem como das suas ligações. Do ponto de vista de escalabilidade, a cooperação do ACOR é seletiva, o que significa que as informações de controlo são trocadas de forma dinâmica apenas entre os CDPs analisados. Além disso, a sincronização é feita através do conceito proposto de Recursos Virtuais Sobre-Provisionado (VOPR), que partilha as reservas de cada interface para cada árvore que usa a interface. Assim, cada CDP pode processar pedidos de sessão numa ou mais árvores, sem a necessidade de sincronização entre os CDPs correlacionados, enquanto o VOPR da árvore não estiver esgotado. Os resultados analíticos e de simulação demonstram que o controlo de sobre-reserva é agregado em cenários descentralizados, mantendo a sinalização de QoS baixa sem perda de largura de banda. Também é desenvolvido um protocolo de controlo de sinalização chamado ACOR Protocol (ACOR-P) para suportar as arquitecturas de centralização e descentralização deste trabalho. O ACOR Estendido (E-ACOR) agrega a VOPR de todas as árvores que se originam no mesmo CDP, e mais pedidos de sessão podem ser processados sem a necessidade de sincronização quando comparado com ACOR. Além disso, E-ACOR introduz um mecanismo para controlar as informações àcerca do congestionamento da rede, e impede a sincronização desnecessária durante o tempo de congestionamento quando os VOPRs esgotam consoante cada pedido de sessão. A avaliação de desempenho, através de resultados analíticos e de simulação, mostra a superioridade do E-ACOR em minimizar o controlo geral da carga da sinalização, mantendo todas as vantagens do ACOR, sem apresentar violações de QoS ou desperdício de recursos. A última parte desta Tese inclui a proposta para recuperação a falhas, o Survivability ACOR (SACOR), o qual permite ter QoS estável em caso de falhas de ligações e nós. Os resultados de desempenho analisados mostram uma capacidade flexível de sobrevivência caracterizada por um tempo de convergência rápido e diferenciação de tráfego com uma utilização eficiente dos recursos. Em resumo, os mecanismos de controlo de recursos propostos nesta Tese fornecem um suporte eficiente e escalável para controlo da rede, como também para os seus principais sub-sistemas (por exemplo, QoS, controlo de recursos, engenharia de tráfego, multicast, etc) e, assim, permitir a otimização do desempenho da rede a nível do controlo global