Towards green scientific data compression through high-level I/O interfaces

Every HPC system today has to cope with a deluge of data generated by scientific applications, simulations or large- scale experiments. The upscaling of supercomputer systems and infrastructures, generally results in a dramatic increase of their energy consumption. In this paper, we argue that techniques like data compression can lead to significant gains in terms of power efficiency by reducing both network and storage requirements. To that end, we propose a novel methodology for achieving on-the-fly intelligent determination of energy efficient data reduction for a given data set by leveraging state-of-the-art compression algorithms and meta data at application-level I/O. We motivate our work by analyzing the energy and storage saving needs of real-life scientific HPC applications, and review the various compression techniques that can be applied. We find that the resulting data reduction can decrease the data volume transferred and stored by as much as 80% in some cases, consequently leading to significant savings in storage and networking costs

A multi-tier cached I/O architecture for massively parallel supercomputers

Recent advances in storage technologies and high performance interconnects have made possible in the last years to build, more and more potent storage systems that serve thousands of nodes. The majority of storage systems of clusters and supercomputers from Top 500 list are managed by one of three scalable parallel file systems: GPFS, PVFS, and Lustre. Most large-scale scientific parallel applications are written in Message Passing Interface (MPI), which has become the de-facto standard for scalable distributed memory machines. One part of the MPI standard is related to I/O and has among its main goals the portability and efficiency of file system accesses. All of the above mentioned parallel file systems may be accessed also through the MPI-IO interface. The I/O access patterns of scientific parallel applications often consist of accesses to a large number of small, non-contiguous pieces of data. For small file accesses the performance is dominated by the latency of network transfers and disks. Parallel scientific applications lead to interleaved file access patterns with high interprocess spatial locality at the I/O nodes. Additionally, scientific applications exhibit repetitive behaviour when a loop or a function with loops issues I/O requests. When I/O access patterns are repetitive, caching and prefetching can effectively mask their access latency. These characteristics of the access patterns motivated several researchers to propose parallel I/O optimizations both at library and file system levels. However, these optimizations are not always integrated across different layers in the systems. In this dissertation we propose a novel generic parallel I/O architecture for clusters and supercomputers. Our design is aimed at large-scale parallel architectures with thousands of compute nodes. Besides acting as middleware for existing parallel file systems, our architecture provides on-line virtualization of storage resources. Another objective of this thesis is to factor out the common parallel I/O functionality from clusters and supercomputers in generic modules in order to facilitate porting of scientific applications across these platforms. Our solution is based on a multi-tier cache architecture, collective I/O, and asynchronous data staging strategies hiding the latency of data transfer between cache tiers. The thesis targets to reduce the file access latency perceived by the data-intensive parallel scientific applications by multi-layer asynchronous data transfers. In order to accomplish this objective, our techniques leverage the multi-core architectures by overlapping computation with communication and I/O in parallel threads. Prototypes of our solutions have been deployed on both clusters and Blue Gene supercomputers. Performance evaluation shows that the combination of collective strategies with overlapping of computation, communication, and I/O may bring a substantial performance benefit for access patterns common for parallel scientific applications.-----------------------------------------------------------------------------------------------------------------------------En los últimos años se ha observado un incremento sustancial de la cantidad de datos producidos por las aplicaciones científicas paralelas y de la necesidad de almacenar estos datos de forma persistente. Los sistemas de ficheros paralelos como PVFS, Lustre y GPFS han ofrecido una solución escalable para esta demanda creciente de almacenamiento. La mayoría de las aplicaciones científicas son escritas haciendo uso de la interfaz de paso de mensajes (MPI), que se ha convertido en un estándar de-facto de programación para las arquitecturas de memoria distribuida. Las aplicaciones paralelas que usan MPI pueden acceder a los sistemas de ficheros paralelos a través de la interfaz ofrecida por MPI-IO. Los patrones de acceso de las aplicaciones científicas paralelas consisten en un gran número de accesos pequeños y no contiguos. Para tamaños de acceso pequeños, el rendimiento viene limitado por la latencia de las transferencias de red y disco. Además, las aplicaciones científicas llevan a cabo accesos con una alta localidad espacial entre los distintos procesos en los nodos de E/S. Adicionalmente, las aplicaciones científicas presentan típicamente un comportamiento repetitivo. Cuando los patrones de acceso de E/S son repetitivos, técnicas como escritura demorada y lectura adelantada pueden enmascarar de forma eficiente las latencias de los accesos de E/S. Estas características han motivado a muchos investigadores en proponer optimizaciones de E/S tanto a nivel de biblioteca como a nivel del sistema de ficheros. Sin embargo, actualmente estas optimizaciones no se integran siempre a través de las distintas capas del sistema. El objetivo principal de esta tesis es proponer una nueva arquitectura genérica de E/S paralela para clusters y supercomputadores. Nuestra solución está basada en una arquitectura de caches en varias capas, una técnica de E/S colectiva y estrategias de acceso asíncronas que ocultan la latencia de transferencia de datos entre las distintas capas de caches. Nuestro diseño está dirigido a arquitecturas paralelas escalables con miles de nodos de cómputo. Además de actuar como middleware para los sistemas de ficheros paralelos existentes, nuestra arquitectura debe proporcionar virtualización on-line de los recursos de almacenamiento. Otro de los objeticos marcados para esta tesis es la factorización de las funcionalidades comunes en clusters y supercomputadores, en módulos genéricos que faciliten el despliegue de las aplicaciones científicas a través de estas plataformas. Se han desplegado distintos prototipos de nuestras soluciones tanto en clusters como en supercomputadores. Las evaluaciones de rendimiento demuestran que gracias a la combicación de las estratégias colectivas de E/S y del solapamiento de computación, comunicación y E/S, se puede obtener una sustancial mejora del rendimiento en los patrones de acceso anteriormente descritos, muy comunes en las aplicaciones paralelas de caracter científico

Peer-to-Peer Bartering: Swapping Amongst Self-interested Agents

Large--scale distributed environments can be seen as a conflict between the selfish aims of the participants and the group welfare of the population as a whole. In order to regulate the behavior of the participants it is often necessary to introduce mechanisms that provide incentives and stimulate cooperative behavior in order to mitigate for the resultant potentially undesirable availability outcomes which could arise from individual actions.The history of economics contains a wide variety of incentive patterns for cooperation. In this thesis, we adopt bartering incentive pattern as an attractive foundation for a simple and robust form of exchange to re-allocate resources. While bartering is arguably the world's oldest form of trade, there are still many instances where it surprises us. The success and survivability of the barter mechanisms adds to its attractiveness as a model to study.In this thesis we have derived three relevant scenarios where a bartering approach is applied. Starting from a common model of bartering: - We show the price to be paid for dealing with selfish agents in a bartering environment, as well as the impact on performance parameters such as topology and disclosed information.- We show how agents, by means of bartering, can achieve gains in goods without altruistic agents needing to be present.- We apply a bartering--based approach to a real application, the directory services.The core of this research is the analysis of bartering in the Internet Age. In previous times, usually economies dominated by bartering have suffered from high transaction costs (i.e. the improbability of the wants, needs that cause a transaction occurring at the same time and place). Nowadays, the world has a global system of interconnected computer networks called Internet. This interconnected world has the ability to overcome many challenges of the previous times. This thesis analysis the oldest system of trade within the context of this new paradigm. In this thesis we aim is to show thatbartering has a great potential, but there are many challenges that can affect the realistic application of bartering that should be studied.The purpose of this thesis has been to investigate resource allocation using bartering mechanism, with particular emphasis on applications in largescale distributed systems without the presence of altruistic participants in the environment.Throughout the research presented in this thesis we have contributed evidence that supports the leitmotif that best summarizes our work: investigation interactions amongst selfish, rational, and autonomous agents with incomplete information, each seeking to maximize its expected utility by means of bartering. We concentrate on three scenarios: one theoretical, a case of use, and finally a real application. All of these scenarios are used for evaluating bartering. Each scenario starts from a common origin, but each of them have their own unique features.The final conclusion is that bartering is still relevant in the modern world

Commemorating 50 Years (1967-2017) 50th Anniversary Celebratory Volume, Asian-Pacific Weed Science Society (APWSS); Indian Society of Weed Science (ISWS), India and The Weed Science Society of Japan (WSSJ)

The impetus for this 50th Anniversary Celebratory Volume of the Asian-Pacific Weed Science Society (APWSS) came from our firm conviction of the immense effort by the Society’s founding fathers, and those who followed in their footsteps, to nurture the discipline in a way beneficial to the people and cultures in the Asian-Pacific region. After 50 years of existence, there is reason for the success of this ‘interchange’ of knowledge and association of like-minded people, to be celebrated. In this Celebratory Volume, with contributions from several members, we have attempted to contextualize the contributions of the APWSS, in terms of its origin and development, as well as its activities, which are firmly rooted in promoting the understanding of weeds and responsibly managing weed impacts with appropriate methods..