216 research outputs found

    Contribution à la convergence d'infrastructure entre le calcul haute performance et le traitement de données à large échelle

    Get PDF
    The amount of produced data, either in the scientific community or the commercialworld, is constantly growing. The field of Big Data has emerged to handle largeamounts of data on distributed computing infrastructures. High-Performance Computing (HPC) infrastructures are traditionally used for the execution of computeintensive workloads. However, the HPC community is also facing an increasingneed to process large amounts of data derived from high definition sensors andlarge physics apparati. The convergence of the two fields -HPC and Big Data- iscurrently taking place. In fact, the HPC community already uses Big Data tools,which are not always integrated correctly, especially at the level of the file systemand the Resource and Job Management System (RJMS).In order to understand how we can leverage HPC clusters for Big Data usage, andwhat are the challenges for the HPC infrastructures, we have studied multipleaspects of the convergence: We initially provide a survey on the software provisioning methods, with a focus on data-intensive applications. We contribute a newRJMS collaboration technique called BeBiDa which is based on 50 lines of codewhereas similar solutions use at least 1000 times more. We evaluate this mechanism on real conditions and in simulated environment with our simulator Batsim.Furthermore, we provide extensions to Batsim to support I/O, and showcase thedevelopments of a generic file system model along with a Big Data applicationmodel. This allows us to complement BeBiDa real conditions experiments withsimulations while enabling us to study file system dimensioning and trade-offs.All the experiments and analysis of this work have been done with reproducibilityin mind. Based on this experience, we propose to integrate the developmentworkflow and data analysis in the reproducibility mindset, and give feedback onour experiences with a list of best practices.RésuméLa quantité de données produites, que ce soit dans la communauté scientifiqueou commerciale, est en croissance constante. Le domaine du Big Data a émergéface au traitement de grandes quantités de données sur les infrastructures informatiques distribuées. Les infrastructures de calcul haute performance (HPC) sont traditionnellement utilisées pour l’exécution de charges de travail intensives en calcul. Cependant, la communauté HPC fait également face à un nombre croissant debesoin de traitement de grandes quantités de données dérivées de capteurs hautedéfinition et de grands appareils physique. La convergence des deux domaines-HPC et Big Data- est en cours. En fait, la communauté HPC utilise déjà des outilsBig Data, qui ne sont pas toujours correctement intégrés, en particulier au niveaudu système de fichiers ainsi que du système de gestion des ressources (RJMS).Afin de comprendre comment nous pouvons tirer parti des clusters HPC pourl’utilisation du Big Data, et quels sont les défis pour les infrastructures HPC, nousavons étudié plusieurs aspects de la convergence: nous avons d’abord proposé uneétude sur les méthodes de provisionnement logiciel, en mettant l’accent sur lesapplications utilisant beaucoup de données. Nous contribuons a l’état de l’art avecune nouvelle technique de collaboration entre RJMS appelée BeBiDa basée sur 50lignes de code alors que des solutions similaires en utilisent au moins 1000 fois plus.Nous évaluons ce mécanisme en conditions réelles et en environnement simuléavec notre simulateur Batsim. En outre, nous fournissons des extensions à Batsimpour prendre en charge les entrées/sorties et présentons le développements d’unmodèle de système de fichiers générique accompagné d’un modèle d’applicationBig Data. Cela nous permet de compléter les expériences en conditions réellesde BeBiDa en simulation tout en étudiant le dimensionnement et les différentscompromis autours des systèmes de fichiers.Toutes les expériences et analyses de ce travail ont été effectuées avec la reproductibilité à l’esprit. Sur la base de cette expérience, nous proposons d’intégrerle flux de travail du développement et de l’analyse des données dans l’esprit dela reproductibilité, et de donner un retour sur nos expériences avec une liste debonnes pratiques

    Proceedings of the First PhD Symposium on Sustainable Ultrascale Computing Systems (NESUS PhD 2016)

    Get PDF
    Proceedings of the First PhD Symposium on Sustainable Ultrascale Computing Systems (NESUS PhD 2016) Timisoara, Romania. February 8-11, 2016.The PhD Symposium was a very good opportunity for the young researchers to share information and knowledge, to present their current research, and to discuss topics with other students in order to look for synergies and common research topics. The idea was very successful and the assessment made by the PhD Student was very good. It also helped to achieve one of the major goals of the NESUS Action: to establish an open European research network targeting sustainable solutions for ultrascale computing aiming at cross fertilization among HPC, large scale distributed systems, and big data management, training, contributing to glue disparate researchers working across different areas and provide a meeting ground for researchers in these separate areas to exchange ideas, to identify synergies, and to pursue common activities in research topics such as sustainable software solutions (applications and system software stack), data management, energy efficiency, and resilience.European Cooperation in Science and Technology. COS

    The Role of Objects in Decision-Making Processes:The Case of an Energy Renovation

    Get PDF

    An Intelligent Robust Mouldable Scheduler for HPC & Elastic Environments

    Get PDF
    Traditional scheduling techniques are of a by-gone era and do not cater for the dynamism of new and emerging computing paradigms. Budget constraints now push researchers to migrate their workloads to public clouds or to buy into shared computing services as funding for large capital expenditures are few and far between. The sites still hosting large or shared computing infrastructure have to ensure that the system utilisation and efficiency is as high as ossible. However, the efficiency can not come at the cost of quality of service as the availability of public clouds now means that users can move away. This thesis presents a novel scheduling system to improve job turn-around-time. The Robust Mouldable Scheduler outlined in these pages utilises real application benchmarks to profile system performance and predict job execution times at different allocations, something no other scheduler does at present. The system is able to make an allocation decisions ensuring the jobs can fit into spaces available on the system using fewer resources without delaying the job completion time. The results demonstrate significant improvement in workload turn-around-times using real High Performance Computing (HPC) trace logs. Utilising three years of the University of Huddersfield trace logs the mouldable scheduler consistently simulated faster workload completion. Further, the results establish that by not relying on the user to suggest resource allocations for jobs the system is able to mitigate bad-put into the system leading to improved efficiency. A thorough investigation of Research Computing Systems (RCS), workload management systems, scheduling algorithms and strategies, benchmarking and profiling toolkits, and simulators is presented to establish the state of the art. Within this thesis a method to profile applications and workloads that leverages common open-source tools on HPC systems is presented. The resultant toolkit is used to profile the University of Huddersfield workload. This workload forms the basis to evaluate the mouldable scheduler. The research includes advance computing paradigms such as utilising Artificial Intelligence methods to improve the efficiency of the scheduler, or Surge Computing, where workloads are scaled beyond institutional firewalls through elastic compute systems

    Fourth Annual Workshop on Space Operations Applications and Research (SOAR 90)

    Get PDF
    The proceedings of the SOAR workshop are presented. The technical areas included are as follows: Automation and Robotics; Environmental Interactions; Human Factors; Intelligent Systems; and Life Sciences. NASA and Air Force programmatic overviews and panel sessions were also held in each technical area

    Scalability of a robotic inspection and repair system

    Get PDF
    Shift2Rail and In2Smart are two initiatives that will be part of the development of the necessary technologies to complete the Single European Railway Area (SERA). The target of this proposal is to accelerate the integration of new and advanced technologies into innovative rail product solutions. Shift2Rail has a robust framework to meet ambitious objectives. The most important is to double the capacity of the European rail system and increase its reliability and service quality by 50% while having life-cycle costs. In2Smart, as a project directed mainly of Network Rail, is measured in Technology Readiness Levels (TRL). These levels will indicate the maturity of technology for the application into the industry. The intention of this project is to reach a homogeneous TRL 3/4 demonstrator of a system capable to secure proper maintenance of rails, which is a Robotic Inspection and Repair System (RIRS). This research is focused on the scalability of the RIRS, taking into consideration the creation of a representative demonstrator that will authenticate the concept, the validation and verification of that demonstrator and finally the simulation of a scale-up system that will be more robust and will upgrade the TRL. This document contains the development of the control diagrams and schematics for the future incorporation of this control to a higher TRL prototype. The initial demonstrator consists of an autonomous railway vehicle equipped with a robotic arm that will scan the rails searching for faults and simulate a repairing process with a 3D printed polymer. The V&V of the physical demonstrator was a result of tests in the laboratory and the display of the demonstrator in several conferences and events.Manufacturin

    Surface engineering by titanium particulate injection mounding

    Get PDF
    In a recent study a structural hold down component was designed and produced using the particulate injection moulding (PIM) process. The material of choice was titanium due not only to the material properties but also due to the desire to create custom made components for a state-of-the-art marine vessel. On removal from the mould the green parts were seen to have an irregular surface on the top face. The irregular surface presented no through part defects and although the surface irregularities were caused by separation of the two-phases the effect was restricted to the outer surface of the parts. In a more historic study by the author the surface properties of titanium dental implants were modified by the use of adaptive mould inserts during the moulding phase of PIM. These two contrasting studies are considered and have become the basis of a current investigation looking to engineer surface irregularities in an ordered fashion. The application of meso-machining, and additive manufacture are considered and the functionality which may arise are presented

    Development of innovative cross-disciplinary engineering showcase

    Get PDF
    The development of engineering education relies substantially on interactive showcases and practical knowledge. The cross-disciplinary engineering showcase is designed to be fully interactive by having user input, producing a tangible output, and to understand distinct elements from each of the engineering disciplines such as, civil, mechanical and electrical (CME). The showcase operates from the input of mechanical rotational energy by the user pedalling the exercycle. Mechanical energy is then transferred to the pump via a gear train, which converts the user input of 30 rpm to the optimal pump operating speed of 2900 rpm. Further, it is used to pump water from the lower eservoir to the upper reservoir via one of the three flow paths, which the user can select by opening or closing flow valves. Once the water reaches a given height, it then flows back to the lower reservoir via a micro-hydro generator. As a result, it generates electrical energy stored in a power bank that can be used by the user to charge a digital device. Also, the showcase has a QR code to digital media, which will provide an additional explanation/exposition of the presented engineering principles to the user/students. The aim of this project is to develop a cross- disciplinary engineering showcase to enhance student learnings by interpreting the CME engineering principles in schools, institutes, and universities
    • …
    corecore