8,130 research outputs found

    Disk-to-Disk Data Transfer using A Software Defined Networking Solution

    Get PDF
    There have been efforts towards improving the network performance using software defined net-working solutions. One such work is Steroid OpenFlow Service (SOS), which utilizes multiple parallel TCP connections to enhance the network performance transparently to the user. SOS has shown significant improvements in the memory-to-memory data transfer throughput; however, it’s perfor-mance for disk-to-disk data transfer hasn’t been studied. For computing applications involving big data, the data files are stored on non-volatile storage devices separate from the computing servers. Before computing can occur, large volumes of data must be fetched from the “remote” storage devices to the computing server’s local storage device. Since hard drives are the most commonly adopted storage devices today, the process is often called “disk-to-disk” data transfer. For production high performance computing facilities, specialized high throughput data transfer software will be provided for users to copy the data first to a data transfer node before copying to the computing server. Disk-to-Disk data transfer’s throughput performance depends on the network throughput be-tween servers and disk access performance between each server and its storage device. Due to large data sizes the storage devices are typically parallel file systems spanning multiple disks. Disk oper-ations in the disk-to-disk data transfer includes disk read and write operations. The read operation in the transfer is to read the data from the disks and store it in memory. The second step in the transfer is to send out the data to the network through the network interface. Data reaching the destination server is then stored to the disk. Data transfer is faced by multiple delays and is limited at each step of the transfer. To date, one commonly adopted data transfer solution is GridFTP developed by the Argonne National Laboratory. It requires custom application installations and configurations on the hosts. SOS, on the other hand, is a transparent network application without special user software. In this thesis, disk-to-disk data transfer performance is studied with both GridFTP and SOS. The thesis focuses on to two topics, one is the detailed analysis of transfer components for each tool and the second part consists of a systematic experiment to study the two. The experimentation and analysis of the results shows that configuring the data nodes and network with correct parameters results in maximum performance for disk-to-disk data transfer. The GridFTP, for example, is able to get to close to 7Gbps by using four parallel connections with TCP buffer size of 16MB. It achieves the maximum performance by filling the network pipe which has 10Gbps end-to-end link with round trip time (RTT) of 53ms

    Optimization and Management of Large-scale Scientific Workflows in Heterogeneous Network Environments: From Theory to Practice

    Get PDF
    Next-generation computation-intensive scientific applications feature large-scale computing workflows of various structures, which can be modeled as simple as linear pipelines or as complex as Directed Acyclic Graphs (DAGs). Supporting such computing workflows and optimizing their end-to-end network performance are crucial to the success of scientific collaborations that require fast system response, smooth data flow, and reliable distributed operation.We construct analytical cost models and formulate a class of workflow mapping problems with different mapping objectives and network constraints. The difficulty of these mapping problems essentially arises from the topological matching nature in the spatial domain, which is further compounded by the resource sharing complicacy in the temporal dimension. We provide detailed computational complexity analysis and design optimal or heuristic algorithms with rigorous correctness proof or performance analysis. We decentralize the proposed mapping algorithms and also investigate these optimization problems in unreliable network environments for fault tolerance.To examine and evaluate the performance of the workflow mapping algorithms before actual deployment and implementation, we implement a simulation program that simulates the execution dynamics of distributed computing workflows. We also develop a scientific workflow automation and management platform based on an existing workflow engine for experimentations in real environments. The performance superiority of the proposed mapping solutions are illustrated by extensive simulation-based comparisons with existing algorithms and further verified by large-scale experiments on real-life scientific workflow applications through effective system implementation and deployment in real networks

    Functional requirements document for the Earth Observing System Data and Information System (EOSDIS) Scientific Computing Facilities (SCF) of the NASA/MSFC Earth Science and Applications Division, 1992

    Get PDF
    Five scientists at MSFC/ESAD have EOS SCF investigator status. Each SCF has unique tasks which require the establishment of a computing facility dedicated to accomplishing those tasks. A SCF Working Group was established at ESAD with the charter of defining the computing requirements of the individual SCFs and recommending options for meeting these requirements. The primary goal of the working group was to determine which computing needs can be satisfied using either shared resources or separate but compatible resources, and which needs require unique individual resources. The requirements investigated included CPU-intensive vector and scalar processing, visualization, data storage, connectivity, and I/O peripherals. A review of computer industry directions and a market survey of computing hardware provided information regarding important industry standards and candidate computing platforms. It was determined that the total SCF computing requirements might be most effectively met using a hierarchy consisting of shared and individual resources. This hierarchy is composed of five major system types: (1) a supercomputer class vector processor; (2) a high-end scalar multiprocessor workstation; (3) a file server; (4) a few medium- to high-end visualization workstations; and (5) several low- to medium-range personal graphics workstations. Specific recommendations for meeting the needs of each of these types are presented

    Simulation of networks of spiking neurons: A review of tools and strategies

    Full text link
    We review different aspects of the simulation of spiking neural networks. We start by reviewing the different types of simulation strategies and algorithms that are currently implemented. We next review the precision of those simulation strategies, in particular in cases where plasticity depends on the exact timing of the spikes. We overview different simulators and simulation environments presently available (restricted to those freely available, open source and documented). For each simulation tool, its advantages and pitfalls are reviewed, with an aim to allow the reader to identify which simulator is appropriate for a given task. Finally, we provide a series of benchmark simulations of different types of networks of spiking neurons, including Hodgkin-Huxley type, integrate-and-fire models, interacting with current-based or conductance-based synapses, using clock-driven or event-driven integration strategies. The same set of models are implemented on the different simulators, and the codes are made available. The ultimate goal of this review is to provide a resource to facilitate identifying the appropriate integration strategy and simulation tool to use for a given modeling problem related to spiking neural networks.Comment: 49 pages, 24 figures, 1 table; review article, Journal of Computational Neuroscience, in press (2007

    Natural computing for vehicular networks

    Get PDF
    La presente tesis aborda el diseño inteligente de soluciones para el despliegue de redes vehiculares ad-hoc (vehicular ad hoc networks, VANETs). Estas son redes de comunicación inalámbrica formada principalmente por vehículos y elementos de infraestructura vial. Las VANETs ofrecen la oportunidad para desarrollar aplicaciones revolucionarias en el ámbito de la seguridad y eficiencia vial. Al ser un dominio tan novedoso, existe una serie de cuestiones abiertas, como el diseño de la infraestructura de estaciones base necesaria y el encaminamiento (routing) y difusión (broadcasting) de paquetes de datos, que todavía no han podido resolverse empleando estrategias clásicas. Es por tanto necesario crear y estudiar nuevas técnicas que permitan de forma eficiente, eficaz, robusta y flexible resolver dichos problemas. Este trabajo de tesis doctoral propone el uso de computación inspirada en la naturaleza o Computación Natural (CN) para tratar algunos de los problemas más importantes en el ámbito de las VANETs, porque representan una serie de algoritmos versátiles, flexibles y eficientes para resolver problemas complejos. Además de resolver los problemas VANET en los que nos enfocamos, se han realizado avances en el uso de estas técnicas para que traten estos problemas de forma más eficiente y eficaz. Por último, se han llevado a cabo pruebas reales de concepto empleando vehículos y dispositivos de comunicación reales en la ciudad de Málaga (España). La tesis se ha estructurado en cuatro grandes fases. En la primera fase, se han estudiado los principales fundamentos en los que se basa esta tesis. Para ello se hizo un estudio exhaustivo sobre las tecnologías que emplean las redes vehiculares, para así, identificar sus principales debilidades. A su vez, se ha profundizado en el análisis de la CN como herramienta eficiente para resolver problemas de optimización complejos, y de cómo utilizarla en la resolución de los problemas en VANETs. En la segunda fase, se han abordado cuatro problemas de optimización en redes vehiculares: la transferencia de archivos, el encaminamiento (routing) de paquetes, la difusión (broadcasting) de mensajes y el diseño de la infraestructura de estaciones base necesaria para desplegar redes vehiculares. Para la resolución de dichos problemas se han propuesto diferentes algoritmos CN que se clasifican en algoritmos evolutivos (evolutionary algorithms, EAs), métodos de inteligencia de enjambre (swarm intelligence, SI) y enfriamiento simulado (simulated annealing, SA). Los resultados obtenidos han proporcionado protocolos de han mejorado de forma significativa las comunicaciones en VANETs. En la tercera y última fase, se han realizado experimentos empleando vehículos reales circulando por las carreteras de Málaga y que se comunicaban entre sí. El principal objetivo de estas pruebas ha sido el validar las mejoras que presentan los protocolos que se han optimizado empleando CN. Los resultados obtenidos de las fases segunda y tercera confirman la hipótesis de trabajo, que la CN es una herramienta eficiente para tratar el diseño inteligente en redes vehiculares
    corecore