Efficient HTTP based I/O on very large datasets for high performance computing with the libdavix library

Remote data access for data analysis in high performance computing is commonly done with specialized data access protocols and storage systems. These protocols are highly optimized for high throughput on very large datasets, multi-streams, high availability, low latency and efficient parallel I/O. The purpose of this paper is to describe how we have adapted a generic protocol, the Hyper Text Transport Protocol (HTTP) to make it a competitive alternative for high performance I/O and data analysis applications in a global computing grid: the Worldwide LHC Computing Grid. In this work, we first analyze the design differences between the HTTP protocol and the most common high performance I/O protocols, pointing out the main performance weaknesses of HTTP. Then, we describe in detail how we solved these issues. Our solutions have been implemented in a toolkit called davix, available through several recent Linux distributions. Finally, we describe the results of our benchmarks where we compare the performance of davix against a HPC specific protocol for a data analysis use case.Comment: Presented at: Very large Data Bases (VLDB) 2014, Hangzho

Building distributed heterogeneous smart phone Java applications an evaluation from a development perspective

The advances in mobile phone technology have enabled such devices to be programmed to run general-purpose applications using a special edition of the Java programming language. Java is designed to be a heterogeneous programming language targeting different platforms. Such ability when coupled with the provision of high-speed mobile Internet access would open the door for a new breed of distributed mobile applications. This paper explores the capabilities and limitations of this technology and addresses the considerations that must be taken when designing and developing such distributed applications. Our findings are verified by building a test client-server system where the clients in this system are mobile phones behaving as active processing elements not just mere service requesters

Cloud Storage Performance and Security Analysis with Hadoop and GridFTP

Even though cloud server has been around for a few years, most of the web hosts today have not converted to cloud yet. If the purpose of the cloud server is distributing and storing files on the internet, FTP servers were much earlier than the cloud. FTP server is sufficient to distribute content on the internet. Therefore, is it worth to shift from FTP server to cloud server? The cloud storage provider declares high durability and availability for their users, and the ability to scale up for more storage space easily could save users tons of money. However, does it provide higher performance and better security features? Hadoop is a very popular platform for cloud computing. It is free software under Apache License. It is written in Java and supports large data processing in a distributed environment. Characteristics of Hadoop include partitioning of data, computing across thousands of hosts, and executing application computations in parallel. Hadoop Distributed File System allows rapid data transfer up to thousands of terabytes, and is capable of operating even in the case of node failure. GridFTP supports high-speed data transfer for wide-area networks. It is based on the FTP and features multiple data channels for parallel transfers. This report describes the technology behind HDFS and enhancement to the Hadoop security features with Kerberos. Based on data transfer performance and security features of HDFS and GridFTP server, we can decide if we should replace GridFTP server with HDFS. According to our experiment result, we conclude that GridFTP server provides better throughput than HDFS, and Kerberos has minimal impact to HDFS performance. We proposed a solution which users authenticate with HDFS first, and get the file from HDFS server to the client using GridFTP

DIVERSE: a Software Toolkit to Integrate Distributed Simulations with Heterogeneous Virtual Environments

We present DIVERSE (Device Independent Virtual Environments- Reconfigurable, Scalable, Extensible), which is a modular collection of complimentary software packages that we have developed to facilitate the creation of distributed operator-in-the-loop simulations. In DIVERSE we introduce a novel implementation of remote shared memory (distributed shared memory) that uses Internet Protocol (IP) networks. We also introduce a new method that automatically extends hardware drivers (not in the operating system kernel driver sense) into inter-process and Internet hardware services. Using DIVERSE, a program can display in a CAVE™, ImmersaDesk™, head mounted display (HMD), desktop or laptop without modification. We have developed a method of configuring user programs at run-time by loading dynamic shared objects (DSOs), in contrast to the more common practice of creating interpreted configuration languages. We find that by loading DSOs the development time, complexity and size of DIVERSE and DIVERSE user applications is significantly reduced. Configurations to support different I/O devices, device emulators, visual displays, and any component of a user application including interaction techniques, can be changed at run-time by loading different sets of DIVERSE DSOs. In addition, interpreted run-time configuration parsers have been implemented using DIVERSE DSOs; new ones can be created as needed. DIVERSE is free software, licensed under the terms of the GNU General Public License (GPL) and the GNU Lesser General Public License (LGPL) licenses. We describe the DIVERSE architecture and demonstrate how DIVERSE was used in the development of a specific application, an operator-in-the-loop Navy ship-board crane simulator, which runs unmodified on a desktop computer and/or in a CAVE with motion base motion queuing

Polynomial Heuristic Algorithms for Inconsistency Characterization in Firewall Rule Sets

Firewalls provide the first line of defence of nearly all networked institutions today. However, Firewall ACLs could have inconsistencies, allowing traffic that should be denied or vice versa. In this paper, we analyze the inconsistency characterization problem as a separate problem of the diagnosis one, and propose formal definitions in order to characterize one-to-many inconsistencies. We identify the combinatorial part of the problem that generates exponential complexities in combined diagnosis and characterization algorithms proposed by other authors. Then we propose a decomposition of the combinatorial problem in several smaller combinatorial ones, which can effectively reduce the complexity of the problem. Finally, we propose an approximate heuristic and algorithms to solve the problem in worst case polynomial time. Although many algorithms have been proposed to address this problem, all of them are combinatorial. The presented algorithms are an heuristic way to solve the problem with polynomial complexity. There are no constraints on how rule field ranges are expressed.Ministerio de Educación y Ciencia DPI2006-15476-C02-0

MpTCP1 controls cell proliferation and redox processes in Marchantia polymorpha

TCP transcription factors are key regulators of angiosperm cell proliferation processes. It is unknown whether their regulatory growth capacities are conserved across land plants, which we examined in liverworts, one of the earliest diverging land plant lineages. We generated knockout mutants for MpTCP1, the single TCP‐P clade gene in Marchantia polymorpha, and characterized its function conducting cell proliferation and morphological analyses as well as mRNA expression, transcriptome, chemical and DNA binding studies. Mptcp1ge lines show a reduced vegetative thallus growth and extra tissue formation in female reproductive structures. Additionally, mutant plants reveal increased H2O2 levels and an enhanced pigmentation in the thallus caused by formation of secondary metabolites, such as aminochromes. MpTCP1 proteins interact redox‐dependently with DNA and regulate the expression of a comprehensive redox network, comprising enzymes involved in H2O2 metabolism. MpTCP1 regulates Marchantia growth context‐dependently. Redox sensitivity of the DNA binding capacity of MpTCP1 proteins provides a mechanism to respond to altered redox conditions. Our data suggest that MpTCP1 activity could thereby have contributed to diversification of land plant morphologies and to adaptations to abiotic and biotic challenges, experienced by liverworts during early land plant colonization