DRIVER Technology Watch Report

This report is part of the Discovery Workpackage (WP4) and is the third report out of four deliverables. The objective of this report is to give an overview of the latest technical developments in the world of digital repositories, digital libraries and beyond, in order to serve as theoretical and practical input for the technical DRIVER developments, especially those focused on enhanced publications. This report consists of two main parts, one part focuses on interoperability standards for enhanced publications, the other part consists of three subchapters, which give a landscape picture of current and surfacing technologies and communities crucial to DRIVER. These three subchapters contain the GRID, CRIS and LTP communities and technologies. Every chapter contains a theoretical explanation, followed by case studies and the outcomes and opportunities for DRIVER in this field

Topological tools for understanding complex systems

The behavior of complex systems is often influenced by their structure. In mathematics, the field of algebraic topology has been especially useful for characterizing mathematical structures. Topological data analysis (TDA) is a growing field in which methods from algebraic topology are applied to studying the structure of data. TDA has been used in a variety of applications, including biological data, granular materials, and demography. Social interactions are heavily informed by space and have complex structure due to patterns in the way humans arrange themselves geographically. Consequently, social applications can benefit from the application of TDA.In this dissertation, I develop topological methods for studying spatial networks and apply them to a wide variety of data sets. In particular, I study methods for building topological spaces (specifically, simplicial complexes) based on data. I present two novel simplicial-complex constructions, the adjacency complex and the level-set complex, for spatial data. I apply both constructions to random networks, cities, voting, and scientific images, gaining insights into the structure of these systems. I also propose a novel simplicial complex construction for studying patterns of neighborhood formation based on combining demographic and spatial data. I present case studies in neighborhood segregation for two U.S. cities. In addition to my topological research, I discuss two projects in the study of social systems using methods from network analysis. I present an extension to multilayer networks of the Hegselmann--Krause model for opinion dynamics and discuss preliminary findings on its convergence properties. I also present a framework for estimating homelessness underreporting in California Local Education agencies (LEAs)

Programming and parallelising applications for distributed infrastructures

The last decade has witnessed unprecedented changes in parallel and distributed infrastructures. Due to the diminished gains in processor performance from increasing clock frequency, manufacturers have moved from uniprocessor architectures to multicores; as a result, clusters of computers have incorporated such new CPU designs. Furthermore, the ever-growing need of scienti c applications for computing and storage capabilities has motivated the appearance of grids: geographically-distributed, multi-domain infrastructures based on sharing of resources to accomplish large and complex tasks. More recently, clouds have emerged by combining virtualisation technologies, service-orientation and business models to deliver IT resources on demand over the Internet. The size and complexity of these new infrastructures poses a challenge for programmers to exploit them. On the one hand, some of the di culties are inherent to concurrent and distributed programming themselves, e.g. dealing with thread creation and synchronisation, messaging, data partitioning and transfer, etc. On the other hand, other issues are related to the singularities of each scenario, like the heterogeneity of Grid middleware and resources or the risk of vendor lock-in when writing an application for a particular Cloud provider. In the face of such a challenge, programming productivity - understood as a tradeo between programmability and performance - has become crucial for software developers. There is a strong need for high-productivity programming models and languages, which should provide simple means for writing parallel and distributed applications that can run on current infrastructures without sacri cing performance. In that sense, this thesis contributes with Java StarSs, a programming model and runtime system for developing and parallelising Java applications on distributed infrastructures. The model has two key features: first, the user programs in a fully-sequential standard-Java fashion - no parallel construct, API call or pragma must be included in the application code; second, it is completely infrastructure-unaware, i.e. programs do not contain any details about deployment or resource management, so that the same application can run in di erent infrastructures with no changes. The only requirement for the user is to select the application tasks, which are the model's unit of parallelism. Tasks can be either regular Java methods or web service operations, and they can handle any data type supported by the Java language, namely les, objects, arrays and primitives. For the sake of simplicity of the model, Java StarSs shifts the burden of parallelisation from the programmer to the runtime system. The runtime is responsible from modifying the original application to make it create asynchronous tasks and synchronise data accesses from the main program. Moreover, the implicit inter-task concurrency is automatically found as the application executes, thanks to a data dependency detection mechanism that integrates all the Java data types. This thesis provides a fairly comprehensive evaluation of Java StarSs on three di erent distributed scenarios: Grid, Cluster and Cloud. For each of them, a runtime system was designed and implemented to exploit their particular characteristics as well as to address their issues, while keeping the infrastructure unawareness of the programming model. The evaluation compares Java StarSs against state-of-the-art solutions, both in terms of programmability and performance, and demonstrates how the model can bring remarkable productivity to programmers of parallel distributed applications

Accounting Facilities in the European Supercomputing Grid DEISA

Account management and resource usage monitoring are essential services for production Grids. The scope of a production Grid infrastructure, the heterogeneity of resources and services, the typical community usage profiles, and the depth of integration of the resource providers regarding operational procedures and policies imply specific requirements for accounting facilities. We present the accounting facilities currently used in production in the Distributed European Infra-structure for the Supercomputing Applications (DEISA). DEISA is a consortium of leading national supercomputing centres currently deploying and operating a persistent, production quality, distributed su-percomputing environment with continental scope. The DEISA accounting facilities gather information from the site-local batch systems and the distributed DEISA user administration system, and generate XML usage records conforming to the OGF usage record specification which are then stored locally in a XML data base at each DEISA site. The distributed accounting information can be fetched by clients such as users, project supervisors, site accounting managers and DEISA supervisors. The information is made available by site-local WSRF-compliant accounting information services that allow for a fine-grained setting of access rights. Each authorized client gets a specific view on the accounting information according to one of the following roles: a) a site accounting manager imports usage records of related home-site users from all DEISA sites for longterm archiving, b) a project supervisor retrieves information to assess the resource usage by his project partners, c) a DEISA supervisor (e.g. someone overlooking the usage on behalf of the DEISA executive committee) gets a report on the global usage of DEISA resources, and d) the user who can retrieve all the accounting information related to his own jobs. The privacy and integrity of the data provided and transferred from the accounting information service running at each site is guaranteed using X.509 certificates for mutual authentication and secure communication channels

An SOA-based model for the integrated provisioning of cloud and grid resources

In the last years, the availability and models of use of networked computing resources within reach of e-Science are rapidly changing and see the coexistence of many disparate paradigms: high-performance computing, grid, and recently cloud. Unfortunately, none of these paradigms is recognized as the ultimate solution, and a convergence of them all should be pursued. At the same time, recent works have proposed a number of models and tools to address the growing needs and expectations in the field of e-Science. In particular, they have shown the advantages and the feasibility of modeling e-Science environments and infrastructures according to the service-oriented architecture. In this paper, we suggest a model to promote the convergence and the integration of the different computing paradigms and infrastructures for the dynamic on-demand provisioning of resources from multiple providers as a cohesive aggregate, leveraging the service-oriented architecture. In addition, we propose a design aimed at endorsing a flexible, modular, workflow-based computing model for e-Science. The model is supplemented by a working prototype implementation together with a case study in the applicative domain of bioinformatics, which is used to validate the presented approach and to carry out some performance and scalability measurements

Polyhedral+Dataflow Graphs

This research presents an intermediate compiler representation that is designed for optimization, and emphasizes the temporary storage requirements and execution schedule of a given computation to guide optimization decisions. The representation is expressed as a dataflow graph that describes computational statements and data mappings within the polyhedral compilation model. The targeted applications include both the regular and irregular scientific domains. The intermediate representation can be integrated into existing compiler infrastructures. A specification language implemented as a domain specific language in C++ describes the graph components and the transformations that can be applied. The visual representation allows users to reason about optimizations. Graph variants can be translated into source code or other representation. The language, intermediate representation, and associated transformations have been applied to improve the performance of differential equation solvers, or sparse matrix operations, tensor decomposition, and structured multigrid methods