Scheduling for Large Scale Distributed Computing Systems: Approaches and Performance Evaluation Issues

Although our everyday life and society now depends heavily oncommunication infrastructures and computation infrastructures,scientists and engineers have always been among the main consumers ofcomputing power. This document provides a coherent overview of theresearch I have conducted in the last 15 years and which targets themanagement and performance evaluation of large scale distributedcomputing infrastructures such as clusters, grids, desktop grids,volunteer computing platforms, ... when used for scientific computing.In the first part of this document, I present how I have addressedscheduling problems arising on distributed platforms (like computinggrids) with a particular emphasis on heterogeneity and multi-userissues, hence in connection with game theory. Most of these problemsare relaxed from a classical combinatorial optimization formulationinto a continuous form, which allows to easily account for keyplatform characteristics such as heterogeneity or complex topologywhile providing efficient practical and distributed solutions.The second part presents my main contributions to the SimGrid project,which is a simulation toolkit for building simulators of distributedapplications (originally designed for scheduling algorithm evaluationpurposes). It comprises a unified presentation of how the questions ofvalidation and scalability have been addressed in SimGrid as well asthoughts on specific challenges related to methodological aspects andto the application of SimGrid to the HPC context

Wide-Area Time-Synchronized Closed-Loop Control of Power Systems And Decentralized Active Distribution Networks

The rapidly expanding power system grid infrastructure and the need to reduce the occurrence of major blackouts and prevention or hardening of systems against cyber-attacks, have led to increased interest in the improved resilience of the electrical grid. Distributed and decentralized control have been widely applied to computer science research. However, for power system applications, the real-time application of decentralized and distributed control algorithms introduce several challenges. In this dissertation, new algorithms and methods for decentralized control, protection and energy management of Wide Area Monitoring, Protection and Control (WAMPAC) and the Active Distribution Network (ADN) are developed to improve the resiliency of the power system. To evaluate the findings of this dissertation, a laboratory-scale integrated Wide WAMPAC and ADN control platform was designed and implemented. The developed platform consists of phasor measurement units (PMU), intelligent electronic devices (IED) and programmable logic controllers (PLC). On top of the designed hardware control platform, a multi-agent cyber-physical interoperability viii framework was developed for real-time verification of the developed decentralized and distributed algorithms using local wireless and Internet-based cloud communication. A novel real-time multiagent system interoperability testbed was developed to enable utility independent private microgrids standardized interoperability framework and define behavioral models for expandability and plug-and-play operation. The state-of-theart power system multiagent framework is improved by providing specific attributes and a deliberative behavior modeling capability. The proposed multi-agent framework is validated in a laboratory based testbed involving developed intelligent electronic device prototypes and actual microgrid setups. Experimental results are demonstrated for both decentralized and distributed control approaches. A new adaptive real-time protection and remedial action scheme (RAS) method using agent-based distributed communication was developed for autonomous hybrid AC/DC microgrids to increase resiliency and continuous operability after fault conditions. Unlike the conventional consecutive time delay-based overcurrent protection schemes, the developed technique defines a selectivity mechanism considering the RAS of the microgrid after fault instant based on feeder characteristics and the location of the IEDs. The experimental results showed a significant improvement in terms of resiliency of microgrids through protection using agent-based distributed communication

Distributed, Low-Cost, Non-Expert Fine Dust Sensing with Smartphones

Diese Dissertation behandelt die Frage, wie mit kostengünstiger Sensorik Feinstäube in hoher zeitlicher und räumlicher Auflösung gemessen werden können. Dazu wird ein neues Sensorsystem auf Basis kostengünstiger off-the-shelf-Sensoren und Smartphones vorgestellt, entsprechende robuste Algorithmen zur Signalverarbeitung entwickelt und Erkenntnisse zur Interaktions-Gestaltung für die Messung durch Laien präsentiert. Atmosphärische Aerosolpartikel stellen im globalen Maßstab ein gravierendes Problem für die menschliche Gesundheit dar, welches sich in Atemwegs- und Herz-Kreislauf-Erkrankungen äußert und eine Verkürzung der Lebenserwartung verursacht. Bisher wird Luftqualität ausschließlich anhand von Daten relativ weniger fester Messstellen beurteilt und mittels Modellen auf eine hohe räumliche Auflösung gebracht, so dass deren Repräsentativität für die flächendeckende Exposition der Bevölkerung ungeklärt bleibt. Es ist unmöglich, derartige räumliche Abbildungen mit den derzeitigen statischen Messnetzen zu bestimmen. Bei der gesundheitsbezogenen Bewertung von Schadstoffen geht der Trend daher stark zu räumlich differenzierenden Messungen. Ein vielversprechender Ansatz um eine hohe räumliche und zeitliche Abdeckung zu erreichen ist dabei Participatory Sensing, also die verteilte Messung durch Endanwender unter Zuhilfenahme ihrer persönlichen Endgeräte. Insbesondere für Luftqualitätsmessungen ergeben sich dabei eine Reihe von Herausforderungen - von neuer Sensorik, die kostengünstig und tragbar ist, über robuste Algorithmen zur Signalauswertung und Kalibrierung bis hin zu Anwendungen, die Laien bei der korrekten Ausführung von Messungen unterstützen und ihre Privatsphäre schützen. Diese Arbeit konzentriert sich auf das Anwendungsszenario Partizipatorischer Umweltmessungen, bei denen Smartphone-basierte Sensorik zum Messen der Umwelt eingesetzt wird und üblicherweise Laien die Messungen in relativ unkontrollierter Art und Weise ausführen. Die Hauptbeiträge hierzu sind: 1. Systeme zum Erfassen von Feinstaub mit Smartphones (Low-cost Sensorik und neue Hardware): Ausgehend von früher Forschung zur Feinstaubmessung mit kostengünstiger off-the-shelf-Sensorik wurde ein Sensorkonzept entwickelt, bei dem die Feinstaub-Messung mit Hilfe eines passiven Aufsatzes auf einer Smartphone-Kamera durchgeführt wird. Zur Beurteilung der Sensorperformance wurden teilweise Labor-Messungen mit künstlich erzeugtem Staub und teilweise Feldevaluationen in Ko-Lokation mit offiziellen Messstationen des Landes durchgeführt. 2. Algorithmen zur Signalverarbeitung und Auswertung: Im Zuge neuer Sensordesigns werden Kombinationen bekannter OpenCV-Bildverarbeitungsalgorithmen (Background-Subtraction, Contour Detection etc.) zur Bildanalyse eingesetzt. Der resultierende Algorithmus erlaubt im Gegensatz zur Auswertung von Lichtstreuungs-Summensignalen die direkte Zählung von Partikeln anhand individueller Lichtspuren. Ein zweiter neuartiger Algorithmus nutzt aus, dass es bei solchen Prozessen ein signalabhängiges Rauschen gibt, dessen Verhältnis zum Mittelwert des Signals bekannt ist. Dadurch wird es möglich, Signale die von systematischen unbekannten Fehlern betroffen sind auf Basis ihres Rauschens zu analysieren und das "echte" Signal zu rekonstruieren. 3. Algorithmen zur verteilten Kalibrierung bei gleichzeitigem Schutz der Privatsphäre: Eine Herausforderung partizipatorischer Umweltmessungen ist die wiederkehrende Notwendigkeit der Sensorkalibrierung. Dies beruht zum einen auf der Instabilität insbesondere kostengünstiger Luftqualitätssensorik und zum anderen auf der Problematik, dass Endbenutzern die Mittel für eine Kalibrierung üblicherweise fehlen. Bestehende Ansätze zur sogenannten Cross-Kalibrierung von Sensoren, die sich in Ko-Lokation mit einer Referenzstation oder anderen Sensoren befinden, wurden auf Daten günstiger Feinstaubsensorik angewendet sowie um Mechanismen erweitert, die eine Kalibrierung von Sensoren untereinander ohne Preisgabe privater Informationen (Identität, Ort) ermöglicht. 4. Mensch-Maschine-Interaktions-Gestaltungsrichtlinien für Participatory Sensing: Auf Basis mehrerer kleiner explorativer Nutzerstudien wurde empirisch eine Taxonomie der Fehler erstellt, die Laien beim Messen von Umweltinformationen mit Smartphones machen. Davon ausgehend wurden mögliche Gegenmaßnahmen gesammelt und klassifiziert. In einer großen summativen Studie mit einer hohen Teilnehmerzahl wurde der Effekt verschiedener dieser Maßnahmen durch den Vergleich vier unterschiedlicher Varianten einer App zur partizipatorischen Messung von Umgebungslautstärke evaluiert. Die dabei gefundenen Erkenntnisse bilden die Basis für Richtlinien zur Gestaltung effizienter Nutzerschnittstellen für Participatory Sensing auf Mobilgeräten. 5. Design Patterns für Participatory Sensing Games auf Mobilgeräten (Gamification): Ein weiterer erforschter Ansatz beschäftigt sich mit der Gamifizierung des Messprozesses um Nutzerfehler durch den Einsatz geeigneter Spielmechanismen zu minimieren. Dabei wird der Messprozess z.B. in ein Smartphone-Spiel (sog. Minigame) eingebettet, das im Hintergrund bei geeignetem Kontext die Messung durchführt. Zur Entwicklung dieses "Sensified Gaming" getauften Konzepts wurden Kernaufgaben im Participatory Sensing identifiziert und mit aus der Literatur zu sammelnden Spielmechanismen (Game Design Patterns) gegenübergestellt

Parallel and Distributed Computing

The 14 chapters presented in this book cover a wide variety of representative works ranging from hardware design to application development. Particularly, the topics that are addressed are programmable and reconfigurable devices and systems, dependability of GPUs (General Purpose Units), network topologies, cache coherence protocols, resource allocation, scheduling algorithms, peertopeer networks, largescale network simulation, and parallel routines and algorithms. In this way, the articles included in this book constitute an excellent reference for engineers and researchers who have particular interests in each of these topics in parallel and distributed computing

Delivering IoT Services in Smart Cities and Environmental Monitoring through Collective Awareness, Mobile Crowdsensing and Open Data

The Internet of Things (IoT) is the paradigm that allows us to interact with the real world by means of networking-enabled devices and convert physical phenomena into valuable digital knowledge. Such a rapidly evolving field leveraged the explosion of a number of technologies, standards and platforms. Consequently, different IoT ecosystems behave as closed islands and do not interoperate with each other, thus the potential of the number of connected objects in the world is far from being totally unleashed. Typically, research efforts in tackling such challenge tend to propose a new IoT platforms or standards, however, such solutions find obstacles in keeping up the pace at which the field is evolving. Our work is different, in that it originates from the following observation: in use cases that depend on common phenomena such as Smart Cities or environmental monitoring a lot of useful data for applications is already in place somewhere or devices capable of collecting such data are already deployed. For such scenarios, we propose and study the use of Collective Awareness Paradigms (CAP), which offload data collection to a crowd of participants. We bring three main contributions: we study the feasibility of using Open Data coming from heterogeneous sources, focusing particularly on crowdsourced and user-contributed data that has the drawback of being incomplete and we then propose a State-of-the-Art algorith that automatically classifies raw crowdsourced sensor data; we design a data collection framework that uses Mobile Crowdsensing (MCS) and puts the participants and the stakeholders in a coordinated interaction together with a distributed data collection algorithm that prevents the users from collecting too much or too less data; (3) we design a Service Oriented Architecture that constitutes a unique interface to the raw data collected through CAPs through their aggregation into ad-hoc services, moreover, we provide a prototype implementation

Communities in (Digital) Space: Creating Networks for Daily Living Through Pervasive Media

Studies of online communities often focus either on communities that produce texts or the texts with which individuals engage. This dissertation examines online communities that practice in ongoing activities, in their leisure time, often with no end goal of producing any final text. Through interviews, surveys, and community forum analysis of running, gaming, and translation communities, this study finds that place and everyday habits factor heavily into the ways that sustained online communities structure their work. “Place” can have several meanings within this context, including the communities valuing specific locations or working with specific individuals because of where they live. Due to the rise in use of pervasive mobile devices, online community access often weaves into members’ offline lives. This knowledge of life ancillary to online community adds a layer of affective work to online community participation. Throughout the data collected from these communities, stories pertaining to the work of community maintenance dominated the conversation. Participants defined “work” as managing community involvement around other obligations, maintaining relationships across distances, and acknowledging the benefits that corporate entities derive from these communities. By investigating work within this context, we expand our understanding of the ways less visible populations work online in their leisure time

Software curating : the politics of curating in/as (an) Open System(s)

The thesis examines how Information technologies have changed the practice of curating. It proposes an Interdisciplinary approach that directly links curating (often understood as an activity of artistic programming), computing (the activity of computer programming) and a relatively recent Interest In software art (in which programming Is understood as artistic practice). Although there Is much contemporary critical work and practice that Is described as art-oriented programming or software art, the thesis aims to explore a perceived gap In discussions around software curating. Curators working with online technologies are presented with the challenge of how to respond to new artistic forms that Involve programming: for Instance program-objects that display dynamic and transformative properties, and that are distributed over socio-technological networks. Although there are many examples of social platforms and highly relevant examples of online 'art platforms', these still largely operate In display mode replicating more conventional models of curating and the operations of art Institutions In general. The tendency Is for these curatorial online systems to concentrate on the display of executed code and pay less attention to source code. New sensibilities are required that simultaneously reflect the significance of source code as art, and software not as a production tool or a display platform but as cultural practice that Is analogous to curating. What Is distinctive about the thesis Is that It speculates on a curatorial model that emphasises the analogy to programming. Consequently, the thesis argues for online software systems that display properties of curating but reprocess established definitions by deliberately collapsing firm distinctions between the fields of programming, artistic practice and curatorial practice. To consider these Issues, the thesis brings together a number of Inter-related fields of critical Inquiry and situates curating In the context of theories of immateriality, a critical discourse around software art practice, and an understanding of open systems. The key Issue for the thesis becomes how power relations, control and agency are expressed In new curatorial forms that Involve programming and networks; In other words, the thesis Is concerned with the politics of curating In/as (an) open system(s). Indeed, curating Itself can be described In terms of open systems, Implying a state In which there Is continuous Interaction with the soclo-technological environment. The system Is opened up to communicative processes that Involve producers/users and to divergent exchanges that take place and that disrupt established social relations of production and distribution. Thus, and Importantly for an understanding of the power relations Involved, software opens up curating to dynamic possibilities and transformations beyond the usual Institutional model (analogous to the model of production associated with the industrial factory) Into the context of networks (and what Is referred to by the Autonomists as the 'social factory'). The suggestion Is that the curatorial process Is now closely Integrated with the dynamic soclo-technological networks and with software that Is not simply used to curate but demonstrates the activity of curatIng In Itself Consequently, the thesis offers an expanded description of curating with respect to software In which agency Is reconstituted to Include alternative dynamics of networks. The curatorial model Is not only theorlsed but also deployed In the production of experimental software for curating source code (kurator) that forms the practical part of the doctoral research. in addition to a written thesis and software, two further projects produced during the registration period 2002-2008 are Included in support of the overall thesis: a conference CuratIng, Immaterlafity, Systems (CIS) (Tate Modern, London 2005) and an edited book Curating immateriality: The Work of The Curator In the Age of Network Systems (CI) (Autonomedia, New York 2006). The kurator software Is a further development of the conference and subsequent book, and offers an online, user-moderated curatorial system for further public modification. In so doing, the argument Is that the curatorial process Is demonstrably a collective and distributed executable that displays machinic agency. This Is what Is referred to in the thesis as software curating.Faculty of Technology and Faculty of Arts, University of Plymouth

A vision-based optical character recognition system for real-time identification of tractors in a port container terminal

Automation has been seen as a promising solution to increase the productivity of modern sea port container terminals. The potential of increase in throughput, work efficiency and reduction of labor cost have lured stick holders to strive for the introduction of automation in the overall terminal operation. A specific container handling process that is readily amenable to automation is the deployment and control of gantry cranes in the container yard of a container terminal where typical operations of truck identification, loading and unloading containers, and job management are primarily performed manually in a typical terminal. To facilitate the overall automation of the gantry crane operation, we devised an approach for the real-time identification of tractors through the recognition of the corresponding number plates that are located on top of the tractor cabin. With this crucial piece of information, remote or automated yard operations can then be performed. A machine vision-based system is introduced whereby these number plates are read and identified in real-time while the tractors are operating in the terminal. In this paper, we present the design and implementation of the system and highlight the major difficulties encountered including the recognition of character information printed on the number plates due to poor image integrity. Working solutions are proposed to address these problems which are incorporated in the overall identification system.postprin

E-commerce protocol supporting automated online dispute resolution

E-commerce now constitutes a significant part of all commercial activity; however the increase in transactions is also leading to more disputes. These disputes are becoming more frequent, more technologically complicated and more difficult in terms of traceability . This thesis focuses specifically on dispute problems related to soft products, i.e. those that are intangible and therefore requiring no physical delivery. With the growing demand for these types of products, e.g. downloadable films, music, software, and prepaid calling time, the prevention of fraudulent transactions is becoming increasingly important. Reasons for the rise in the number of fraudulent transactions include merchants being unable to see the customer to verify an ID or signature and E-commerce enabling soft-products and services to be acquired via soft delivery methods: email, download or logging in. The introductory section provides a critique of current e-commerce fraud detection and prevention techniques and shows that not all are suitable for e-commerce, especially soft-products, and therefore unable to provide complete protection against fraud. The future relating to the detection and prevention of e-commerce fraud is then discussed, leading to suggestions regarding the improvement of the current state-of-the-art technique, the Address Verification Service (AVS), which is used to accommodate the introduction of soft-products. Apart from the exchange process problems, i.e. those involving money and goods, attention is also paid to other important factors such as timing and quality that are usually neglected in these detection and prevention techniques. Dispute scenarios from many different perspectives have been analysed, viz. computer science, business, legal and that of the participants themselves. From the analyses, all possible dispute cases have been formally listed using the 'Truth Table' approach. This analysis has then led to the design of a comprehensive taxonomy framework for dispute in e-commerce. The term Online Dispute Resolution (ODR), is the online technology applied to Alternative Dispute Resolution (ADR) which is resolving disputes other than via litigation in the courts. Current ODR systems and their suitability for the e-commercial world have been examined, concluding that not all are appropriate for e-commerce situations (since most still involve a human element and often make the resolution process more costly than the actual item under dispute). The proposed solution to the problem is by automating the online dispute resolution process. The total solution is described in two parts (i) an E-commerce Transaction Protocol (ETP) forming the infrastructure where the transaction will take place and be able to accommodate any new improvements in the future, and (ii) an Automated Online Dispute Resolution (AODR) system which should automatically resolve any dispute occurring within the proposed e-commerce model. In order for the AODR to resolve any dispute, a product/payment specific plug-in (add-on) has been incorporated into the system. For illustration purposes, credit cards as a payment method has been selected and the appropriate plug-in specification for soft products and credit cards created. The concept of providing every soft product with a quality certificate has also been discussed. A concluding case study of e-commerce in Saudi Arabia has been used to test the viability of both the e-commerce dispute taxonomy and the proposed model. The case study shows the suitability of using ETP with AODR in order to resolve soft-product disputes automatically. Limitations of the work and further research possibilities have then been identified.EThOS - Electronic Theses Online ServiceDepartment of Computing Science, Newcastle UniversityGBUnited Kingdo