A Policy-Based Resource Brokering Environment for Computational Grids

With the advances in networking infrastructure in general, and the Internet in particular, we can build grid environments that allow users to utilize a diverse set of distributed and heterogeneous resources. Since the focus of such environments is the efficient usage of the underlying resources, a critical component is the resource brokering environment that mediates the discovery, access and usage of these resources. With the consumer\u27s constraints, provider\u27s rules, distributed heterogeneous resources and the large number of scheduling choices, the resource brokering environment needs to decide where to place the user\u27s jobs and when to start their execution in a way that yields the best performance for the user and the best utilization for the resource provider. As brokering and scheduling are very complicated tasks, most current resource brokering environments are either specific to a particular grid environment or have limited features. This makes them unsuitable for large applications with heterogeneous requirements. In addition, most of these resource brokering environments lack flexibility. Policies at the resource-, application-, and system-levels cannot be specified and enforced to provide commitment to the guaranteed level of allocation that can help in attracting grid users and contribute to establishing credibility for existing grid environments. In this thesis, we propose and prototype a flexible and extensible Policy-based Resource Brokering Environment (PROBE) that can be utilized by various grid systems. In designing PROBE, we follow a policy-based approach that provides PROBE with the intelligence to not only match the user\u27s request with the right set of resources, but also to assure the guaranteed level of the allocation. PROBE looks at the task allocation as a Service Level Agreement (SLA) that needs to be enforced between the resource provider and the resource consumer. The policy-based framework is useful in a typical grid environment where resources, most of the time, are not dedicated. In implementing PROBE, we have utilized a layered architecture and façade design patterns. These along with the well-defined API, make the framework independent of any architecture and allow for the incorporation of different types of scheduling algorithms, applications and platform adaptors as the underlying environment requires. We have utilized XML as a base for all the specification needs. This provides a flexible mechanism to specify the heterogeneous resources and user\u27s requests along with their allocation constraints. We have developed XML-based specifications by which high-level internal structures of resources, jobs and policies can be specified. This provides interoperability in which a grid system can utilize PROBE to discover and use resources controlled by other grid systems. We have implemented a prototype of PROBE to demonstrate its feasibility. We also describe a test bed environment and the evaluation experiments that we have conducted to demonstrate the usefulness and effectiveness of our approach

Migration system for Zoe microservices

The Zoe virtual assistant developed by the Linux User Group from Carlos III University is a project that aims to automate various tedious tasks of the association. Its architecture is based on independent microservices (agents) that communicate through a common server and can be implemented in almost any programming language thanks to the plain text messaging protocol. The aim of this project is to extend the Zoe platform in order to add migration capabilities to the agents. That way, it would be possible to distribute the load among different machines, making efficient use of the available hardware resources by means of load balancing algorithms while maintaining the internal data across migrations. The approach followed was to design a portable protocol to deal with file and data migrations and implement it in a distributed architecture using basic functionalities from the original project. Said architecture was devised as an addition to Zoe, at software level, that does not require any modification to the original components and uses SSH tunneling for inter-machine communication. As a result, the system obtained is completely backward compatible and transparent, given that all the process is controlled by the independent Scout agent. Furthermore, the two load balancing algorithms implemented automatically migrate the agents either trying to maintain a regular load among all the machines or to use specific machines, which would allow to shut down unused machines. Moreover, the modular code greatly simplifies the addition of new algorithms. As a conclusion, the proposed solution offers a new set of tools for developers of the Zoe project in the hopes of enriching the agent ecosystem. Finally, by releasing it as free software, anyone can expand the capabilities of this work or apply it to other fields, which could lead to new and interesting, projects.El projecto de asistente virtual Zoe desarrollado por el Grupo de Usuarios de Linux de la Universidad Carlos III de Madrid tiene como fin la automatización de diferentes tareas de la asociación. Su arquitectura se basa en microservicios independientes (agentes) que se comunican por medio de un servidor común y pueden implementarse en casi cualquier lenguaje de programación gracias al protocol de mensajería basado en texto plano. El objetivo de este proyecto es extender Zoe para habilitar la migración de agentes. Así, sería posible distribuir la carga entre diferentes máquinas, haciendo un uso eficiente del hardware disponible mediante algoritmos de balanceo de carga, manteniendo además los datos internos durante la migración. Para ello se diseñó un protocolo que se encargara de la migración de archivos y datos y se implementó en una arquitectura distribuida usando funcionalidades básicas del proyecto original. Dicha arquitectura se ideó como un añadido a Zoe a nivel de software que no requiere modificación de los componentes originales y utiliza túneles SSH para la comunicación entre máquinas. El sistema obtenido como resultado es completamente compatible con el original, además de transparente, teniendo en cuenta que el proceso se controla por medio del agente independiente Scout. Además, los dos algoritmos de balanceo de carga implementados migran agentes automáticamente, ya sea para mantener una carga regular entre máquinas o utilizar máquinas específicas, lo que permitiría apagar aquellas en desuso. Aparte de esto, el código modular simplifica enormemente el añadir nuevos algoritmos. Como conclusión, la solución propuesta ofrece un nuevo conjunto de herramientas para desarrolladores del proyecto Zoe con la esperanza de poder enriquecer el ecosistema de agentes. Finalmente, al liberarlo como software libre, cualquiera puede expandir este trabajo o aplicarlo a otros campos, lo cual podría llevar a proyectos interesantes.Ingeniería Informátic

Algorithms in nature: the convergence of systems biology and computational thinking

Biologists rely on computational methods to analyze and integrate large data sets, while several computational methods were inspired by the high-level design principles of biological systems. This Perspectives discusses the recent convergence of these two ways of thinking

Amoeba: Circumventing ML-supported Network Censorship via Adversarial Reinforcement Learning

Embedding covert streams into a cover channel is a common approach to circumventing Internet censorship, due to censors' inability to examine encrypted information in otherwise permitted protocols (Skype, HTTPS, etc.). However, recent advances in machine learning (ML) enable detecting a range of anti-censorship systems by learning distinct statistical patterns hidden in traffic flows. Therefore, designing obfuscation solutions able to generate traffic that is statistically similar to innocuous network activity, in order to deceive ML-based classifiers at line speed, is difficult. In this paper, we formulate a practical adversarial attack strategy against flow classifiers as a method for circumventing censorship. Specifically, we cast the problem of finding adversarial flows that will be misclassified as a sequence generation task, which we solve with Amoeba, a novel reinforcement learning algorithm that we design. Amoeba works by interacting with censoring classifiers without any knowledge of their model structure, but by crafting packets and observing the classifiers' decisions, in order to guide the sequence generation process. Our experiments using data collected from two popular anti-censorship systems demonstrate that Amoeba can effectively shape adversarial flows that have on average 94% attack success rate against a range of ML algorithms. In addition, we show that these adversarial flows are robust in different network environments and possess transferability across various ML models, meaning that once trained against one, our agent can subvert other censoring classifiers without retraining

SNAP: Stateful Network-Wide Abstractions for Packet Processing

Early programming languages for software-defined networking (SDN) were built on top of the simple match-action paradigm offered by OpenFlow 1.0. However, emerging hardware and software switches offer much more sophisticated support for persistent state in the data plane, without involving a central controller. Nevertheless, managing stateful, distributed systems efficiently and correctly is known to be one of the most challenging programming problems. To simplify this new SDN problem, we introduce SNAP. SNAP offers a simpler "centralized" stateful programming model, by allowing programmers to develop programs on top of one big switch rather than many. These programs may contain reads and writes to global, persistent arrays, and as a result, programmers can implement a broad range of applications, from stateful firewalls to fine-grained traffic monitoring. The SNAP compiler relieves programmers of having to worry about how to distribute, place, and optimize access to these stateful arrays by doing it all for them. More specifically, the compiler discovers read/write dependencies between arrays and translates one-big-switch programs into an efficient internal representation based on a novel variant of binary decision diagrams. This internal representation is used to construct a mixed-integer linear program, which jointly optimizes the placement of state and the routing of traffic across the underlying physical topology. We have implemented a prototype compiler and applied it to about 20 SNAP programs over various topologies to demonstrate our techniques' scalability