Improving the Performance of Internet Data Transport

With the explosion of the World Wide Web, the Internet infrastructure faces new challenges in providing high performance for data traffic. First, it must be able to pro-vide a fair-share of congested link bandwidth to every flow. Second, since web traffic is inherently interactive, it must minimize the delay for data transfer. Recent studies have shown that queue management algorithms such as Tail Drop, RED and Blue are deficient in providing high throughput, low delay paths for a data flow. Two major shortcomings of the current algorithms are: they allow TCP flows to get synchronized and thus require large buffers during congestion to enable high throughput; and they allow unfair bandwidth usage for shorter round-trip time TCP flows. We propose algorithms using multiple queues and discard policies with hysteresis at bottleneck routers to address both these issues. Us-ing ns-2 simulations, we show that these algorithms can significantly outperform RED and Blue, especially at smaller buffer sizes. Using multiple queues raises two new concerns: scalability and excess memory bandwidth usage caused by dropping packets which have been queued. We propose and evaluate an architecture using Bloom filters to evenly distribute flows among queues to improve scalability. We have also developed new intelligent packet discard algorithms that discard packets on arrival and are able to achieve performance close to that of policies that may discard packets that have already been queued. Finally, we propose better methods for evaluating the performance of fair-queueing methods. In the current literature, fair-queueing methods are evaluated based on their worst-case performance. This can exaggerate the differences among algorithms, since the worst-case behavior is dependent on the the precise timing of packet arrivals. This work seeks to understand what happens under more typical circumstances

Enhancing the genetic-based scheduling in computational grids by a structured hierarchical population

Independent Job Scheduling is one of the most useful versions of scheduling in grid systems. It aims at computing efficient and optimal mapping of jobs and/or applications submitted by independent users to the grid resources. Besides traditional restrictions, mapping of jobs to resources should be computed under high degree of heterogeneity of resources, the large scale and the dynamics of the system. Because of the complexity of the problem, the heuristic and meta-heuristic approaches are the most feasible methods of scheduling in grids due to their ability to deliver high quality solutions in reasonable computing time. One class of such meta-heuristics is Hierarchic Genetic Strategy (HGS). It is defined as a variant of Genetic Algorithms (GAs) which differs from the other genetic methods by its capability of concurrent search of the solution space. In this work, we present an implementation of HGS for Independent Job Scheduling in dynamic grid environments. We consider the bi-objective version of the problem in which makespan and flowtime are simultaneously optimized. Based on our previous work, we improve the HGS scheduling strategy by enhancing its main branching operations. The resulting HGS-based scheduler is evaluated under the heterogeneity, the large scale and dynamics conditions using a grid simulator. The experimental study showed that the HGS implementation outperforms existing GA-based schedulers proposed in the literature.Peer ReviewedPostprint (author's final draft

Distributed, Secure Load Balancing with Skew, Heterogeneity, and Churn

Numerous proposals exist for load balancing in peer-to-peer (p2p) networks. Some focus on namespace balancing, making the distance between nodes as uniform as possible. This technique works well under ideal conditions, but not under those found empirically. Instead, researchers have found heavytailed query distributions (skew), high rates of node join and leave (churn), and wide variation in node network and storage capacity (heterogeneity). Other approaches tackle these less-thanideal conditions, but give up on important security properties. We propose an algorithm that both facilitates good performance and does not dilute security. Our algorithm, k-Choices, achieves load balance by greedily matching nodes’ target workloads with actual applied workloads through limited sampling, and limits any fundamental decrease in security by basing each nodes’ set of potential identifiers on a single certificate. Our algorithm compares favorably to four others in trace-driven simulations. We have implemented our algorithm and found that it improved aggregate throughput by 20% in a widely heterogeneous system in our experiments.Engineering and Applied Science

Efficient Management of Short-Lived Data

Motivated by the increasing prominence of loosely-coupled systems, such as mobile and sensor networks, which are characterised by intermittent connectivity and volatile data, we study the tagging of data with so-called expiration times. More specifically, when data are inserted into a database, they may be tagged with time values indicating when they expire, i.e., when they are regarded as stale or invalid and thus are no longer considered part of the database. In a number of applications, expiration times are known and can be assigned at insertion time. We present data structures and algorithms for online management of data tagged with expiration times. The algorithms are based on fully functional, persistent treaps, which are a combination of binary search trees with respect to a primary attribute and heaps with respect to a secondary attribute. The primary attribute implements primary keys, and the secondary attribute stores expiration times in a minimum heap, thus keeping a priority queue of tuples to expire. A detailed and comprehensive experimental study demonstrates the well-behavedness and scalability of the approach as well as its efficiency with respect to a number of competitors.Comment: switched to TimeCenter latex styl

Geometric Algorithms and Data Structures for Simulating Diffusion Limited Reactions

Radiation therapy is one of the most effective means for treating cancers. An important calculation in radiation therapy is the estimation of dose distribution in the treated patient, which is key to determining the treatment outcome and potential side effects of the therapy. Biological dose — the level of biological damage (e.g., cell killing ratio, DNA damage, etc.) inflicted by the radiation is the best measure of treatment quality, but it is very difficult to calculate. Therefore, most clinics today use physical dose - the energy deposited by incident radiation per unit body mass - for planning radiation therapy, which can be calculated accurately using kinetic Monte Carlo simulations. Studies have found that physical dose correlates with biological dose, but exhibits a very complex relationship that is not yet well understood. Generally speaking, the calculation of biological dose involves four steps: (1) the calculation of physical dose distribution, (2) the generation of radiochemicals based on the physical dose distribution, (3) the simulation of interactions between radiochemicals and bio-matter in the body, and (4) the estimation of biological damage based on the distribution of radiochemicals. This dissertation focuses on the development of a more efficient and effective simulation algorithm to speed up step (3). The main contribution of this research is the development of an efficient and effective kinetic Monte Carlo (KMC) algorithm for simulating diffusion-limited chemical reactions in the context of radiation therapy. The central problem studied is - given n particles distributed among a small number of particle species, all allowed to diffuse and chemically react according to a small number of chemical reaction equations - predict the radiochemical yield over time. The algorithm presented makes use of a sparse grid structure, with one grid per species per radiochemical reactant used to group particles in a way that makes the nearest neighbor search efficient, where particles are stored only once, yet are represented in grids of all appropriate reaction radii. A kinetic data structure is used as the time stepping mechanism, which provides spatially local updates to the simulation at a frequency which captures all events - retaining accuracy. A serial and three parallel versions of the algorithm have been developed. The parallel versions implement the kinetic data structure using both a standard priority queue and a treap data structure in order to investigate the algorithms scalability. The treap provides a way for each thread of execution to do more work in a particular region of space. A comparison with a spatial discretization variant of the algorithm is also provided

Analysis of cryptocurrency markets from 2016 to 2019

This thesis explores machine learning techniques in algorithmic trading. We implement a trading computer program that balances a portfolio of cryptocurrencies. We try to outperform an equally weighted strategy. As our machine learning technique, we use deep reinforcement learning. Cryptocurrencies are digital mediums of exchange that use cryptography to secure transactions. The most well-known example is Bitcoin. They are interesting to analyze due to high volatility and lack of previous research. The availability of data is also exceptional. We introduce an algorithmic trading agent – a computer program powered by machine learning. The agent follows some pre-determined instructions and executes market orders. Traditionally a human trader determines these instructions by using some technical indicators. We instead give the trading agent raw price data as input and let it figure out its instructions. The agent uses machine learning to figure out the trading rules. We evaluate the performance of the agent in seven different backtest stories. Each backtest story reflects some unique and remarkable period in cryptocurrency history. One backtest period was from December 2017 when Bitcoin reached its all-time high price. Another one is from April 2017 when Bitcoin almost lost its place as the most valued cryptocurrency. The stories show the market conditions where the agent excels and reveals its risks. The algorithmic trading agent has two goals. First, it chooses initial weights, and then it rebalances these weights periodically. Choosing proper initial weights is crucial since transaction costs make trade action costly. We evaluate the trading agent’s performance in these two tasks by using two agents: a static and a dynamic agent. The static agent only does the weight initialization and does not rebalance. The dynamic agent also rebalances. We find that the agent does a poor job in choosing initial weights. We also want to find out the optimal time-period for rebalancing for the dynamic agent. Therefore, we compare rebalancing periods from 15 minutes to 1 day. To make our results robust, we ran over a thousand simulations. We found that 15 – 30 minutes rebalancing periods tend to work the best. We find that the algorithmic trading agent closely follows an equally weighted strategy. This finding suggests that the agent is unavailable to decipher meaningful signals from the noisy price data. The machine learning approach does not provide an advantage over equally weighted strategy. Nevertheless, the trading agent excels in volatile and mean reverting market conditions. On these periods, the dynamic agent has lower volatility and a higher Sharpe ratio. However, it has a dangerous tendency of following the looser. Our results contribute to the field of algorithmic finance. We show that frequent rebalancing is a useful tool in the risk management of highly volatile asset classes. Further investigation is required to extend these findings beyond cryptocurrencies

Off-chain Transaction Routing in Payment Channel Networks: A Machine Learning Approach

Blockchain is a foundational technology that has the potential to create new prospects for our economic and social systems. However, the scalability problem limits the capability to deliver a target throughput and latency, compared to the traditional financial systems, with increasing workload. Layer-two is a collective term for solutions designed to help solve the scalability by handling transactions off the main chain, also known as layer one. These solutions have the capability to achieve high throughput, fast settlement, and cost efficiency without sacrificing network security. For example, bidirectional payment channels are utilized to allow the execution of fast transactions between two parties, thus forming the so-called payment channel networks (PCNs). Consequently, an efficient routing protocol is needed to find the payment path from the sender to the receiver, with the lowest transaction fees. This routing protocol needs to consider, among other factors, the unexpected online/offline behavior of the constituent payment nodes as well as payment channel imbalance. This study proposes a novel machine learning-based routing technique for fully distributed and efficient off-chain transactions to be used within the PCNs. For this purpose, the effect of the offline nodes and channel imbalance on the payment channels network are modeled. The simulation results demonstrate a good tradeoff among success ratio, transaction fees, routing efficiency, transaction overhead, and transaction maintenance overhead as compared to other techniques that have been previously proposed for the same purpose

Scalability in extensible and heterogeneous storage systems

The evolution of computer systems has brought an exponential growth in data volumes, which pushes the capabilities of current storage architectures to organize and access this information effectively: as the unending creation and demand of computer-generated data grows at an estimated rate of 40-60% per year, storage infrastructures need increasingly scalable data distribution layouts that are able to adapt to this growth with adequate performance. In order to provide the required performance and reliability, large-scale storage systems have traditionally relied on multiple RAID-5 or RAID-6 storage arrays, interconnected with high-speed networks like FibreChannel or SAS. Unfortunately, the performance of the current, most commonly-used storage technology-the magnetic disk drive-can't keep up with the rate of growth needed to sustain this explosive growth. Moreover, storage architectures based on solid-state devices (the successors of current magnetic drives) don't seem poised to replace HDD-based storage for the next 5-10 years, at least in data centers. Though the performance of SSDs significantly improves that of hard drives, it would cost the NAND industry hundreds of billions of dollars to build enough manufacturing plants to satisfy the forecasted demand. Besides the problems derived from technological and mechanical limitations, the massive data growth poses more challenges: to build a storage infrastructure, the most flexible approach consists in using pools of storage devices that can be expanded as needed by adding new devices or replacing older ones, thus seamlessly increasing the system's performance and capacity. This approach however, needs data layouts that can adapt to these topology changes and also exploit the potential performance offered by the hardware. Such strategies should be able to rebuild the data layout to accommodate the new devices in the infrastructure, extracting the utmost performance from the hardware and offering a balanced workload distribution. An inadequate data layout might not effectively use the enlarged capacity or better performance provided by newer devices, thus leading to unbalancing problems like bottlenecks or resource underusage. Besides, massive storage systems will inevitably be composed of a collection of heterogeneous hardware: as capacity and performance requirements grow, new storage devices must be added to cope with demand, but it is unlikely that these devices will have the same capacity or performance of those installed. Moreover, upon failure, disks are most commonly replaced by faster and larger ones, since it is not always easy (or cheap) to find a particular model of drive. In the long run, any large-scale storage system will have to cope with a myriad of devices. The title of this dissertation, "Scalability in Extensible and Heterogeneous Storage Systems", refers to the main focus of our contributions in scalable data distributions that can adapt to increasing volumes of data. Our first contribution is the design of a scalable data layout that can adapt to hardware changes while redistributing only the minimum data to keep a balanced workload. With the second contribution, we perform a comparative study on the influence of pseudo-random number generators in the performance and distribution quality of randomized layouts and prove that a badly chosen generator can degrade the quality of the strategy. Our third contribution is an an analysis of long-term data access patterns in several real-world traces to determine if it is possible to offer high performance and a balanced load with less than minimal data rebalancing. In our final contribution, we apply the knowledge learnt about long-term access patterns to design an extensible RAID architecture that can adapt to changes in the number of disks without migrating large amounts of data, and prove that it can be competitive with current RAID arrays with an overhead of at most 1.28% the storage capacity.L'evolució dels sistemes de computació ha dut un creixement exponencial dels volums de dades, que porta al límit la capacitat d'organitzar i accedir informació de les arquitectures d'emmagatzemament actuals. Amb una incessant creació de dades que creix a un ritme estimat del 40-60% per any, les infraestructures de dades requereixen de distribucions de dades cada cop més escalables que puguin adaptar-se a aquest creixement amb un rendiment adequat. Per tal de proporcionar aquest rendiment, els sistemes d'emmagatzemament de gran escala fan servir agregacions RAID5 o RAID6 connectades amb xarxes d'alta velocitat com FibreChannel o SAS. Malauradament, el rendiment de la tecnologia més emprada actualment, el disc magnètic, no creix prou ràpid per sostenir tal creixement explosiu. D'altra banda, les prediccions apunten que els dispositius d'estat sòlid, els successors de la tecnologia actual, no substituiran els discos magnètics fins d'aquí 5-10 anys. Tot i que el rendiment és molt superior, la indústria NAND necessitarà invertir centenars de milions de dòlars per construir prou fàbriques per satisfer la demanda prevista. A més dels problemes derivats de limitacions tècniques i mecàniques, el creixement massiu de les dades suposa més problemes: la solució més flexible per construir una infraestructura d'emmagatzematge consisteix en fer servir grups de dispositius que es poden fer créixer bé afegint-ne de nous, bé reemplaçant-ne els més vells, incrementant així la capacitat i el rendiment del sistema de forma transparent. Aquesta solució, però, requereix distribucions de dades que es puguin adaptar a aquests canvis a la topologia i explotar el rendiment potencial que el hardware ofereix. Aquestes distribucions haurien de poder reconstruir la col.locació de les dades per acomodar els nous dispositius, extraient-ne el màxim rendiment i oferint una càrrega de treball balancejada. Una distribució inadient pot no fer servir de manera efectiva la capacitat o el rendiment addicional ofert pels nous dispositius, provocant problemes de balanceig com colls d¿ampolla o infrautilització. A més, els sistemes d'emmagatzematge massius estaran inevitablement formats per hardware heterogeni: en créixer els requisits de capacitat i rendiment, es fa necessari afegir nous dispositius per poder suportar la demanda, però és poc probable que els dispositius afegits tinguin la mateixa capacitat o rendiment que els ja instal.lats. A més, en cas de fallada, els discos són reemplaçats per d'altres més ràpids i de més capacitat, ja que no sempre és fàcil (o barat) trobar-ne un model particular. A llarg termini, qualsevol arquitectura d'emmagatzematge de gran escala estarà formada per una miríade de dispositius diferents. El títol d'aquesta tesi, "Scalability in Extensible and Heterogeneous Storage Systems", fa referència a les nostres contribucions a la recerca de distribucions de dades escalables que es puguin adaptar a volums creixents d'informació. La primera contribució és el disseny d'una distribució escalable que es pot adaptar canvis de hardware només redistribuint el mínim per mantenir un càrrega de treball balancejada. A la segona contribució, fem un estudi comparatiu sobre l'impacte del generadors de números pseudo-aleatoris en el rendiment i qualitat de les distribucions pseudo-aleatòries de dades, i provem que una mala selecció del generador pot degradar la qualitat de l'estratègia. La tercera contribució és un anàlisi dels patrons d'accés a dades de llarga duració en traces de sistemes reals, per determinar si és possible oferir un alt rendiment i una bona distribució amb una rebalanceig inferior al mínim. A la contribució final, apliquem el coneixement adquirit en aquest estudi per dissenyar una arquitectura RAID extensible que es pot adaptar a canvis en el número de dispositius sense migrar grans volums de dades, i demostrem que pot ser competitiva amb les distribucions ideals RAID actuals, amb només una penalització del 1.28% de la capacita

Autonomous mobility in multilevel networks

Autonomous Mobile Programs (AMPs) are mobile agents that are aware of their resource needs and sensitive to the execution environment. AMPs are unusual in that, instead of using some external load management system, each AMP periodically recalculates network and program parameters and independently moves to a new location if it provides a better execution environment. Dynamic load management emerges from the behaviour of collections of AMPs. AMPs have previously been measured using mobile languages like Java Voyager on local area networks (LANs). The thesis develops an accurate simulation for AMPs on networks and validates it by reproducing the behaviour of collections of AMPs on homogeneous and heterogeneous LANs. The analysis shows that AMPs exhibit thrashing like other distributed load balancers. This thrashing is investigated in collections of AMPs, and two types of redundant movement (greedy effect) are identified. The thesis explores the extent of greedy effects by simulating collections of AMPs, and proposes negotiating AMPs (NAMPs) to ameliorate the problem. The design of AMPs with a competitive negotiation scheme (cNAMPs) is presented, followed by a performance comparison AMPs and cNAMPs using simulation. To estimate the significance of the greedy effects the properties of balanced states are established, such as independent balance, singleton optimality, and consecutive optimality. The balanced states are characterised for homogeneous and heterogeneous networks where AMPs are analysed as the general case. The significance of the cNAMP greedy effect is established by conducting a worst case analysis of redundant movements, and the maximum number, and probability of, redundant movements are calculated for homogeneous and heterogeneous networks. One of three theorems proves that in a heterogeneous network of q subnetworks the number of redundant movements does not exceed q − 1. i The thesis proposes and evaluates a multilevel cNAMP architecture that abstracts over network topologies to effectively distribute cNAMPs in large networks. The thesis investigates alternatives for implementation of this multilevel architecture and proposes a fusion-based scheme where information is first available to neighbour nodes. These neighbour nodes modify the information and pass it to remote locations. The effectiveness of the scheme is evaluated by simulating networks with up to five levels, varying the number of locations from 5 to 336, and the number of cNAMPs from 8 to 3360. The experiments investigate the effects depending on the number of levels, topologies, number of locations, number of cNAMPs, work of cNAMPs, type of cNAMPs, speed of locations, and type of rebalancing. The architecture is found to be effective because it delivers performance close to the hypothetical, e.g. each additional level increases mean cNAMP completion time by just 2%

Adaptive mesh simulations of compressible flows using stabilized formulations

This thesis investigates numerical methods that approximate the solution of compressible flow equations. The first part of the thesis is committed to studying the Variational Multi-Scale (VMS) finite element approximation of several compressible flow equations. In particular, the one-dimensional Burgers equation in the Fourier space, and the compressible Navier-Stokes equations written in both conservative and primitive variables are considered. The approximations made for the VMS formulation are extensively researched; the design of the matrix of stabilization parameters, the definition of the space where the subscales live, the inclusion of the temporal derivatives of the subscales, and the non-linear tracking of the subscales are formulated. Also, the addition of local artificial diffusion in the form of shock capturing techniques is included. The accuracy of the formulations is studied for several regimes of the compressible flow, from aeroacoustic flows at low Mach numbers to supersonic shocks. The second part of the thesis is devoted to make the solution of the smallest fluctuating scales of the compressible flow affordable. To this end, a novel algorithm for $h-$refinement of computational physics meshes in a distributed parallel setting, together with the solution of some refinement test cases in supercomputers are presented. The definition of an explicit a-posteriori error estimator that can be used in the adaptive mesh refinement simulations of compressible flows is also developed; the proposed methodology employs the variational subscales as a local error estimate that drives the mesh refinement. The numerical methods proposed in this thesis are capable to describe the high-frequency fluctuations of compressible flows, especially, the ones corresponding to complex aeroacoustic applications. Precisely, the direct simulation of the fricative [s] sound inside a realistic geometry of the human vocal tract is achieved at the end of the thesis.Esta tesis investiga métodos numéricos que aproximan la solución de las ecuaciones de flujo compresible. La primera parte de la tesis está dedicada al estudio de la aproximación numérica del flujo compresible por medio del método multiescala variacional (VMS) en elementos finitos. En particular, se consideran la ecuación de Burgers unidimensional descrita en el espacio de Fourier y las ecuaciones de Navier-Stokes de flujo compresible escritas en variables conservativas y primitivas. Las aproximaciones hechas para plantear la formulación VMS son ampliamente investigadas; el diseño de la matriz de parámetros de estabilización, la definición del espacio donde viven las subescalas, la inclusión de las derivadas temporales de las subescalas y el seguimiento no lineal de las subescalas son particularidades de la formulación que se analizan para cada una de las ecuaciones consideradas. Además, se incluye la adición de difusión artificial local en forma de técnicas de captura de choque. La precisión de las formulaciones se estudia para varios regímenes del flujo compresible, desde flujos aeroacústicos a bajos números de Mach hasta choques supersónicos. La segunda parte de la tesis está dedicada a hacer asequible la solución de las escalas fluctuantes más pequeñas del flujo compresible. Con este fin, se presenta un algoritmo novedoso para el refinamiento $h$ de las mallas de física computacional usadas en computación distribuida en paralelo. Además, se demuestra la solución en superordenadores de algunos casos de prueba del refinamiento de mallas. También se desarrolla la definición de un estimador de error explícito a posteriori que se puede usar en las simulaciones adaptativas de refinamiento de malla de flujos compresibles; la metodología propuesta emplea las subescalas variacionales como una estimación de error local que induce el refinamiento de la malla. Los métodos numéricos propuestos en esta tesis son capaces de describir las fluctuaciones de alta frecuencia de los flujos compresibles, especialmente los correspondientes a aplicaciones aeroacústicas complejas. Precisamente, la simulación directa del sonido consonántico fricativo [s] dentro de una geometría realista del tracto vocal humano se demuestra al final de la tesis