Benchmarking Hadoop performance on different distributed storage systems

Distributed storage systems have been in place for years, and have undergone significant changes in architecture to ensure reliable storage of data in a cost-effective manner. With the demand for data increasing, there has been a shift from disk-centric to memory-centric computing - the focus is on saving data in memory rather than on the disk. The primary motivation for this is the increased speed of data processing. This could, however, mean a change in the approach to providing the necessary fault-tolerance - instead of data replication, other techniques may be considered. One example of an in-memory distributed storage system is Tachyon. Instead of replicating data files in memory, Tachyon provides fault-tolerance by maintaining a record of the operations needed to generate the data files. These operations are replayed if the files are lost. This approach is termed lineage. Tachyon is already deployed by many well-known companies. This thesis work compares the storage performance of Tachyon with that of the on-disk storage systems HDFS and Ceph. After studying the architectures of well-known distributed storage systems, the major contribution of the work is to integrate Tachyon with Ceph as an underlayer storage system, and understand how this affects its performance, and how to tune Tachyon to extract maximum performance out of it

Identification of Genetic Variants Underlying Simple and Complex Canine Myopathies

The unique population structure of dog breeds, arising from their domestication from gray wolves and subsequent breed formation, and the similarity of their inherited diseases to human disorders make them an ideal comparative genetics model. Herein the genetic basis of three canine muscle diseases, each a model for human disease, is investigated using genome-wide approaches. Nemaline rod myopathy (NM) is one of the most common congenital myopathies in people and is characterized by rod bodies in the muscle fibers, muscle weakness, and reduced muscle tone. We characterized the first large animal model of autosomal recessive NM in a family of American bulldogs and, through a combination of genome-wide SNP profiling and whole exome sequencing, identified a nonsense mutation in NEB. Limb girdle muscular dystrophy (LGMD) affects the hip and shoulder muscles and may cause respiratory and cardiac muscle degeneration. We determined that a muscular dystrophy in a family of Boston terriers is a sarcoglycanopathy, a type of LGMD caused by mutations in one of six sarcoglycan genes, and identified a 2 bp deletion in SGCD through direct whole exome sequencing. Further, an unrelated Boston terrier having LGMD harbors a 19.4 kb deletion, omitting exons 7 and 8 of SGCD. The NEB and SGCD mutations are present only in the affected families. Juvenile dermatomyositis (JDM) is an autoimmune disease with a complex mode of inheritance and an environmental trigger, affecting children ages 2 to 17. In dogs, dermatomyositis (DMS) affects collies and Shetland sheepdogs and causes skin lesions and muscle weakness. We conducted genome-wide association studies in both breeds and identified linkage disequilibrium with SNPs on chromosomes 10 and 31. Through whole genome and transcriptome sequencing, we found mutations in two novel genes, PAN2 (chr10) and MAP3K7CL (chr31). Additionally, we identified an association with a haplotype of the major histocompatibility complex class II genes on chromosome 12, alleles of which are also associated with JDM. When variants at all three loci are considered together, genotypes confer low, moderate, or high risk for DMS, with moderate- and high-risk genotypes explaining 93% of cases

InSight2: An Interactive Web Based Platform for Modeling and Analysis of Large Scale Argus Network Flow Data

Monitoring systems are paramount to the proactive detection and mitigation of problems in computer networks related to performance and security. Degraded performance and compromised end-nodes can cost computer networks downtime, data loss and reputation. InSight2 is a platform that models, analyzes and visualizes large scale Argus network flow data using up-to-date geographical data, organizational information, and emerging threats. It is engineered to meet the needs of network administrators with flexibility and modularity in mind. Scalability is ensured by devising multi-core processing by implementing robust software architecture. Extendibility is achieved by enabling the end user to enrich flow records using additional user provided databases. Deployment is streamlined by providing an automated installation script. State-of-the-art visualizations are devised and presented in a secure, user friendly web interface giving greater insight about the network to the end user

Performance evaluation of a distributed storage service in community network clouds

Community networks are self-organized and decentralized communication networks built and operated by citizens, for citizens. The consolidation of today's cloud technologies offers now, for community networks, the possibility to collectively develop community clouds, building upon user-provided networks and extending toward cloud services. Cloud storage, and in particular secure and reliable cloud storage, could become a key community cloud service to enable end-user applications. In this paper, we evaluate in a real deployment the performance of Tahoe least-authority file system (Tahoe-LAFS), a decentralized storage system with provider-independent security that guarantees privacy to the users. We evaluate how the Tahoe-LAFS storage system performs when it is deployed over distributed community cloud nodes in a real community network such as Guifi.net. Furthermore, we evaluate Tahoe-LAFS in the Microsoft Azure commercial cloud platform, to compare and understand the impact of homogeneous network and hardware resources on the performance of the Tahoe-LAFS. We observed that the write operation of Tahoe-LAFS resulted in similar performance when using either the community network cloud or the commercial cloud. However, the read operation achieved better performance in the Azure cloud, where the reading from multiple nodes of Tahoe-LAFS benefited from the homogeneity of the network and nodes. Our results suggest that Tahoe-LAFS can run on community network clouds with suitable performance for the needed end-user experience.Peer ReviewedPreprin

Nu@ge: Towards a solidary and responsible cloud computing service

Best Paper AwardInternational audienceThe adoption of cloud computing is still limited by several legal concerns from companies. One of those reasons is the data sovereignty, as data can be physically host in sensible locations, resulting in a lack of control for companies. In this paper, we present the Nu@ge project aimed at building a federation of container-sized datacenter on the French territory. Nu@ge provides a software stack that enables companies to put independent datacenters in cooperation in a national mesh. Additionally, a prototype of a container-sized datacenter has been validated and patented

VM Image Repository and Distribution Models for Federated Clouds: State of the Art, Possible Directions and Open Issues

The emerging trend of Federated Cloud models enlist virtualization as a significant concept to offer a large scale distributed Infrastructure as a Service collaborative paradigm to end users. Virtualization leverage Virtual Machines (VM) instantiated from user specific templates labelled as VM Images (VMI). To this extent, the rapid provisioning of VMs with varying user requests ensuring Quality of Service (QoS) across multiple cloud providers largely depends upon the image repository architecture and distribution policies. We discuss the possible state-of-art in VMI storage repository and distribution mechanisms for efficient VM provisioning in federated clouds. In addition, we present and compare various representative systems in this realm. Furthermore, we define a design space, identify current limitations, challenges and open trends for VMI repositories and distribution techniques within federated infrastructure

Context Awareness in Swarm Systems

Recent swarms of Uncrewed Systems (UxS) require substantial human input to support their operation. The little 'intelligence' on these platforms limits their potential value and increases their overall cost. Artificial Intelligence (AI) solutions are needed to allow a single human to guide swarms of larger sizes. Shepherding is a bio-inspired swarm guidance approach with one or a few sheepdogs guiding a larger number of sheep. By designing AI-agents playing the role of sheepdogs, humans can guide the swarm by using these AI agents in the same manner that a farmer uses biological sheepdogs to muster sheep. A context-aware AI-sheepdog offers human operators a smarter command and control system. It overcomes the current limiting assumption in the literature of swarm homogeneity to manage heterogeneous swarms and allows the AI agents to better team with human operators. This thesis aims to demonstrate the use of an ontology-guided architecture to deliver enhanced contextual awareness for swarm control agents. The proposed architecture increases the contextual awareness of AI-sheepdogs to improve swarm guidance and control, enabling individual and collective UxS to characterise and respond to ambiguous swarm behavioural patterns. The architecture, associated methods, and algorithms advance the swarm literature by allowing improved contextual awareness to guide heterogeneous swarms. Metrics and methods are developed to identify the sources of influence in the swarm, recognise and discriminate the behavioural traits of heterogeneous influencing agents, and design AI algorithms to recognise activities and behaviours. The proposed contributions will enable the next generation of UxS with higher levels of autonomy to generate more effective Human-Swarm Teams (HSTs)

openstack

Σκοπός της Διπλωματικής εργασίας είναι η παρουσίαση του OpenStack. Ένα ανοιχτό λογισμικό διαχείρισης των τηλεπικοινωνιακών πόρων σε cloud περιβάλλον. Για την εκπόνηση της Διπλωματικής εργασίας γίνεται αρχικά μία περιγραφή της αρχιτεκτονικής του cloud περιβάλλοντος, και των μοντέλων εξυπηρέτησης που χρησιμοποιούνται. Εν συνεχεία παρουσιάζεται η αρχιτεκτονική Network Function Virtualization που εφαρμόζεται στις τηλεπικοινωνίες σύμφωνα με τα πρότυπα που έχει θέσει ο Ευρωπα’ι’κός Οργανισμός Τηλεπικοινωνιακών Προτύπων. Το κύριο θέμα της Διπλωματικής Εργασίας είναι η παρουσίαση του λογισμικού OpenStack που χρησιμοποιείται από την NFV αρχιτεκτονική. Στα κεφάλαια αυτά γίνεται μία προσπάθεια όσο το δυνατόν λεπτομερέστερης και πληρέστερης περιγραφής των λειτουργιών του OpenStack καθώς και τα μέρη από τα αποία αποτελείται. Τέλος γίνεται μία τεχνοοικονομική ανάλυση του κόστους εφαρμογής της NFV αρχιτεκτονικής με την τωρινή αρχιτεκτονική που εφαρμόζεται στα Τηλεπιοκοινωνιακά δίκτυα. Τα αποτελέσματα τα οποία προκύπτουν από την παρούσα εργασία είναι η πολλές δυνατότητες υλοπόιησης και εφαρμογής της νέας αρχιτεκτονικής καθώς και το πολύ χαμηλό κόστος λειτουργίας της σε σχέση με την υφιστάμενη εώς τώρα τεχνολογίαThe aim of this thesis is the presentation of OpenStack. An open software management of telecommunications resources in cloud environment. For the preparation of this thesis is first a description of the architecture of cloud environment, and service models used. Architecture Network Function Virtualization occurs then applied to telecommunications in accordance with standards set by the European Telecommunications Standards Institute. The main topic of the thesis is to present the OpenStack software used by the NFV architecture. In these chapters an attempt is as detailed and comprehensive description of the OpenStack functions and parts of the colony consists. Finally there is one techno-economic analysis of the cost of implementing NFV architecture with the current architecture applied to Telecommunication networks. The results derived from this work is a lot of potential implementation and application of the new architecture and the very low operating costs compared with existing technology up to no

Sheepdog or watchdog? The role of statutory public involvement institutions in political management of the NHS, 1974-2010

Since 1974, governments have created a series of statutory public involvement institutions in the English NHS: Community Health Councils; Patient and Public Involvement Forums; Local Involvement Networks; and, from 2013, local Healthwatch. This study presents the role of these institutions as a puzzle, given the growth of alternative forms of public involvement. Public involvement in the NHS tends to be studied for its contribution to democratising the NHS or for its role in a choice-led consumer market, but these analyses generally focus on involvement led by NHS personnel or by independent patient organisations. This dissertation uses a different body of political science theory to assess roles that statutory public involvement institutions may play in Ministers’ political management of the NHS. One approach is to see these institutions as ‘sheepdogs’, rounding up and organising diverse groups, thus providing a form of corporatist interest intermediation. Alternatively, they could be ‘watchdogs’, raising the alarm when standards slip critically and thereby helping to safeguard the NHS against disasters. The explanatory value of these two interpretations is reviewed over the period 1974-2010, using policy documents and archive material, including the records of these organisations and the archives of public inquiries into problems in hospitals. The findings suggest that at various times national political actors have used statutory public involvement institutions to manage the representation and mobilisation of interests and to alert them to problems in local health services. There is more recent evidence for the watchdog than for the sheepdog role. The watchdog role has been reappraised following the failure of statistical monitoring and regulatory police patrols to prevent disasters in the NHS. The discussion also shows how the sheepdog powers of these institutions mean that they can round up opposition, rather than moderating it, resulting in Ministerial reforms to statutory public involvement institutions