WiSer: A Highly Available HTAP DBMS for IoT Applications

In a classic transactional distributed database management system (DBMS), write transactions invariably synchronize with a coordinator before final commitment. While enforcing serializability, this model has long been criticized for not satisfying the applications' availability requirements. When entering the era of Internet of Things (IoT), this problem has become more severe, as an increasing number of applications call for the capability of hybrid transactional and analytical processing (HTAP), where aggregation constraints need to be enforced as part of transactions. Current systems work around this by creating escrows, allowing occasional overshoots of constraints, which are handled via compensating application logic. The WiSer DBMS targets consistency with availability, by splitting the database commit into two steps. First, a PROMISE step that corresponds to what humans are used to as commitment, and runs without talking to a coordinator. Second, a SERIALIZE step, that fixes transactions' positions in the serializable order, via a consensus procedure. We achieve this split via a novel data representation that embeds read-sets into transaction deltas, and serialization sequence numbers into table rows. WiSer does no sharding (all nodes can run transactions that modify the entire database), and yet enforces aggregation constraints. Both readwrite conflicts and aggregation constraint violations are resolved lazily in the serialized data. WiSer also covers node joins and departures as database tables, thus simplifying correctness and failure handling. We present the design of WiSer as well as experiments suggesting this approach has promise

Planetary Scale Data Storage

The success of virtualization and container-based application deployment has fundamentally changed computing infrastructure from dedicated hardware provisioning to on-demand, shared clouds of computational resources. One of the most interesting effects of this shift is the opportunity to localize applications in multiple geographies and support mobile users around the globe. With relatively few steps, an application and its data systems can be deployed and scaled across continents and oceans, leveraging the existing data centers of much larger cloud providers. The novelty and ease of a global computing context means that we are closer to the advent of an Oceanstore, an Internet-like revolution in personalized, persistent data that securely travels with its users. At a global scale, however, data systems suffer from physical limitations that significantly impact its consistency and performance. Even with modern telecommunications technology, the latency in communication from Brazil to Japan results in noticeable synchronization delays that violate user expectations. Moreover, the required scale of such systems means that failure is routine. To address these issues, we explore consistency in the implementation of distributed logs, key/value databases and file systems that are replicated across wide areas. At the core of our system is hierarchical consensus, a geographically-distributed consensus algorithm that provides strong consistency, fault tolerance, durability, and adaptability to varying user access patterns. Using hierarchical consensus as a backbone, we further extend our system from data centers to edge regions using federated consistency, an adaptive consistency model that gives satellite replicas high availability at a stronger global consistency than existing weak consistency models. In a deployment of 105 replicas in 15 geographic regions across 5 continents, we show that our implementation provides high throughput, strong consistency, and resiliency in the face of failure. From our experimental validation, we conclude that planetary-scale data storage systems can be implemented algorithmically without sacrificing consistency or performance

Building global and scalable systems with atomic multicast

The rise of worldwide Internet-scale services demands large distributed systems. Indeed, when handling several millions of users, it is common to operate thousands of servers spread across the globe. Here, replication plays a central role, as it contributes to improve the user experience by hiding failures and by providing acceptable latency. In this thesis, we claim that atomic multicast, with strong and well-defined properties, is the appropriate abstraction to efficiently design and implement globally scalable distributed systems. Internet-scale services rely on data partitioning and replication to provide scalable performance and high availability. Moreover, to reduce user-perceived response times and tolerate disasters (i.e., the failure of a whole datacenter), services are increasingly becoming geographically distributed. Data partitioning and replication, combined with local and geographical distribution, introduce daunting challenges, including the need to carefully order requests among replicas and partitions. One way to tackle this problem is to use group communication primitives that encapsulate order requirements. While replication is a common technique used to design such reliable distributed systems, to cope with the requirements of modern cloud based ``always-on'' applications, replication protocols must additionally allow for throughput scalability and dynamic reconfiguration, that is, on-demand replacement or provisioning of system resources. We propose a dynamic atomic multicast protocol which fulfills these requirements. It allows to dynamically add and remove resources to an online replicated state machine and to recover crashed processes. Major efforts have been spent in recent years to improve the performance, scalability and reliability of distributed systems. In order to hide the complexity of designing distributed applications, many proposals provide efficient high-level communication abstractions. Since the implementation of a production-ready system based on this abstraction is still a major task, we further propose to expose our protocol to developers in the form of distributed data structures. B-trees for example, are commonly used in different kinds of applications, including database indexes or file systems. Providing a distributed, fault-tolerant and scalable data structure would help developers to integrate their applications in a distribution transparent manner. This work describes how to build reliable and scalable distributed systems based on atomic multicast and demonstrates their capabilities by an implementation of a distributed ordered map that supports dynamic re-partitioning and fast recovery. To substantiate our claim, we ported an existing SQL database atop of our distributed lock-free data structure. Here, replication plays a central role, as it contributes to improve the user experience by hiding failures and by providing acceptable latency. In this thesis, we claim that atomic multicast, with strong and well-defined properties, is the appropriate abstraction to efficiently design and implement globally scalable distributed systems. Internet-scale services rely on data partitioning and replication to provide scalable performance and high availability. Moreover, to reduce user-perceived response times and tolerate disasters (i.e., the failure of a whole datacenter), services are increasingly becoming geographically distributed. Data partitioning and replication, combined with local and geographical distribution, introduce daunting challenges, including the need to carefully order requests among replicas and partitions. One way to tackle this problem is to use group communication primitives that encapsulate order requirements. While replication is a common technique used to design such reliable distributed systems, to cope with the requirements of modern cloud based ``always-on'' applications, replication protocols must additionally allow for throughput scalability and dynamic reconfiguration, that is, on-demand replacement or provisioning of system resources. We propose a dynamic atomic multicast protocol which fulfills these requirements. It allows to dynamically add and remove resources to an online replicated state machine and to recover crashed processes. Major efforts have been spent in recent years to improve the performance, scalability and reliability of distributed systems. In order to hide the complexity of designing distributed applications, many proposals provide efficient high-level communication abstractions. Since the implementation of a production-ready system based on this abstraction is still a major task, we further propose to expose our protocol to developers in the form of distributed data structures. B- trees for example, are commonly used in different kinds of applications, including database indexes or file systems. Providing a distributed, fault-tolerant and scalable data structure would help developers to integrate their applications in a distribution transparent manner. This work describes how to build reliable and scalable distributed systems based on atomic multicast and demonstrates their capabilities by an implementation of a distributed ordered map that supports dynamic re-partitioning and fast recovery. To substantiate our claim, we ported an existing SQL database atop of our distributed lock-free data structure

Blockchain's adoption in IoT: The challenges, and a way forward

© 2018 Elsevier Ltd The underlying technology of Bitcoin is blockchain, which was initially designed for financial value transfer only. Nonetheless, due to its decentralized architecture, fault tolerance and cryptographic security benefits such as pseudonymous identities, data integrity and authentication, researchers and security analysts around the world are focusing on the blockchain to resolve security and privacy issues of IoT. However, presently, not much work has been done to assess blockchain's viability for IoT and the associated challenges. Hence, to arrive at intelligible conclusions, this paper carries out a systematic study of the peculiarities of the IoT environment including its security and performance requirements and progression in blockchain technologies. We have identified the gaps by mapping the security and performance benefits inferred by the blockchain technologies and some of the blockchain-based IoT applications against the IoT requirements. We also discovered some practical issues involved in the integration of IoT devices with the blockchain. In the end, we propose a way forward to resolve some of the significant challenges to the blockchain's adoption in IoT

Consensus protocols exploiting network programmability

Services rely on replication mechanisms to be available at all time. The service demanding high availability is replicated on a set of machines called replicas. To maintain the consistency of replicas, a consensus protocol such as Paxos or Raft is used to synchronize the replicas' state. As a result, failures of a minority of replicas will not affect the service as other non-faulty replicas continue serving requests. A consensus protocol is a procedure to achieve an agreement among processors in a distributed system involving unreliable processors. Unfortunately, achieving such an agreement involves extra processing on every request, imposing a substantial performance degradation. Consequently, performance has long been a concern for consensus protocols. Although many efforts have been made to improve consensus performance, it continues to be an important problem for researchers. This dissertation presents a novel approach to improving consensus performance. Essentially, it exploits the programmability of a new breed of network devices to accelerate consensus protocols that traditionally run on commodity servers. The benefits of using programmable network devices to run consensus protocols are twofold: The network switches process packets faster than commodity servers and consensus messages travel fewer hops in the network. It means that the system throughput is increased and the latency of requests is reduced. The evaluation of our network-accelerated consensus approach shows promising results. Individual components of our FPGA- based and switch-based consensus implementations can process 10 million and 2.5 billion consensus messages per second, respectively. Our FPGA-based system as a whole delivers 4.3 times performance of a traditional software consensus implementation. The latency is also better for our system and is only one third of the latency of the software consensus implementation when both systems are under half of their maximum throughputs. In order to drive even higher performance, we apply a partition mechanism to our switch-based system, leading to 11 times better throughput and 5 times better latency. By dynamically switching between software-based and network-based implementations, our consensus systems not only improve performance but also use energy more efficiently. Encouraged by those benefits, we developed a fault-tolerant non-volatile memory system. A prototype using software memory controller demonstrated reasonable overhead over local memory access, showing great promise as scalable main memory. Our network-based consensus approach would have a great impact in data centers. It not only improves performance of replication mechanisms which relied on consensus, but also enhances performance of services built on top of those replication mechanisms. Our approach also motivates others to move new functionalities into the network, such as, key-value store and stream processing. We expect that in the near future, applications that typically run on traditional servers will be folded into networks for performance

Blockchain-Based Digitalization of Logistics Processes—Innovation, Applications, Best Practices

Blockchain technology is becoming one of the most powerful future technologies in supporting logistics processes and applications. It has the potential to destroy and reorganize traditional logistics structures. Both researchers and practitioners all over the world continuously report on novel blockchain-based projects, possibilities, and innovative solutions with better logistic service levels and lower costs. The idea of this Special Issue is to provide an overview of the status quo in research and possibilities to effectively implement blockchain-based solutions in business practice. This Special Issue reprint contained well-prepared research reports regarding recent advances in blockchain technology around logistics processes to provide insights into realized maturity

Physiological ecology of Trichodesmium and its microbiome in the oligotrophic ocean

The colonial, N2 fixing cyanobacterium Trichodesmium is a keystone species in oligotrophic ocean ecosystems. Trichodesmium is responsible for approximately 50% of the total biologically fixed N2 in the ocean, and this “new” nitrogen fuels primary productivity and the amount of carbon sequestered by the ocean. Trichodesmium does not exist in isolation. Colonies occur ubiquitously with an assemblage of epibiotic microorganisms that are distinct from planktonic microbes and modulated across environments, yet the implications of this relationship have not been explored. In this thesis, the ecology, physiology, and potential geochemical impact of interactions within the Trichodesmium host-microbiome system were examined across three different oligotrophic ocean environments. First, to establish the metabolic diversity contributed by the microbiome to Trichodesmium consortia, a whole community metagenomic sequencing approach was used across a transect the western North Atlantic. This study demonstrated that the microbiome contributes a large amount of unique functional potential and is modulated across a geochemical gradient. In the following study, metatranscriptomics was used to show that such metabolic potential in Trichodesmium and the microbiome was expressed and modulated across the environment. Colonies were sampled in the western tropical South Pacific and gene expression dynamics indicated co-limitation by iron and phosphorus, and revealed a mechanism for phosphate reduction by Trichodesmium and subsequent utilization by the microbiome. These activities were verified with phosphate reduction rate measurements and indicated cryptic phosphorus cycling within colonies. Next, the suite of potential physiological interactions between host and microbiome was assessed with metatranscriptome sequencing on high frequency samples of Trichodesmium colonies from the North Pacific subtropical gyre. Synchronized day-night gene expression periodicity between consortia members indicated tightly linked metabolisms. The functional annotations of these synchronous genes indicated intra-consortia cycling of nitrogen, phosphorus and iron, as well as a microbiome dependence on Trichodesmium-derived cobalamin—interactions that could alter the transfer of these resources to the surrounding water column. In the final study, the effect of the microbiome on Trichodesmium N2 fixation was assessed. Using colonies obtained from the North Atlantic, activity in the microbiome was selectively modified using quorum sensing acyl homoserine lactone cell-cell signaling, a mechanism that Trichodesmium itself does not possess. These experiments indicated that the microbiome has the potential to increase or decrease Trichodesmium N2 fixation to a degree that rivals the effects of alterations in nutrient concentration, but at a more rapid rate. In all, the research presented in this thesis demonstrates the integral importance of the microbiome to Trichodesmium physiology and ecology, highlighting the importance of an unexplored facet of marine microbial systems that likely influences the biogeochemistry of the planet

Development of a reference architecture for the design of tracking and tracing solutions in the supply chain using blockchain technology

Thesis (MEng)--Stellenbosch University, 2020.ENGLISH ABSTRACT: This masters thesis involves the development of a design reference architecture for the use of blockchain technology in the supply chain. It aims to further the research on blockchains’ use and development, specifically with regards to tracking and tracing. Blockchain is a revolutionary new technology, capable of disrupting today’s supply chains. Originally envisioned for finance, the technology behind the Bitcoin cryptocurrency, blockchain can maintain an immutable, secure and trusted record of transactions between a network of individuals, who do not necessarily trust one other. This makes it highly applicable to supply chains as it can enable the secure sharing of information between different parties. Supply chains are facing problems due to a lack of information transparency and trust. Traditional supply chains contain a combination of paper-based processes and digital systems, where information is held up in ‘silos’. Lack of end-to-end process integration and information sharing, results in problems regarding tracking and tracing of products. Tracking and tracing is key to digital supply chains as changing consumer behaviour and market conditions demand higher supply chain visibility. Traditional methods are, however, slow, fragmented and ineffective. This makes it problematic to validate a products origin, location or specific characteristics. Blockchain has the potential to facilitate trust and enable end-to-end supply chain visibility. A lack of academic research has caused a paucity of knowledge on the use and implementation of blockchain in the supply chain. Supply chain companies do not understand blockchain, including the various aspects involved in its design. There are currently no identified frameworks, models or architectures that aid in supporting its use in supply chain applications. The main aim of this project is to construct a reference architecture that can support the design of blockchain in the supply chain. Reference architectures define recommended practices and guidelines, that are used as a frame of reference to improve the quality, speed and cost of the information modelling process. Blockchain technology was investigated, its latest technological architectures were identified and its role in supply chain digitalization was defined. Aided by existing reference architectures, a methodology was formed for designing a blockchain reference architecture. The important blockchain design requirements were identified and extracted from blockchain frameworks, case studies and technical architectures. The design reference architecture, for the use of blockchain technology in the supply chain, was constructed. It contains the relevant design guidelines relating to strategy, feasibility, technology, supply chain and product sub-components. It serves as a guide that can be consulted by companies in order to aid decision making. A practical case study application in the FMCG foods supply chain, along with semi structured interviews with industry experts, tested the validity and practicality of the reference architecture. The results motivated that the architecture formalized a collection of knowledge, aiding in the understanding and effective decision making, of blockchain in the supply chain. The design reference architecture furthers knowledge on the application of blockchain technology in the supply chain, and provides a basis for future research and development.AFRIKAANSE OPSOMMING: Hierdie meesterstesis behels die ontwikkeling van ’n ontwerp-verwysingsargitektuur vir die gebruik van blokketting-tegnologie in die voorsieningsketting. Die projek se doelwit is om die navorsing en gebruik van blokketting-tegnologie in die voorsieningsketting te bevorder, met spesifieke klem op die naspeurbaarheid van produkte. Blokketting, ’n rewolusionêre nuwe tegnologie met die potentisiaal om die huidige voorsieningskettings te ontwrig, is oorspronklik ontwikkel vir finansiele toepassings, as die tegnologie waarop die Bitcoin kriptogeldeenheid gebaseer is. Blokketting het die vermoë om ’n onveranderlike, veilige en betroubare rekord van transaskies te behou, tussen ’n network van individue wat mekaar nie noodwendig hoef te vertrou nie. Dit beteken dat hierdie tegnologie die geleenthied skep vir volle naspeurbaarheid binne enige voorsieningsketting. Huidige voorsieningskettings ervaar probleme as gevolg van ’n gebrek aan inligtingsdeursigtigheid en vertroue. Tradisionele voorsieningskettings gebruik ’n kombinasie van papier-gebaseerde prosesse en digitale stelsels, wat veroorsaak dat inligting in ‘silos’ vasgehou word. Die gebrek aan gedeelde inligting, veroorsaak probleme met die naspeurbaarheid van produkte. Naspeurbaarheid is ’n belangrike komponent van digitale voorsieningskettings, aangesien dit die deursigtigheid van informasie kan verseker. Huidige naspeur metodes is stadig en ondoeltreffend. Dit bemoeilik dus die bevestiging van die oorsprong en kwalitiet van produkte. Blokketting het die vermoë om vertroue in sovel as deursigtighied van die voorsieningsketting te bewerkstellig. Daar is tans ’n gebrek aan akademiese navorsing rondom blokketting-tegnologie wat lei tot ’n gebrek aan kennis oor die gebruik en implementering daarvan in die voorsieningsketting. Daar is tans geen geïdentifiseerde raamwerke, modelle of argitekture, wat die gebruik daarvan in die voorsieningsketting ondersteun nie. Die hoofdoel van die projek is om ’n verwysingsargitektuur te bou wat die ontwerpproses van blokkettingstelsels kan ondersteun. Verwysingsargitekture bevat aanbevole praktyke en riglyne wat gebruik kan word as ’n verwysing om die gehalte, spoed en koste van ’n inligtingstesel ontwerp te verbeter. Die blokketting-tegnologie is nagevors, die nuutste tegnologiese argitekture is geïdentifiseer en die rol daarvan in voorsieningsketting-digitalisering is omskryf. Met behulp van bestaande verwysingsargitekture, is ’n metodologie ontwikkel om ’n blokketting verwysingsargitektuur te ontwerp. Die belangrike vereistes vir die ontwerp van ’n blokketting stelsel geïdentifiseer is vanuit navorsing oor huidige blokketting raamwerke, gevallestudies en tegniese argitekture. Die ontwerpversysingsargitektuur vir die gebruik van blokketting-tegnologie in die voorsieningsketting is gebou. Dit bevat relevante ontwerp riglyne met betrekking tot strategie, tegnologie, uitvoerbaarheid, voorsieningsketting en produk komponente. Dit dien as ’n gids wat deur ondernemings geraadpleeg kan word om besluitneming te ondersteun. ’n Praktiese gevallestudietoepassing in die voedselvoorsieningsketting, tesame met semi-gestruktureerde onderhoude met kundiges in die bedryf, het die geldigheid en praktiese toepassing van die verwysingsargitektuur getoets. Die resultate motiveer dat die argitektuur die nodige inligting bevat wat bydrae kan lewer tot beter begrip en doeltreffende besluitneming van blokketting-tegnologie in die voorsieningsketting. Hierdie ontwerpverwysingsargitektuur bevorder kennis oor die toepassing van blokketting-tegnologie in die voorsieningsketting, en bied ’n basis vir toekomstige navorsing en ontwikkeling.Master