Real-Time Data Processing With Lambda Architecture

Data has evolved immensely in recent years, in type, volume and velocity. There are several frameworks to handle the big data applications. The project focuses on the Lambda Architecture proposed by Marz and its application to obtain real-time data processing. The architecture is a solution that unites the benefits of the batch and stream processing techniques. Data can be historically processed with high precision and involved algorithms without loss of short-term information, alerts and insights. Lambda Architecture has an ability to serve a wide range of use cases and workloads that withstands hardware and human mistakes. The layered architecture enhances loose coupling and flexibility in the system. This a huge benefit that allows understanding the trade-offs and application of various tools and technologies across the layers. There has been an advancement in the approach of building the LA due to improvements in the underlying tools. The project demonstrates a simplified architecture for the LA that is maintainable

Developing a distributed electronic health-record store for India

The DIGHT project is addressing the problem of building a scalable and highly available information store for the Electronic Health Records (EHRs) of the over one billion citizens of India

Permission-based fault tolerant mutual exclusion algorithm for mobile Ad Hoc networks

This study focuses on resolving the problem of mutual exclusion in mobile ad hoc networks. A Mobile Ad Hoc Network (MANET) is a wireless network without fixed infrastructure. Nodes are mobile and topology of MANET changes very frequently and unpredictably. Due to these limitations, conventional mutual exclusion algorithms presented for distributed systems (DS) are not applicable for MANETs unless they attach to a mechanism for dynamic changes in their topology. Algorithms for mutual exclusion in DS are categorized into two main classes including token-based and permission-based algorithms. Token-based algorithms depend on circulation of a specific message known as token. The owner of the token has priority for entering the critical section. Token may lose during communications, because of link failure or failure of token host. However, the processes for token-loss detection and token regeneration are very complicated and time-consuming. Token-based algorithms are generally non-fault-tolerant (although some mechanisms are utilized to increase their level of fault-tolerance) because of common problem of single token as a single point of failure. On the contrary, permission-based algorithms utilize the permission of multiple nodes to guarantee mutual exclusion. It yields to high traffic when number of nodes is high. Moreover, the number of message transmissions and energy consumption increase in MANET by increasing the number of mobile nodes accompanied in every decision making cycle. The purpose of this study is to introduce a method of managing the critical section,named as Ancestral, having higher fault-tolerance than token-based and fewer message transmissions and traffic rather that permission-based algorithms. This method makes a tradeoff between token-based and permission-based. It does not utilize any token, that is similar to permission-based, and the latest node having the critical section influences the entrance of the next node to the critical section, that is similar to token-based algorithms. The algorithm based on ancestral is named as DAD algorithms and increases the availability of fully connected network between 2.86 to 59.83% and decreases the number of message transmissions from 4j-2 to 3j messages (j as number of nodes in partition). This method is then utilized as the basis of dynamic ancestral mutual exclusion algorithm for MANET which is named as MDA. This algorithm is presented and evaluated for different scenarios of mobility of nodes, failure, load and number of nodes. The results of study show that MDA algorithm guarantees mutual exclusion,dead lock freedom and starvation freedom. It improves the availability of CS to minimum 154.94% and 113.36% for low load and high load of CS requests respectively compared to other permission-based lgorithm.Furthermore, it improves response time up to 90.69% for high load and 75.21% for low load of CS requests. It degrades the number of messages from n to 2 messages in the best case and from 3n/2 to n in the worst case. MDA algorithm is resilient to transient partitioning of network that is normally occurs due to failure of nodes or links