Dynamic Load Balancing and Autoscaling in Distributed Stream Processing Systems

In big data world, Hadoop and other batch-processing tools are widely used to analyze data and get results in minutes. However, minutes of latency still cannot satisfy the proliferated needs for real-time decision in many fields such as live stock and trading feeds in financial services, telecommunications, sensor networks, online advertisement, etc. Distributed stream processing (DSP) systems aim to process, analyze and make decisions on-the-fly based on immense quantities of data streams being dynamically generated at high rates. As the rates of data streams may vary over time, DSP systems require an architecture that is elastic to handle dynamic load. Although many dynamic load balancing and autoscaling techniques for general pull-based distributed systems have been well studied, these solutions cannot be directly applied to DSP systems because DSP systems are push-based, they process data streams with different types of operators, each running on a cluster node. One research problem is to allocate data processing operators on nodes of clusters and balance the workload dynamically. Since the data volume and rate can be unpredictable, static mapping between operators and cluster resources often results in unbalanced operator load distribution. Furthermore, the problem of making DSP system scalable requires autoscaling at runtime. In this context, the operators need to be relocated among newly provisioned nodes. The contribution of this thesis is three folds. First, we proposes a software layer that is load-adaptive between a DSP engine and clusters. The architecture allows dynamic transferring of an operator to different cluster nodes at runtime and keeps the process transparent to developers. Second, an optimization method that combines correlation of resource utilization of nodes and capacity of clusters is proposed to balance load dynamically. Lastly, we design the autoscaling mechanism and algorithm to detect overload and provision nodes at runtime. We implement our design on S4, an open-source DSP engine first developed by Yahoo!. The implementation is evaluated by a top-N topic list application on Twitter streams using clusters on Amazon Web Services. The results demonstrate a 75.79% improvement on stream processing throughputs, and a 294.47% improvement on cluster resource utilization

Real Time Hybrid Intrusion Detection System Using Apache Storm

Networks are prone to intrusions and detecting intruders on the internet is a major problem. Many Intrusion Detection Systems have been proposed to detect these intrusions. However, as the internet grows day by day, there is a huge amount of data (big data) that needs to be processed to detect intruders. For this reason, intrusion detection has to be done in real- time before intruders can inflict damage, and previous detection systems do not satisfy this need for big data.Using Apache Storm, a Real time Hybrid Intrusion Detection System has been developed in our thesis. Apache Storm serves as a distributed, fault tolerant, real time big data stream processor. The hybrid detection system consists of two neural networks. The CC4 instan- taneous neural network acts as an anomaly-based detection for unknown attacks and the Multi Layer Perceptron neural network acts as a misuse-based detection for known attacks. Based on the outputs from these two neural networks, the incoming data will be classified as �attack� or �normal.� We found the average accuracy of hybrid detection system is 89% with a 4.32% false positive rate. This model is appropriate for real time detection since Apache Storm acts as a real time streaming processor, which can also handle big data.Computer Scienc

Real-Time QoS Monitoring and Anomaly Detection on Microservice-based Applications in Cloud-Edge Infrastructure

Ph. D. Thesis.Microservices have emerged as a new approach for developing and deploying cloud applications that require higher levels of agility, scale, and reliability. A microservicebased cloud application architecture advocates decomposition of monolithic application components into independent software components called \microservices". As the independent microservices can be developed, deployed, and updated independently of each other, it leads to complex run-time performance monitoring and management challenges. The deployment environment for microservices in multi-cloud environments is very complex as there are numerous components running in heterogeneous environments (VM/container) and communicating frequently with each other using REST-based/REST-less APIs. In some cases, multiple components can also be executed inside a VM/container making any failure or anomaly detection very complicated. It is necessary to monitor the performance variation of all the service components to detect any reason for failure. Microservice and container architecture allows to design loose-coupled services and run them in a lightweight runtime environment for more e cient scaling. Thus, containerbased microservice deployment is now the standard model for hosting cloud applications across industries. Despite the strongest scalability characteristic of this model which opens the doors for further optimizations in both application structure and performance, such characteristic adds an additional level of complexity to monitoring application performance. Performance monitoring system can lead to severe application outages if it is not able to successfully and quickly detecting failures and localizing their causes. Machine learning-based techniques have been applied to detect anomalies in microservice-based cloud-based applications. The existing research works used di erent tracking algorithms to search the root cause if anomaly observed behaviour. However, linking the observed failures of an application with their root causes by the use of these techniques is still an open research problem. Osmotic computing is a new IoT application programming paradigm that's driven by the signi cant increase in resource capacity/capability at the network edge, along with support for data transfer protocols that enable such resources to interact more seamlessly with cloud-based services. Much of the di culty in Quality of Service (QoS) and performance monitoring of IoT applications in an osmotic computing environment is due to the massive scale and heterogeneity (IoT + edge + cloud) of computing environments. To handle monitoring and anomaly detection of microservices in cloud and edge datacenters, this thesis presents multilateral research towards monitoring and anomaly detection on microservice-based applications performance in cloud-edge infrastructure. The key contributions of this thesis are as following: • It introduces a novel system, Multi-microservices Multi-virtualization Multicloud monitoring (M3 ) that provides a holistic approach to monitor the performance of microservice-based application stacks deployed across multiple cloud data centers. • A framework forMonitoring, Anomaly Detection and Localization System (MADLS) which utilizes a simpli ed approach that depends on commonly available metrics o ering a simpli ed deployment environment for the developer. • Developing a uni ed monitoring model for cloud-edge that provides an IoT application administrator with detailed QoS information related to microservices deployed across cloud and edge datacenters.Royal Embassy of Saudi Arabia Cultural Bureau in London, government of Saudi Arabi

Priorities in collective health research in Latin America

Spanish version available in IDRC Digital Library: Prioridades en la investigación de la salud colectiva en América Latin

Innovation, Internationalization and Entrepreneurship

Over the past years, businesses have had to tackle the issues caused by numerous forces from political, technological and societal environment. The changes in the global market and increasing uncertainty require us to focus on disruptive innovations and to investigate this phenomenon from different perspectives. The benefits of innovations are related to lower costs, improved efficiency, reduced risk, and better response to the customers’ needs due to new products, services or processes. On the other hand, new business models expose various risks, such as cyber risks, operational risks, regulatory risks, and others. Therefore, we believe that the entrepreneurial behavior and global mindset of decision-makers significantly contribute to the development of innovations, which benefit by closing the prevailing gap between developed and developing countries. Thus, this Special Issue contributes to closing the research gap in the literature by providing a platform for a scientific debate on innovation, internationalization and entrepreneurship, which would facilitate improving the resilience of businesses to future disruptions

Context-aware mobile learning on the semantic web / by Xiaoyun Zhang.

Progress made in Semantic Web technologies and Ubiquitous Computing has lead to the development of mobile learning services that can adapt to the learner's background, learner's needs, and surrounding environment. In particular, the emerging techniques from these two technologies have the potential to revolutionize the way mobile learning services available on the web are discovered, adapted, and delivered according to context. Context acquisition and management, conceptual knowledge modeling and reasoning, and adaptive services discovery are the main ingredients for designing such context-aware mobile learning systems. However, a number of challenges are still facing the research community in this field. These can be summarized in the following: (i) current mobile learning services act as passive components rather than active components that can be embedded with context awareness mechanisms, (ii) existing approaches for service composition neglect contextual information on surrounding environment, and (iii) lack of context modeling and reasoning techniques for integrating the various contextual features for better personalization. In this thesis an attempt is made to solve the above-mentioned problems. These challenges are addressed by proposing a personalized mobile learning system based on a global ontology space to aggregate and manage context information related to the learner, the used device, the surrounding environment, and the task at hand. The system adopts a unified reasoning mechanism, around the global ontology space, in order to adapt the learning sequence and the learning content based on the learner profile and the perceived contextual information. The adopted approach for ontology reasoning aims at achieving two types of adaptations--system-centric adaptation and--learner-centric adaptation. These are implemented on a Run-Time Environment that identifies new contextual changes and translates them into new adaptation constraints. We developed and tested our system on a number of subject-domain ontologies using various learning scenarios, and the obtained experimental results are very promising

Sustainable Business Models

The dynamically changing world economy, in an era of intensive development and globalization, creates new needs in both the theoretical models of management and in the practical discussion related to the perception of business. Because of new economic phenomena related to the crisis, there is a need for the design and operationalization of innovative business models for companies. Due to the fact that in times of crisis, the principles of strategic balance are particularly important; these business models can be sustainable business models. Moreover, it is essential to skillfully use different methods and concepts of management to ensure the continuity of business. It seems that sustainable business models, in their essence, can support companies' effectiveness and contribute to their stable, sustainable functioning in the difficult, ever-changing market. This Special Issue aims to discuss the key mechanisms concerning the design and operationalization of sustainable business models, from a strategic perspective. We invite you to contribute to this Issue by submitting comprehensive reviews, case studies, or research articles. Papers selected for this Special Issue are subject to a rigorous peer review procedure, with the aim of rapid and wide dissemination of research results, developments, and applications