A microservice architecture for the processing of large geospatial data in the Cloud

With the growing number of devices that can collect spatiotemporal information, as well as the improving quality of sensors, the geospatial data volume increases constantly. Before the raw collected data can be used, it has to be processed. Currently, expert users are still relying on desktop-based Geographic Information Systems to perform processing workflows. However, the volume of geospatial data and the complexity of processing algorithms exceeds the capacities of their workstations. There is a paradigm shift from desktop solutions towards the Cloud, which offers virtually unlimited storage space and computational power, but developers of processing algorithms often have no background in computer science and hence no expertise in Cloud Computing. Our research hypothesis is that a microservice architecture and Domain-Specific Languages can be used to orchestrate existing geospatial processing algorithms, and to compose and execute geospatial workflows in a Cloud environment for efficient application development and enhanced stakeholder experience. We present a software architecture that contains extension points for processing algorithms (or microservices), a workflow management component for distributed service orchestration, and a workflow editor based on a Domain-Specific Language. The main aim is to provide both users and developers with the means to leverage the possibilities of the Cloud, without requiring them to have a deep knowledge of distributed computing. In order to conduct our research, we follow the Design Science Research Methodology. We perform an analysis of the problem domain and collect requirements as well as quality attributes for our architecture. To meet our research objectives, we design the architecture and develop approaches to workflow management and workflow modelling. We demonstrate the utility of our solution by applying it to two real-world use cases and evaluate the quality of our architecture based on defined scenarios. Finally, we critically discuss our results. Our contributions to the scientific community can be classified into three pillars. We present a scalable and modifiable microservice architecture for geospatial processing that supports distributed development and has a high availability. Further, we present novel approaches to service integration and orchestration in the Cloud as well as rule-based and dynamic workflow management without a priori design-time knowledge. For the workflow modelling we create a Domain-Specific Language that is based on a novel language design method

Real-time satellite data processing platform architecture

Remote sensing satellites produce massive amounts of data of the earth every day. This earth observation data can be used to solve real world problems in many different fields. Finnish space data company Terramonitor has been using satellite data to produce new information for its customers. The Process for producing valuable information includes finding raw data, analysing it and visualizing it according to the client’s needs. This process contains a significant amount of manual work that is done at local workstations. Because satellite data can quickly become very big, it is not efficient to use unscalable processes that require lot of waiting time. This thesis is trying to solve the problem by introducing an architecture for cloud based real-time processing platform that allows satellite image analysis to be done in cloud environment. The architectural model is built using microservice patterns to ensure that the solution is scalable to match the changing demand

An Analytics Platform for Integrating and Computing Spatio-Temporal Metrics

In large-scale context-aware applications, a central design concern is capturing, managing and acting upon location and context data. The ability to understand the collected data and define meaningful contextual events, based on one or more incoming (contextual) data streams, both for a single and multiple users, is hereby critical for applications to exhibit location- and context-aware behaviour. In this article, we describe a context-aware, data-intensive metrics platform —focusing primarily on its geospatial support—that allows exactly this: to define and execute metrics, which capture meaningful spatio-temporal and contextual events relevant for the application realm. The platform (1) supports metrics definition and execution; (2) provides facilities for real-time, in-application actions upon metrics execution results; (3) allows post-hoc analysis and visualisation of collected data and results. It hereby offers contextual and geospatial data management and analytics as a service, and allow context-aware application developers to focus on their core application logic. We explain the core platform and its ecosystem of supporting applications and tools, elaborate the most important conceptual features, and discuss implementation realised through a distributed, micro-service based cloud architecture. Finally, we highlight possible application fields, and present a real-world case study in the realm of psychological health

Design and Implementation of a Scalable Crowdsensing Platform for Geospatial Data of Tinnitus Patients

Smart devices and low-powered sensors are becoming increasingly ubiquitous and nowadays almost all of these devices are connected, which is a promising foundation for crowdsensing of data related to various environmental phenomena. Resulting data is especially meaningful when it is related to time and location. Interestingly, many existing approaches built their solution on monolithic backends that process data on a per-request basis. However, for many scenarios, such technical setting is not suitable for managing data requests of a large crowd. For example, when dealing with millions of data points, still many challenges arise for modern smartphones if calculations or advanced visualization features must be accomplished directly on the smartphone. Therefore, the work at hand proposes an architectural design for managing geospatial data of tinnitus patients, which combines a cloudnative approach with Big Data concepts used in the Internet of Things. The presented architectural design shall serve as a generic foundation to implement (1) a scalable backend for a platform that covers the aforementioned crowdsensing requirements as well as to provide (2) a sophisticated stream processing concept to calculate and pre-aggregate incoming measurement data of tinnitus patients. Following this, this paper presents a visualization feature to provide users with a comprehensive overview of noise levels in their environment based on noise measurements. This shall help tinnitus or hearing-impaired patients to avoid locations with a burdensome sound level

Service Integration Design Patterns in Microservices

“Microservices” is a new term in software architecture that was defined in 2014 [1]. It is a method to build a software application with a set of small services. Each service has its process to serve a single purpose and communicates with other services through lightweight mechanisms. Because of a great deal of independently distributed services, it is a challenge to integrate the loose services fully. Too many trivial relationships can be messed up easily during deployment. Also, it is hard to modify the relationships if the services are updated as the source codes need to be re-edited and tested. The microservices architecture is attracting much attention recently. More and more software-developers are producing continuous applications and microservices deliveries [2]. There is a need to develop a mechanism to better integrate the scattered services during the application delivery process. The thesis proposes three general design patterns to integrate services in microservices architecture. These patterns are classified by the inter-service communication mechanisms and described with specific problems, contexts, solutions, example implementations and consequences. Besides, the informative guidelines are provided to make these patterns apply in different applications quickly. The service integration design patterns help compose services and facilitate the process of building applications in microservices. All the patterns are helpful tools to address the service integration issues in microservices. Each approach is simple and flexible to apply generally. The structures can be easily modified through these approaches

Dependable workflow management system for smart farms

Smart Farming is a new and emerging domain representing the application of modern technologies into agriculture, leading to a revolution of this classic domain. CLUeFARM is a web platform in the domain of smart farming which main purpose is to help farmers to easily manage and supervise their farms from any device connected to the Internet, offering some useful services. Cloud technologies evolved a lot in recent years and based on this growth, microservices are more and more used. If for the server side, the scalability and reusability are solved in high proportion by microservices, on the client side of web applications, there was no independent solution until the recent emergence of web components. They can be seen as the microservices of the front-end. Microservices and web components are usually used isolated one of each other. This paper proposes and presents the functionality and implementation of a dependable workflow management service by using an end-to-end microservices approach

Design and Implementation of a Scalable Crowdsensing Platform for Geospatial Data

In the recent years smart devices and small low-powered sensors are becoming ubiquitous and nowadays everything is connected altogether, which is a promising foundation for crowdsensing of data related to various environmental and societal phenomena. Very often, such data is especially meaningful when related to time and location, which is possible by already equipped GPS capabilities of modern smart devices. However, in order to gain knowledge from high-volume crowd-sensed data, it has to be collected and stored in a central platform, where it can be processed and transformed for various use cases. Conventional approaches built around classical relational databases and monolithic backends, that load and process the geospatial data on a per-request basis are not suitable for supporting the data requests of a large crowd willing to visualize phenomena. The possibly millions of data points introduce challenges for calculation, data-transfer and visualization on smartphones with limited graphics performance. We have created an architectural design, which combines a cloud-native approach with Big Data concepts used in the Internet of Things. The architectural design can be used as a generic foundation to implement a scalable backend for a platform, that covers aspects important for crowdsensing, such as social- and incentive features, as well as a sophisticated stream processing concept to calculate incoming measurement data and store pre-aggregated results. The calculation is based on a global grid system to index geospatial data for efficient aggregation and building a hierarchical geospatial relationship of averaged values, that can be directly used to rapidly and efficiently provide data on requests for visualization. We introduce the Noisemap project as an exemplary use case of such a platform and elaborate on certain requirements and challenges also related to frontend implementations. The goal of the project is to collect crowd-sensed noise measurements via smartphones and provide users information and a visualization of noise levels in their environment, which requires storing and processing in a central platform. A prototypic implementation for the measurement context of the Noisemap project is showing that the architectural design is indeed feasible to realize

Bonus Computing: An Evolution from and a Supplement to Volunteer Computing

Despite the huge success in various worldwide projects, volunteer computing also suffers from the possible lack of computing resources (one volunteered device can join one project at a time) and from the uncertain job interruptions (the volunteered device can crash or disconnect from the Internet at any time). To relieve the challenges faced by volunteer computing, we have proposed bonus computing that exploits the free quotas of public Cloud resources particularly to deal with problems composed of fine-grained, short-running, and compute-intensive tasks. In addition to explaining the loosely-coupled functional architecture and six architectural patterns of bonus computing in this paper, we also employ the Monte-Carlo approximation of Pi (π) as a use case demonstration both to facilitate understanding and to help validate its functioning mechanism. The results exhibit not only effectiveness but also multiple advantages of bonus computing, which makes it a valuable evolution from and supplement to volunteer computing