Improving lifecycle query in integrated toolchains using linked data and MQTT-based data warehousing

The development of increasingly complex IoT systems requires large engineering environments. These environments generally consist of tools from different vendors and are not necessarily integrated well with each other. In order to automate various analyses, queries across resources from multiple tools have to be executed in parallel to the engineering activities. In this paper, we identify the necessary requirements on such a query capability and evaluate different architectures according to these requirements. We propose an improved lifecycle query architecture, which builds upon the existing Tracked Resource Set (TRS) protocol, and complements it with the MQTT messaging protocol in order to allow the data in the warehouse to be kept updated in real-time. As part of the case study focusing on the development of an IoT automated warehouse, this architecture was implemented for a toolchain integrated using RESTful microservices and linked data.Comment: 12 pages, worksho

A unified view of data-intensive flows in business intelligence systems : a survey

Data-intensive flows are central processes in today’s business intelligence (BI) systems, deploying different technologies to deliver data, from a multitude of data sources, in user-preferred and analysis-ready formats. To meet complex requirements of next generation BI systems, we often need an effective combination of the traditionally batched extract-transform-load (ETL) processes that populate a data warehouse (DW) from integrated data sources, and more real-time and operational data flows that integrate source data at runtime. Both academia and industry thus must have a clear understanding of the foundations of data-intensive flows and the challenges of moving towards next generation BI environments. In this paper we present a survey of today’s research on data-intensive flows and the related fundamental fields of database theory. The study is based on a proposed set of dimensions describing the important challenges of data-intensive flows in the next generation BI setting. As a result of this survey, we envision an architecture of a system for managing the lifecycle of data-intensive flows. The results further provide a comprehensive understanding of data-intensive flows, recognizing challenges that still are to be addressed, and how the current solutions can be applied for addressing these challenges.Peer ReviewedPostprint (author's final draft

The Indonesian digital library network is born to struggle with the digital divide

IndonesiaDLN –The Indonesian Digital Library Network– is a distributed collection of digital library networks, digital library servers, full local contents, metadata, and people for the development of the Indonesian knowledge-based society. Beside the general issues of digital library such as publishing, quality control, authentication, networking, and information retrieval, we also face other issue –namely digital divide– in designing and implementing the Network. This paper describes basic design of the Network that able to handle the typical problems in developing digital library network in Indonesia as a developing country, such as internet accessibility, bandwidth capacity, and network delays. We also will describe our experiences in implementing the Network that currently has 14 successfully connected partners and more than 15 partners are in progress of developing their digital library servers

Model-driven Engineering IDE for Quality Assessment of Data-intensive Applications

This article introduces a model-driven engineering (MDE) integrated development environment (IDE) for Data-Intensive Cloud Applications (DIA) with iterative quality enhancements. As part of the H2020 DICE project (ICT-9-2014, id 644869), a framework is being constructed and it is composed of a set of tools developed to support a new MDE methodology. One of these tools is the IDE which acts as the front-end of the methodology and plays a pivotal role in integrating the other tools of the framework. The IDE enables designers to produce from the architectural structure of the general application along with their properties and QoS/QoD annotations up to the deployment model. Administrators, quality assurance engineers or software architects may also run and examine the output of the design and analysis tools in addition to the designer in order to assess the DIA quality in an iterative process

Data Warehousing Modernization: Big Data Technology Implementation

Considering the challenges posed by Big Data, the cost to scale traditional data warehouses is high and the performances would be inadequate to meet the growing needs of the volume, variety and velocity of data. The Hadoop ecosystem answers both of the shortcomings. Hadoop has the ability to store and analyze large data sets in parallel on a distributed environment but cannot replace the existing data warehouses and RDBMS systems due to its own limitations explained in this paper. In this paper, I identify the reasons why many enterprises fail and struggle to adapt to Big Data technologies. A brief outline of two different technologies to handle Big Data will be presented in this paper: Using IBM’s Pure Data system for analytics (Netezza) usually used in reporting, and Hadoop with Hive which is used in analytics. Also, this paper covers the Enterprise architecture consisting of Hadoop that successful companies are adapting to analyze, filter, process, and store the data running along a massively parallel processing data warehouse. Despite, having the technology to support and process Big Data, industries are still struggling to meet their goals due to the lack of skilled personnel to study and analyze the data, in short data scientists and data statisticians

Exploring the Future Shape of Business Intelligence: Mapping Dynamic Capabilities of Information Systems to Business Intelligence Agility

A major challenge in today’s turbulent environments is to make appropriate decisions to sustainably steer an organization. Business intelligence (BI) systems are often used as a basis for decision making. But achieving agility in BI and cope with dynamic environments is no trivial endeavor as the classical, data-warehouse (DWH)-based BI is primarily used to retrospectively reflect an organization’s performance. Using an exploratory approach, this paper investigates how current trends affect the concept of BI and thus their ability to support adequate decision making. The key focus is to understand dynamic capabilities in the field of information systems (IS) and how they are connected to BI agility. We therefore map dynamic capabilities from the IS literature to agility dimensions of BI. Additionally, we propose a structural model that focusses on DWH-based BI and analyze how current BI-related trends and environmental turbulence affect the way that BI is shaped in the future

Augmenting data warehousing architectures with Hadoop

As the volume of available data increases exponentially, traditional data warehouses struggle to transform this data into actionable knowledge. This study explores the potentialities of Hadoop as a data transformation tool in the setting of a traditional data warehouse environment. Hadoop’s distributed parallel execution model and horizontal scalability offer great capabilities when the amounts of data to be processed require the infrastructure to expand. Through a typification of the SQL statements, responsible for the data transformation processes, we were able to understand that Hadoop, and its distributed processing model, delivers outstanding performance results associated with the analytical layer, namely in the aggregation of large data sets. We demonstrate, empirically, the performance gains that can be extracted from Hadoop, in comparison to a Relational Database Management System, regarding speed, storage usage, and scalability potential, and suggest how this can be used to evolve data warehouses into the age of Big Data