MOMIS Dashboard: a powerful data analytics tool for Industry 4.0

In this work we present the MOMIS Dashboard, an interactive data analytics tool to explore and visualize data sources content through several kind of dynamic views (e.g. maps, bar, line, pie, etc.). The software tool is very versatile, and supports the connection to the main relational DBMS and Big Data sources. Moreover, it can be connected to MOMIS, a powerful Open Source Data Integration system, able to integrate heterogeneous data sources as enterprise information systems as well as sensors data. MOMIS Dashboard provides a secure permission management to limit data access on the basis of a user role, and a Designer to create and share personalized insights on the company KPIs, facilitating the enterprise collaboration. We illustrate the MOMIS Dashboard efficacy in a real enterprise scenario: a production monitoring platform to analyze real-time and historical data collected through sensors located on production machines that optimize production, energy consumption, and enable preventive maintenance

Blockchain based Access Control for Enterprise Blockchain Applications

Access control is one of the fundamental security mechanisms of IT systems. Most existing access control schemes rely on a centralized party to manage and enforce access control policies. As blockchain technologies, especially permissioned networks, find more applicability beyond cryptocurrencies in enterprise solutions, it is expected that the security requirements will increase. Therefore, it is necessary to develop an access control system that works in a decentralized environment without compromising the unique features of a blockchain. A straightforward method to support access control is to deploy a firewall in front of the enterprise blockchain application. However, this approach does not take advantage of the desirable features of blockchain. In order to address these concerns, we propose a novel blockchain‐based access control scheme, which keeps the decentralization feature for access control–related operations. The newly proposed system also provides the capability to protect user\u27s privacy by leveraging ring signature. We implement a prototype of the scheme using Hyperledger Fabric and assess its performance to show that it is practical for real‐world applications

On Efficiently Partitioning a Topic in Apache Kafka

Apache Kafka addresses the general problem of delivering extreme high volume event data to diverse consumers via a publish-subscribe messaging system. It uses partitions to scale a topic across many brokers for producers to write data in parallel, and also to facilitate parallel reading of consumers. Even though Apache Kafka provides some out of the box optimizations, it does not strictly define how each topic shall be efficiently distributed into partitions. The well-formulated fine-tuning that is needed in order to improve an Apache Kafka cluster performance is still an open research problem. In this paper, we first model the Apache Kafka topic partitioning process for a given topic. Then, given the set of brokers, constraints and application requirements on throughput, OS load, replication latency and unavailability, we formulate the optimization problem of finding how many partitions are needed and show that it is computationally intractable, being an integer program. Furthermore, we propose two simple, yet efficient heuristics to solve the problem: the first tries to minimize and the second to maximize the number of brokers used in the cluster. Finally, we evaluate its performance via large-scale simulations, considering as benchmarks some Apache Kafka cluster configuration recommendations provided by Microsoft and Confluent. We demonstrate that, unlike the recommendations, the proposed heuristics respect the hard constraints on replication latency and perform better w.r.t. unavailability time and OS load, using the system resources in a more prudent way.Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. This work was funded by the European Union's Horizon 2020 research and innovation programme MARVEL under grant agreement No 95733

Построение архитектуры интеллектуальной системы управления городской рельсовой транспортной системой

The increase in the volume of passenger transportation in megalopolises and large urban agglomerations is efficiently provided by the integration of urban public transit systems and city railways. Traffic management under those conditions requires creating intelligent centralised multi-level traffic control systems that implement the required indicators of quality, comfort, and traffic safety regarding passenger transportation. Besides, modern control systems contribute to traction power saving, are foundation and integral part of the systems of digitalisation of urban transit and the cities. Building systems solving the traffic planning and control tasks is implemented using algorithms based on the methods of artificial intelligence, principles of hierarchically structured centralised systems, opportunities provided by Big Data technology. Under those conditions it is necessary to consider growing requirements towards software as well as theoretical design and practical implementation of network organisation.This article discusses designing architecture and shaping requirements for developed applications and their integration with databases to create a centralised intelligent control system for the urban rail transit system (CICS URTS). The article proposes the original architecture of the network, routing of information flows and software of CICS URTS. The routing design is based on a fully connected network. This allows to significantly increase the network bandwidth and meet the requirements regarding information protection, since information flows are formed based on the same type of protocols, which prevents emergence of covert transmission channels. The implementation of the core using full connectivity allows, according to the tags of information flows, to pre-form the routes for exchange of information between servers and applications deployed in CICS URTS. The use of encrypted tags of information flows makes it much more difficult to carry out attacks and organise collection of information about the network structure.Platforms for development of intelligent control systems (ICS), which include CICS URTS, high computing power, data storage capacity and new frameworks are becoming more available for researchers and developers and allow rapid development of ICS. The article discusses the issues of interaction of applications with databases through a combination of several approaches used in the field of Big Data, substantiates combination of Internet of Things (IoT) methodology and microservice architecture. This combination will make it possible to single out business processes in the system and form streaming data processing requiring operational analysis by a human, which is shown by relevant examples.Thus, the objective of the article is to formalise the principles of organising data exchange between CICS URTS and automated control systems (ACS) of railway companies (in our case, using the example of JSC Russian Railways), URTS services providers, and city government bodies, implement the developed approaches into the architecture of CICS URTS and formalise principles of organisation of microservice architecture of CICS URTS software. The main research methods are graph theory, Big Data and IoT methods.Рост объёма пассажирских перевозок в условиях крупных городских агломераций и мегаполисов эффективно обеспечивается объединением общественного транспорта и городских линий железных дорог. Управление движением в этих условиях требует создания интеллектуальных централизованных многоуровневых систем управления, реализующих заданные показатели качества, комфорта и безопасности перевозок пассажиров. Современные системы управления дополнительно решают задачи экономии энергии на тягу подвижного состава, являются фундаментом и составной частью систем цифровизации городского транспорта и города в целом. Построение систем, решающих задачи планирования и управления движением, реализуется с применением алгоритмов, использующих методы искусственного интеллекта, принципы иерархического построения централизованных систем, возможности технологии Big Data. В этих условиях необходимо учитывать возросшие требования не только к программному обеспечению, но и к теоретическим и практическим решениям организации сети.В данной статье рассматриваются вопросы формирования архитектуры и требований к разрабатываемым приложениям и их интеграции с базами данных для создания централизованной интеллектуальной системы управления городской рельсовой транспортной системой (ЦИСУ ГРТС). В статье предлагаются оригинальные подходы к архитектуре сети, маршрутизации информационных потоков и программному обеспечению ЦИСУ ГРТС. В основе построения маршрутизации лежит использование полносвязной сети. Это позволяет значительно увеличить пропускную способность сети и выполнить требования по защите информации, так как информационные потоки формируются на базе однотипных протоколов, что препятствует образованию скрытых каналов передачи. Реализация ядра с использованием полносвязности позволяет по меткам информационных потоков заранее сформировать маршруты обмена информации между серверами и приложениями, развёрнутыми в ЦИСУ ГРТС. Использование шифрованных меток информационных потоков значительно усложняет проведение атак и организацию сбора информации о структуре сети.Платформы для разработки интеллектуальных систем управления (ИСУ), к которым относится ЦИСУ ГРТС, огромные вычислительные мощности, хранилища данных и новые фреймворки становятся всё более доступными для учёных и разработчиков и позволяют быстро развиваться ИСУ. В статье рассматриваются вопросы взаимодействия приложений с базами данных на основе комбинации нескольких подходов, используемых в области Big Data, обосновывается сочетание методологии интернета вещей (IoT) и микросервисной архитектуры. Данная комбинация позволит выделить в системе бизнес-процессы и сформировать потоковую обработку данных, требующих оперативного анализа человеком, что доказывается соответствующими примерами.Таким образом, целью статьи является формализация принципов организации информационного обмена ЦИСУ ГРТС и автоматизированных систем управления (АСУ) железнодорожных компаний (в нашем случае – на примере ОАО «РЖД»), организаций, предоставляющих услуги ГРТС, и городских органов управления, реализация сформулированных принципов в архитектуре ЦИСУ ГРТС и формализация принципов организации микросервисной архитектуры программного обеспечения ЦИСУ ГРТС. Основными методами исследования являются теория графов, методы Big Data, IoT

AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing

Distributed Stream Processing Systems (DSPSs) are among the currently most emerging topics in data management, with applications ranging from real-time event monitoring to processing complex dataflow programs and big data analytics. The major market players in this domain are clearly represented by Apache Spark and Flink, which provide a variety of frontend APIs for SQL, statistical inference, machine learning, stream processing, and many others. Yet rather few details are reported on the integration of these engines into the underlying High-Performance Computing (HPC) infrastructure and the communication protocols they use. Spark and Flink, for example, are implemented in Java and still rely on a dedicated master node for managing their control flow among the worker nodes in a compute cluster. In this paper, we describe the architecture of our AIR engine, which is designed from scratch in C++ using the Message Passing Interface (MPI), pthreads for multithreading, and is directly deployed on top of a common HPC workload manager such as SLURM. AIR implements a light-weight, dynamic sharding protocol (referred to as "Asynchronous Iterative Routing"), which facilitates a direct and asynchronous communication among all client nodes and thereby completely avoids the overhead induced by the control flow with a master node that may otherwise form a performance bottleneck. Our experiments over a variety of benchmark settings confirm that AIR outperforms Spark and Flink in terms of latency and throughput by a factor of up to 15; moreover, we demonstrate that AIR scales out much better than existing DSPSs to clusters consisting of up to 8 nodes and 224 cores.Comment: 16 pages, 6 figures, 15 plot