Storage Solutions for Big Data Systems: A Qualitative Study and Comparison

Big data systems development is full of challenges in view of the variety of application areas and domains that this technology promises to serve. Typically, fundamental design decisions involved in big data systems design include choosing appropriate storage and computing infrastructures. In this age of heterogeneous systems that integrate different technologies for optimized solution to a specific real world problem, big data system are not an exception to any such rule. As far as the storage aspect of any big data system is concerned, the primary facet in this regard is a storage infrastructure and NoSQL seems to be the right technology that fulfills its requirements. However, every big data application has variable data characteristics and thus, the corresponding data fits into a different data model. This paper presents feature and use case analysis and comparison of the four main data models namely document oriented, key value, graph and wide column. Moreover, a feature analysis of 80 NoSQL solutions has been provided, elaborating on the criteria and points that a developer must consider while making a possible choice. Typically, big data storage needs to communicate with the execution engine and other processing and visualization technologies to create a comprehensive solution. This brings forth second facet of big data storage, big data file formats, into picture. The second half of the research paper compares the advantages, shortcomings and possible use cases of available big data file formats for Hadoop, which is the foundation for most big data computing technologies. Decentralized storage and blockchain are seen as the next generation of big data storage and its challenges and future prospects have also been discussed

Semantic data mining and linked data for a recommender system in the AEC industry

Even though it can provide design teams with valuable performance insights and enhance decision-making, monitored building data is rarely reused in an effective feedback loop from operation to design. Data mining allows users to obtain such insights from the large datasets generated throughout the building life cycle. Furthermore, semantic web technologies allow to formally represent the built environment and retrieve knowledge in response to domain-specific requirements. Both approaches have independently established themselves as powerful aids in decision-making. Combining them can enrich data mining processes with domain knowledge and facilitate knowledge discovery, representation and reuse. In this article, we look into the available data mining techniques and investigate to what extent they can be fused with semantic web technologies to provide recommendations to the end user in performance-oriented design. We demonstrate an initial implementation of a linked data-based system for generation of recommendations

Decentralized Control and Adaptation in Distributed Applications via Web and Semantic Web Technologies

The presented work provides an approach and an implementation for enabling decentralized control in distributed applications composed of heterogeneous components by benefiting from the interoperability provided by the Web stack and relying on semantic technologies for enabling data integration. In particular, the concept of Smart Components enables adaptability at runtime through an adaptation layer and is complemented by a reference architecture as well as a prototypical implementation

An event-based resource management framework for distributed decision-making in decentralized virtual power plants

The Smart Grid incorporates advanced information and communication technologies (ICTs) in power systems, and is characterized by high penetration of distributed energy resources (DERs). Whether it is the nation-wide power grid or a single residential building, the energy management involves different types of resources that often depend on and influence each other. The concept of virtual power plant (VPP) has been proposed to represent the aggregation of energy resources in the electricity market, and distributed decision-making (DDM) plays a vital role in VPP due to its complex nature. This paper proposes a framework for managing different resource types of relevance to energy management for decentralized VPP. The framework views VPP as a hierarchical structure and abstracts energy consumption/generation as contractual resources, i.e., contractual offerings to curtail load/supply energy, from third party VPP participants for DDM. The proposed resource models, event-based approach to decision making, multi-agent system and ontology implementation of the framework are presented in detail. The effectiveness of the proposed framework is then demonstrated through an application to a simulated campus VPP with real building energy data

Linked data as medium for distributed Multi-Agent Systems

The conceptual design and discussion of multi-agents systems (MAS) typically focuses on agents and their models, and the elements and effects in the environment which they perceive. This view, however, leaves out potential pitfalls in the later implementation of the system that may stem from limitations in data models, interfaces, or protocols by which agents and environments exchange information. By today, the research community agrees that for this, that the environment should be understood as well as abstraction layer by which agents access, interpret, and modify elements within the environment. This, however, blurs the the line of the environment being the sum of interactive elements and phenomena perceivable by agents, and the underlying technology by which this information and interactions are offered to agents. This thesis proposes as remedy to consider as third component of multi agent systems, besides agents and environments, the digital medium by which the environment is provided to agents. "Medium" then refers to exactly this technological component via which environment data is published interactively towards the agents, and via which agents perceive, interpret, and finally, modify the underlying environment data. Furthermore, this thesis will detail how MAS may use capabilities of a properly chosen medium to achieve coordinating system behaviors. A suitable candidate technology for digital agent media comes from the Semantic Web in form of Linked Data. In addition to conceptual discussions about the notions of digital agent media, this thesis will provide in detail a specification of a Linked Data agent medium, and detail on means to implement MAS around Linked Data media technologies.Sowohl der konzeptuelle Entwurf von, als auch die wissenschaftliche Diskussion über Multi-Agenten-Systeme (MAS) konzentrieren sich für gewöhnlich auf die Agenten selbst, die Agentenmodelle, sowie die Elemente und Effekte, die sie in ihrer Umgebung wahrnehmen. Diese Betrachtung lässt jedoch mögliche Probleme in einer späteren Implementierung aus, die von Einschränkungen in Datenmodellen, Schnittstellen, oder Protokollen herrühren können, über die Agenten und ihre Umgebung Informationen miteinander austauschen. Heutzutage ist sich die Forschungsgemeinschaft einig, dass die Umgebung als solche als Abstraktionsschicht verstanden werden sollte, über die Agenten Umgebungseffekte und -elemente wahrnehmen, interpretieren, und mit ihnen interagieren. Diese Betrachtungsweise verschleiert jedoch die Trennung zwischen der Umgebung als die Sammlung interaktiver Elemente und wahrnehmbarer Phänomene auf der einen Seite, und der zugrundeliegenden Technologie, über die diese Information den Agenten bereitgestellt wird, auf der anderen. Diese Dissertation schlägt als Lösung vor, zusätzlich zu Agenten undUmgebung ein digitales Medium, über das Agenten die Umgebung bereitgestellt wird, als drittes Element von Multi-Agenten-Systemen zu betrachten. Der Begriff "Medium" bezieht sich dann genau auf diese technologische Komponente, über die Umgebungsinformationen Agenten interaktiv bereitgestellt werden, und über die Agenten die zugrundeliegenden Daten wahrnehmen, interpretieren, und letztendlich modifizieren. Desweiteren wird diese Dissertation aufzeigen, wie die Eigenschaften eines sorgfältig gewählten Mediums ausgenutzt werden können, um ein koordiniertes Systemverhalten zu erreichen. Ein geeigneter Kandidat für ein digitales Agentenmedium findet sich im Ökosystem des „Semantic Web”, in Form von „Linked Data”, wörtlich („verknüpfte Daten”). Zusätzlich zu einer konzeptionellen Diskussion über die Natur digitaler Agenten- Media, spezifiziert diese Dissertation „Linked Data” als Agentenmedium detailliert aus, und beschreibt im Detail die Mittel, wie sich MAS um Linked Data Technologien herum implementieren lassen

SARA – A Semantic Access Point Resource Allocation Service for Heterogenous Wireless Networks

In this paper, we present SARA, a Semantic Access point Resource Allocation service for heterogenous wireless networks with various wireless access technologies existing together. By automatically reasoning on the knowledge base of the full system provided by a knowledge based autonomic network management system - SEANET, SARA selects the access point providing the best quality of service among the different access technologies. Based on an ontology assisted knowledge based system SEANET, SARA can also adapt the access point selection strategy according to customer defined rules automatically. Results of our evaluation based on emulated networks with hybrid access technologies and various scales show that SARA is able to improve the channel condition, in terms of throughput, evidently. Comparisons with current AP selection algorithms demonstrate that SARA outperforms the existing AP selection algorithms. The overhead in terms of time expense is reasonable and is shown to be faster than traditional access point selection approaches

A Social Network-Based Peer-To-Peer Model For Resource Discovery

Peer-to-Peer (P2P) systems are distributed systems consisting of interconnected nodes which provide scalability, fault tolerance, decentralized coordination, self-organization, anonymity, distributed resources and services sharing, lower cost of ownership and better support for creating ad hoc networks. Data sharing, a subset of resource sharing, is one of the attractive topic in P2P systems. Because of autonomy of the nodes, decentralized coordination and volatility of network caused by the autonomy, data sharing is not an easy task in P2P system. Furthermore, there is no guarantee that a node stays in the network for a specific period of time. Hence, the answers to a particular query may be retrieved from different nodes every time. Moreover, the lack of centralized coordinators makes this process harder. These problems in P2P systems lead to a well known problem which is called resource discovery

SEMANTIC APPROACH TO SMART CONTRACT VERIFICATION

Vulnerabilities of smart contract are certainly one of the limiting factors for wider adoption of blockchain technology. Smart contracts written in Solidity language are considered due to common adoption of the Ethereum blockchain platform. Despite its popularity, the semantics of the language is not completely documented and relies on implicit mechanisms not publicly available and as such vulnerable to possible attacks. In addition, creating formal semantics for the higher-level language provides support to verification mechanisms. In this paper, a novel approach to smart contact verification is presented that uses ontologies in order to leverage semantic annotations of the smart contract source code combined with semantic representation of domain-specific aspects. The following aspects of smart contracts, apart from source code are taken into consideration for verification: business logic, domain knowledge, run-time state changes and expert knowledge about vulnerabilities. Main advantages of the proposed verification approach are platform independence and extendability

Sextant: Visualizing time-evolving linked geospatial data

The linked open data cloud is constantly evolving as datasets get continuously updated with newer versions. As a result, representing, querying, and visualizing the temporal dimension of linked data is crucial. This is especially important for geospatial datasets that form the backbone of large scale open data publication efforts in many sectors of the economy (e.g., the public sector, the Earth Observation sector). Although there has been some work on the representation and querying of linked geospatial data that change over time, to the best of our knowledge, there is currently no tool that offers spatio-temporal visualization of such data. This is in contrast with the existence of many tools for the visualization of the temporal evolution of geospatial data in the GIS area. In this article, we present Sextant, a Web-based system for the visualization and exploration of time-evolving linked geospatial data and the creation, sharing, and collaborative editing of “temporally-enriched” thematic maps which are produced by combining different sources of such data. We present the architecture of Sextant, give examples of its use and present applications in which we have deployed it

Migrating microservices to graph database

Microservice architecture is a popular approach to structuring web backend services. Another emerging trend, after a period of hibernation, is utilizing modern graph database management systems for managing complex, richly connected data. The two approaches have rarely been used in tandem, as microservices emphasize modularization and decoupling of services, while graph data models favor data integration. In this study, literature on microservices and graph databases is reviewed and a synthesis between the two paradigms is presented. Based on the theoretical discussion, a software architecture combining the two elements is formulated and implemented using microservices serving content metadata at Yleisradio, the Finnish national broadcasting company. The architecture design follows the Design Science Research Process model. Finally, the renewed system is evaluated using quantitative and qualitative metrics. The performance of the system is measured using automated API queries and load tests. The new system was compared to an earlier version based on a PostgreSQL database. The tests gave slight indication that the renewed system performed better for complex queries, where a large number of relations were traversed, but worse in terms of throughput under heavy load. Based on the these findings, a number of performance-enhancing optimizations to the system are introduced. Observations and perpectives are also gathered in a project retrospective session. It is concluded that the resulting architecture holds promise for managing complex data rich in relations in a safe manner. In it, the different domains of the knowledge graph are decoupled into distinct named graphs managed by different microservices