Evaluation of ACE properties of traditional SQL and NoSQL big data systems

Traditional SQL and NoSQL big data systems are the backbone for managing data in cloud, fog and edge computing. This paper develops a new system and adopts the TPC-DS industry standard benchmark in order to evaluate three key properties, availability, consistency and efficiency (ACE) of SQL and NoSQL systems. The contributions of this work are manifold. It evaluates and analyses the tradeoff between the ACE properties. It provides insight into the NoSQL systems and how they can be improved to be sustainable for a more wide range of applications. The evaluation shows that SQL provides stronger consistency, but at the expense of low efficiency and availability. NoSQL provides better efficiency and availability but lacks support for stronger consistency. In order for NoSQL systems to be more sustainable they need to implement transactional schemes that enforce stronger consistency as well as better efficiency and availability

Empowering a Relational Database with LSD: Lazy State Determination

Computer systems are a part of today’s most common activities and, more often than not, involve some type of interaction with a database. In this scheme, databases play a big role, where even small operational delays could cost millions to big tech companies. It is then, of utmost importance that such systems are responsive and adapt automatically to different types of workload. To this date, Relational Database Management System remain the most popular database type, which allows the executing of concurrent transactions with Atomicity, Consistency, Isolation and Durability guarantees. Enforcing such properties requires strict control over the execution of transactions. However, maintaining such properties and controlling the transactions’ concurrency may hamper performance of the system, being this specially the case when database contention is high. Motivated by such behavior, we propose the lazy evaluation of database SQL queries — using Futures/Promises and Java Database Connectivity (JDBC) — by empowering a relational database with Lazy State Determination (LSD). This novel Application Programming Interface (API) allows delaying operations to the commit time, which in the end reduces the transaction window where conflicts may occur. We observed that, by introducing our implementation of a JDBC-LSD driver, in high contention scenarios the throughput increased by 50% and latency reduced by 40%.Os sistemas informáticos são parte das atividades mais comuns na atualidade e, na maioria das vezes, envolvem algum tipo de interação com uma base de dados. Neste cenário, as bases de dados têm um grande papel, sendo que pequenos atrasos operacionais podem custar milhões às grandes empresas tecnológicas. Até os dias de hoje, os Sistemas de Gestão de Bases de Dados Relacionais continuam a ser o tipo de bases de dados mais popular, permitindo a execução concorrente de transações garantindo as propriedades de Atomicidade, Consistência, Isolamento e Durabilidade. A aplicação de tais propriedades requer um controlo rigoroso sobre a execução de transações. No entanto, manter tais propriedades e controlar a concorrência das transacções pode diminuir o desempenho do sistema, sendo especialmente o caso em bases onde a contenção é elevada. Assim, propomos o atraso na execução de queries SQL na base de dados através da introdução do protocolo de controlo de concorrência Lazy State Determination (LSD), com a utilização de Futuros/Promessas e Java Database Connectivity (JDBC). Esta nova Interface de Programação de Aplicações (API) permite adiar as operações para o momento do commit, o que acaba por reduzir a janela da transação onde conflitos podem ocorrer. Observamos que, ao utilizar LSD em um cliente JDBC, nós conseguimos aumentar a taxa de execução de transações em 50% e reduzir a latência em 40% num ambiente de contenção elevada

Data migration between different data models of NOSQL databases

Orientador : Marcos Didonet Del FabroDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa: Curitiba, 17/02/2017Inclui referências : f. 76-79Resumo: Desde sua origem, as bases de dados Nosql têm alcançado um uso generalizado. Devido à falta de padrões de desenvolvimento nesta nova tecnologia emergem grandes desafios. Existem modelos de dados , linguagens de acesso e frameworks heterogêneos, o que torna a migração de dados ainda mais complexa. A maior parte das soluções disponíveis hoje se concentra em fornecer uma representação abstrata e genérica para todos os modelos de dados. Essas soluções se concentram em adaptadores para acessar homogeneamente os dados, mas não para implementar especificamente transformações entre eles. Essas abordagens muitas vezes precisam de um framework para acessar os dados, o que pode impedir de usá-los em alguns cenários. Entre estes desafios, a migração de dados entre as várias soluções revelou-se particularmente difícil. Esta dissertação propõe a criação de um metamodelo e uma série de regras capazes de auxiliar na tarefa de migração de dados. Os dados podem ser convertidos para vários formatos desejados através de um estado intermediário. Para validar a solução foram realizados vários testes com diversos sistemas e utilizando dados reais disponíveis. Palavras Chave: NoSql Databases. Metamodelo. Migração de Dados.Abstract: Since its origin the NoSql Database have achieved widespread use. Due to the lack of standards for development in this new technology great challenges emerges. Among these challenges, the data migration between the various solutions has proved particularly difficult. There are heterogeneous datamodels, access languages and frameworks available, which makes data migration even more complex. Most part of the solutions available today focus on providing an abstract and generic representation for all data models. These solutions focus in design adapters to homogeneously access the data, but not to specifically implement transformations between them. These approaches often need a framework to access the data, which may prevent from using them in some scenarios. This dissertation proposes the creation of a metamodel and a series of rules capable of assisting in the data migration task. The data can be converted to various desired formats through an intermediate state. To validate the solution several tests were performed with different systems and using real data available. Key-words: NoSql Databases. Metamodel. Data Migration

Relatório de Estágio - Solução de BI Roaming Data Science (RoaDS) em ambiente Vodafone

A telecom company (Vodafone), had the need to implement a Business Intelligence solution for Roaming data across a wide set of different data sources. Based on the data visualization of this solution, its key users with decision power, can make a business analysis and needs of infrastructure and software expansion. This document aims to expose the scientific papers produced with the various stages of production of the solution (state of the art, architecture design and implementation results), this Business Intelligence solution was designed and implemented with OLAP methodologies and technologies in a Data Warehouse composed of Data Marts arranged in constellation, the visualization layer was custom made in JavaScript (VueJS). As a base for the results a questionnaire was created to be filled in by the key users of the solution. Based on this questionnaire it was possible to ascertain that user acceptance was satisfactory. The proposed objectives for the implementation of the BI solution with all the requirements was achieved with the infrastructure itself created from scratch in Kubernetes. This BI platform can be expanded using column storage databases created specifically with OLAP workloads in mind, removing the need for an OLAP cube layer. Based on Machine Learning algorithms, the platform will be able to perform the predictions needed to make decisions about Vodafone's Roaming infrastructure

Incorporation of ontologies in data warehouse/business intelligence systems - A systematic literature review

Semantic Web (SW) techniques, such as ontologies, are used in Information Systems (IS) to cope with the growing need for sharing and reusing data and knowledge in various research areas. Despite the increasing emphasis on unstructured data analysis in IS, structured data and its analysis remain critical for organizational performance management. This systematic literature review aims at analyzing the incorporation and impact of ontologies in Data Warehouse/Business Intelligence (DW/BI) systems, contributing to the current literature by providing a classification of works based on the field of each case study, SW techniques used, and the authors’ motivations for using them, with a focus on DW/BI design, development and exploration tasks. A search strategy was developed, including the definition of keywords, inclusion and exclusion criteria, and the selection of search engines. Ontologies are mainly defined using the Ontology Web Language standard to support multiple DW/BI tasks, such as Dimensional Modeling, Requirement Analysis, Extract-Transform-Load, and BI Application Design. Reviewed authors present a variety of motivations for ontology-driven solutions in DW/BI, such as eliminating or solving data heterogeneity/semantics problems, increasing interoperability, facilitating integration, or providing semantic content for requirements and data analysis. Further, implications for practice and research agenda are indicated.info:eu-repo/semantics/publishedVersio

An Investigation into Possible Attacks on HTML5 IndexedDB and their Prevention

This thesis presents an analysis of, and enhanced security model for IndexedDB, the persistent HTML5 browser-based data store. In versions of HTML prior to HTML5, web sites used cookies to track user preferences locally. Cookies are however limited both in file size and number, and must also be added to every HTTP request, which increases web traffic unnecessarily. Web functionality has however increased significantly since cookies were introduced by Netscape in 1994. Consequently, web developers require additional capabilities to keep up with the evolution of the World Wide Web and growth in eCommerce. The response to this requirement was the IndexedDB API, which became an official W3C recommendation in January 2015. The IndexedDB API includes an Object Store, indices, and cursors and so gives HTML5 - compliant browsers a transactional database capability. Furthermore, once downloaded, IndexedDB data stores do not require network connectivity. This permits mobile web- based applications to work without a data connection. Such IndexedDB data stores will be used to store customer data, they will inevitably become targets for attackers. This thesis firstly argues that the design of IndexedDB makes it unavoidably insecure. That is, every implementation is vulnerable to attacks such as Cross Site Scripting, and even data that has been deleted from databases may be stolen using appropriate software tools. This is demonstrated experimentally on both mobile and desktop browsers. IndexedDB is however capable of high performance even when compared to servers running optimized local databases. This is demonstrated through the development of a formal performance model. The performance predictions for IndexedDB were tested experimentally, and the results showed high conformance over a range of usage scenarios. This implies that IndexedDB is potentially a useful HTML5 API if the security issues can be addressed. In the final component of this thesis, we propose and implement enhancements that correct the security weaknesses identified in IndexedDB. The enhancements use multifactor authentication, and so are resistant to Cross Site Scripting attacks. This enhancement is then demonstrated experimentally, showing that HTML5 IndexedDB may be used securely both online and offline. This implies that secure, standards compliant browser based applications with persistent local data stores may both feasible and efficient

Desenvolvimento Software Têxtil

The Fourth Industrial Revolution is a new phase of the textile industry. This uprising focuses on the connectivity and automation of manual processes, forcing textile companies to look for goods that make them go beyond the archaic level of outdated methods. After contacting an engineer in the textile area, with several years of experience, the need for companies in the sector to use a technology capable of handling and storing the data inherent to their methodologies was evident. Thus, the objective of idealizing and developing a solution oriented to the needs of the company in which the textile professional worked was aligned, namely Empresa Têxtil da Maganha. This document is divided into 7 chapters, each of which has its responsibilities, in more detail, the introduction, the state of the art, the value analysis, the requirements and design analysis, the implementation, the experimentation and evaluation and the conclusion. The initial part presents the context, approach, problems and objectives, in a perspective that invites the reader to understand the theme and seek a solution. Next, the planning and technical specification are presented, showing the state of the art decisions made. The technical description in the report highlights the requirements analysis and architectural design, followed by the details of the system implementation. With the development of the textile software disappearing, a quality solution was achieved, following the standards and models described.A Quarta Revolução Industrial trata-se de uma nova fase da indústria têxtil. Esta insurreição dá foco à conectividade e automação dos processos manuais, obrigando as empresas têxteis a procurar por bens que as façam ultrapassar o patamar arcaico de métodos desatualizados. Após o contacto com um engenheiro da área têxtil, com vários anos de experiência, foi evidente a necessidade de as empresas do ramo utilizarem uma tecnologia capaz de manipular e guardar os dados inerentes às suas metodologias. Assim, alinhou-se o objetivo de idealizar e desenvolver uma solução orientada às necessidades da empresa na qual o profissional da área têxtil trabalhou, a Empresa Têxtil da Maganha. O presente documento divide-se em 7 capítulos, cada um dos quais com as suas responsabilidades, mais detalhadamente, a introdução, o estado de arte, a análise de valor, a análise de requisitos e conceção, a implementação, a experimentação e avaliação e a conclusão. A parte inicial apresenta o contexto, abordagem, problemas e objetivos, numa perspetiva que convida o leitor a compreender o tema e a visar uma solução. De seguida são expostos o planeamento e a especificação técnica, evidenciando-se no estado da arte as decisões tomadas. A descrição técnica no relatório destaca a análise dos requisitos e desenho arquitetural, seguidos pelo detalhamento da implementação do sistema. Sumindo a elaboração do software têxtil, foi alcançada uma solução com qualidade, seguindo os padrões e modelos descritos