Adaptive Cost Estimation for Client-Server based Heterogeneous Database Systems

In this paper, we propose a new method for estimating query cost in client-server based heterogeneous database management system. The cost estimation parameters are adjusted by an Adaptive Cost Estimation (ACE) module which uses query execution feedback yielding more and more accurate cost estimates. The most important features of ACE are its detailed cost model which accounts for all costs incurred, its rapid convergence to the actual parameter values, and its low overhead which permits continuous adaptation during the run time of the system. ACE has been implemented and tested with Oracle 6, Oracle 7, Ingres, and ADMS. Extensive experiments performed on these systems show that the ACE's time estimates are within 20% of the real wall-clock time for more than 92% of the queries. This percentage surpasses 98% for queries over 20 seconds. (Also cross-referenced as UMIACS-TR-96-37

Evolutionary techniques for updating query cost models in a dynamic multidatabase environment

Deriving local cost models for query optimization in a dynamic multidatabase system (MDBS) is a challenging issue. In this paper, we study how to evolve a query cost model to capture a slowly-changing dynamic MDBS environment so that the cost model is kept up-to-date all the time. Two novel evolutionary techniques, i.e., the shifting method and the block-moving method, are proposed. The former updates a cost model by taking up-to-date information from a new sample query into consideration at each step, while the latter considers a block (batch) of new sample queries at each step. The relevant issues, including derivation of recurrence updating formulas, development of efficient algorithms, analysis and comparison of complexities, and design of an integrated scheme to apply the two methods adaptively, are studied. Our theoretical and experimental results demonstrate that the proposed techniques are quite promising in maintaining accurate cost models efficiently for a slowly changing dynamic MDBS environment. Besides the application to MDBSs, the proposed techniques can also be applied to the automatic maintenance of cost models in self-managing database systems.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/47868/1/778_2003_Article_110.pd

A Study of Query Execution Strategies for Client-Server Database Systems

Query processing in a client-server database system raises the question of where to execute queries to minimize the communication costs and response time of a query, and to load-balance the system. This paper evaluates the two common query execution strategies, data shipping and query shipping, and a policy referred to as hybrid shipping. Data shipping determines that queries be executed at clients; query shipping determines that queries be executed at servers; and hybrid shipping provides the flexibility to execute queries at clients and servers. The experiments with a client-server model confirm that the query execution policy is critical for the performance of a system. Neither data nor query shipping are optimal in all situations, and the performance penalities can be substantial. Hybrid shipping at least matches the best performance of data and query shipping and shows better performance than both in many cases. The performance of hybrid shipping plans, however, is shown to be sensitive to changes in the state of the system (e.g., the load of machines and the contents of caches). Initial experiments indicate that an extended version of a 2-step optimization may be an effective strategy for adjusting plans according to the state of the system at runtime. (Also cross-referenced as UMIACS-TR-95-85

Steganographic database management system

Master'sMASTER OF SCIENC

Microservice-Based Integration Framework for a Back-Office Solution

Dissertação de Mestrado em Engenharia InformáticaNot long ago, monolithic applications ruled among production servers – these applications had massive scopes which made them difficult to maintain, with constraints of libraries shared between modules and where every change or update is attached with big downtimes. To stray from this approach, enterprises chose to divide their big applications into smaller ones with fewer responsibilities, a clearer notion of boundaries and for the better part of it, more maintainable and scalable. The microservice approach allows enterprises to better divide themselves among teams that follow the full stack and spectrum of development in each application, from the persistence layer through the API and to the client, and from planning, through development to later support. The project exposed in this paper enlightens the scenario of an e-commerce platform’s back-office - where the implementation of a strangler pattern divided a large monolithic application into smaller microservices – leaving the door open for the integration of the multiple client applications to interconnect. The proposed solution intends to integrate the various systems of Jumia and take on this exposed opportunity, resorting to a microservice architecture and integration patterns with the objective of easing the flow of operations for processes that involve several management tools.Recentemente, o desenvolvimento de aplicações mudou à escala mundial, os sistemas distribuídos permitiram a introdução de um novo paradigma. Este paradigma baseia-se na redução de uma grande aplicação (monólito) em pequenos sub-módulos (micro-serviços) que comunicam perfeitamente entre si como se de uma única aplicação se tratasse. Este paradigma veio também refrescar as estruturas internas das empresas, ao distribuir os diversos serviços entre equipas, de forma a que cada uma delas esteja presente em todo o ciclo de vida das aplicações, desde o conceito até ao lançamento, passando pelo desenvolvimento e posterior manutenção e suporte da mesma. As mesmas equipas são também responsáveis por toda a stack que cada micro-serviço contém partindo da user interface (UI), passando por toda a API que contém a lógica de negócio até à camada de acesso de dados. Esta nova abordagem oferece algumas vantagens quando comparada com outras soluções disponíveis no mercado, tais como a liberdade de cada um dos serviços em ser desenvolvido nas tecnologias e linguagens que melhor se adequam ao seu propósito, sem que estejam presas a uma decisão tomada numa ocasião anterior para um propósito diferente ou a restrições de dependências incompatíveis entre si. Sendo que um dos principais problemas da computação distribuída é a possível indisponibilidade de cada um dos seus intervenientes, a arquitetura orientada a micro-serviços (microservice architecture, MSA) prevê que cada um dos seus serviços esteja contido no seu contexto (bounded context) e que disponha de todos os dados que lhe correspondem, desta forma a indisponibilidade de qualquer serviço não deve impactar o desempenho de nenhum dos seus pares. A reduzida dimensão de cada um destes serviços permite a existência de processos de deploy mais rápidos o que acaba por se refletir em downtimes mais reduzidos. Outra das vantagens da redução das dimensões e dos contextos de cada um dos serviços é a sua fácil manutenção, uma vez que o código se torna mais conciso e específico ao propósito que prevê cumprir. A modularidade dos micro-serviços permite-lhes também ajustar o número de réplicas de cada um deles de forma independente de acordo com as necessidades e previsões de volume de tráfego a cada momento. Apesar de todas as vantagens acima expostas, uma MSA traz consigo também alguns desafios tais como os testes de integração, debugging, deploying, retrocompatibilidade com outros serviços, entre outras abordadas em maior detalhe neste documento. O projeto exposto neste documento é um projeto proposto pela Jumia, uma empresa que disponibiliza uma plataforma de comércio online no continente africano. Esta plataforma está disponível em onze países africanos com mais de cem armazéns espalhados por todo o continente e que conta com mais de cinco mil colaboradores espalhados pelo mundo. Tal como muitas outras empresas no mercado a Jumia idealizou os seus processos de operações numa aplicação única que controlava todos os fluxos de negócio e continha em si toda a informação de armazenamento, produtos, entregas, pagamentos, encomendas entre outras. Rapidamente a aplicação de back-office da Jumia tornou-se insustentável e, tal como tinha sido executado noutras empresas do mesmo ramo, foi implementado um strangler pattern. Desta forma tornou-se possível fazer uma separação de dependências gradualmente, isolando cada um dos processos de negócio num serviço independente que persiste todos os dados necessários para a execução de cada uma das operações. No entanto, a implementação deste padrão deu origem a uma lacuna nos processos da empresa, uma vez que cada um dos serviços possui o seu user interface, algumas das operações requerem que os agentes de operações transitem entre aplicações, e necessitem de se autenticar novamente. Este processo acaba por ter impacto no fluxo de operações, refletindo-se no número de encomendas processadas e por consequência nas receitas da empresa. O presente documento pretende explorar a oportunidade de negócio proposta, assim como os mais essenciais padrões de integração de micro-serviços, de forma a apresentar uma solução que consiga colmatar a lacuna apresentada sem pôr em causa a segurança das aplicações e as normas de conformidade exigidas. Esta proposta foi elaborada através da conceção de uma arquitetura orientada a micro-serviços de forma coreografada tendo como objetivo ser integrada nas diversas aplicações de Back-Office com recurso a uma biblioteca importada através do gestor do Node Package Manager

Digital Image Access & Retrieval

The 33th Annual Clinic on Library Applications of Data Processing, held at the University of Illinois at Urbana-Champaign in March of 1996, addressed the theme of "Digital Image Access & Retrieval." The papers from this conference cover a wide range of topics concerning digital imaging technology for visual resource collections. Papers covered three general areas: (1) systems, planning, and implementation; (2) automatic and semi-automatic indexing; and (3) preservation with the bulk of the conference focusing on indexing and retrieval.published or submitted for publicatio

Performance assessment of real-time data management on wireless sensor networks

Technological advances in recent years have allowed the maturity of Wireless Sensor Networks (WSNs), which aim at performing environmental monitoring and data collection. This sort of network is composed of hundreds, thousands or probably even millions of tiny smart computers known as wireless sensor nodes, which may be battery powered, equipped with sensors, a radio transceiver, a Central Processing Unit (CPU) and some memory. However due to the small size and the requirements of low-cost nodes, these sensor node resources such as processing power, storage and especially energy are very limited. Once the sensors perform their measurements from the environment, the problem of data storing and querying arises. In fact, the sensors have restricted storage capacity and the on-going interaction between sensors and environment results huge amounts of data. Techniques for data storage and query in WSN can be based on either external storage or local storage. The external storage, called warehousing approach, is a centralized system on which the data gathered by the sensors are periodically sent to a central database server where user queries are processed. The local storage, in the other hand called distributed approach, exploits the capabilities of sensors calculation and the sensors act as local databases. The data is stored in a central database server and in the devices themselves, enabling one to query both. The WSNs are used in a wide variety of applications, which may perform certain operations on collected sensor data. However, for certain applications, such as real-time applications, the sensor data must closely reflect the current state of the targeted environment. However, the environment changes constantly and the data is collected in discreet moments of time. As such, the collected data has a temporal validity, and as time advances, it becomes less accurate, until it does not reflect the state of the environment any longer. Thus, these applications must query and analyze the data in a bounded time in order to make decisions and to react efficiently, such as industrial automation, aviation, sensors network, and so on. In this context, the design of efficient real-time data management solutions is necessary to deal with both time constraints and energy consumption. This thesis studies the real-time data management techniques for WSNs. It particularly it focuses on the study of the challenges in handling real-time data storage and query for WSNs and on the efficient real-time data management solutions for WSNs. First, the main specifications of real-time data management are identified and the available real-time data management solutions for WSNs in the literature are presented. Secondly, in order to provide an energy-efficient real-time data management solution, the techniques used to manage data and queries in WSNs based on the distributed paradigm are deeply studied. In fact, many research works argue that the distributed approach is the most energy-efficient way of managing data and queries in WSNs, instead of performing the warehousing. In addition, this approach can provide quasi real-time query processing because the most current data will be retrieved from the network. Thirdly, based on these two studies and considering the complexity of developing, testing, and debugging this kind of complex system, a model for a simulation framework of the real-time databases management on WSN that uses a distributed approach and its implementation are proposed. This will help to explore various solutions of real-time database techniques on WSNs before deployment for economizing money and time. Moreover, one may improve the proposed model by adding the simulation of protocols or place part of this simulator on another available simulator. For validating the model, a case study considering real-time constraints as well as energy constraints is discussed. Fourth, a new architecture that combines statistical modeling techniques with the distributed approach and a query processing algorithm to optimize the real-time user query processing are proposed. This combination allows performing a query processing algorithm based on admission control that uses the error tolerance and the probabilistic confidence interval as admission parameters. The experiments based on real world data sets as well as synthetic data sets demonstrate that the proposed solution optimizes the real-time query processing to save more energy while meeting low latency.Fundação para a Ciência e Tecnologi