Self-configured Elastic Database with Deep Q-Learning Approach

Elastic databases have grown in popularity over conventional databases in recent years due to their ability to be allocated with sufficient capacity for peak load. Especially with the support of the cloud platform, which provides flexible resources and low cost, elastic databases on the cloud show their excellent potential in scalability, flexibility, and accessibility. However, the interaction between the cloud layers of virtual machines (VMs) and databases further complicates the issue of cloud configuration to adapt to dynamic workloads. In this paper, I explore a framework for a self-configured elastic database that can optimize the cloud configuration and adaptively allocate resources under the constraints of databases\u27 Service Level Agreement (SLA). At the core of the framework is a Deep Q learning approach, which combines the advantages of Reinforcement Learning (RL) and Deep Learning (DL). The framework is built on Amazon Web Service (AWS)\u27s cloud environment and uses MySQL database for its high availability replication mechanism. Experimental results on the TPC-W benchmark demonstrate that with the implementation of Deep Q learning, the elastic database reduces SLA violation by more than 90\%, in the response to the steep slope of workload change

The End of Slow Networks: It's Time for a Redesign

Next generation high-performance RDMA-capable networks will require a fundamental rethinking of the design and architecture of modern distributed DBMSs. These systems are commonly designed and optimized under the assumption that the network is the bottleneck: the network is slow and "thin", and thus needs to be avoided as much as possible. Yet this assumption no longer holds true. With InfiniBand FDR 4x, the bandwidth available to transfer data across network is in the same ballpark as the bandwidth of one memory channel, and it increases even further with the most recent EDR standard. Moreover, with the increasing advances of RDMA, the latency improves similarly fast. In this paper, we first argue that the "old" distributed database design is not capable of taking full advantage of the network. Second, we propose architectural redesigns for OLTP, OLAP and advanced analytical frameworks to take better advantage of the improved bandwidth, latency and RDMA capabilities. Finally, for each of the workload categories, we show that remarkable performance improvements can be achieved

Fast Distributed Transactions for Partitioned Database Systems.

ABSTRACT Many distributed storage systems achieve high data access throughput via partitioning and replication, each system with its own advantages and tradeoffs. In order to achieve high scalability, however, today's systems generally reduce transactional support, disallowing single transactions from spanning multiple partitions. Calvin is a practical transaction scheduling and data replication layer that uses a deterministic ordering guarantee to significantly reduce the normally prohibitive contention costs associated with distributed transactions. Unlike previous deterministic database system prototypes, Calvin supports disk-based storage, scales near-linearly on a cluster of commodity machines, and has no single point of failure. By replicating transaction inputs rather than effects, Calvin is also able to support multiple consistency levels-including Paxosbased strong consistency across geographically distant replicas-at no cost to transactional throughput

Evaluation of ACE properties of traditional SQL and NoSQL big data systems

Traditional SQL and NoSQL big data systems are the backbone for managing data in cloud, fog and edge computing. This paper develops a new system and adopts the TPC-DS industry standard benchmark in order to evaluate three key properties, availability, consistency and efficiency (ACE) of SQL and NoSQL systems. The contributions of this work are manifold. It evaluates and analyses the tradeoff between the ACE properties. It provides insight into the NoSQL systems and how they can be improved to be sustainable for a more wide range of applications. The evaluation shows that SQL provides stronger consistency, but at the expense of low efficiency and availability. NoSQL provides better efficiency and availability but lacks support for stronger consistency. In order for NoSQL systems to be more sustainable they need to implement transactional schemes that enforce stronger consistency as well as better efficiency and availability

Testing of transactional services in NoSQL key-value databases

Transactional services guarantee the consistency of shared data during the concurrent execution of multiple applications. They have been used in various domains ranging from classical databases through to service-oriented computing systems to NoSQL databases and cloud. Though transactional services aim to ensure data consistency, NoSQL databases prioritize efficiency/availability over data consistency. In order to address these issues various transaction models and protocols have been proposed in the literature. However, testing of transactions in NoSQL database has not been addressed. In this paper, we investigate into the testing of transactional services in NoSQL databases in order to test and analyse the data consistency by taking into account the characteristics of NoSQL databases such as efficiency, velocity, etc. Accordingly, we develop a framework for testing transactional services in NoSQL databases. The novelty and contributions are that we develop a context-aware transactional model that takes into account contextual requirements of NoSQL clients and the system level setting in relation to the data consistency. This can assist NoSQL application developers in choosing between transactional and non-transactional services based on their requirements of the level of data consistency. The framework also provides ways to analyse the impact of the big data requirements and characteristics (e.g., velocity, efficiency) on the data consistency of NoSQL databases. The evaluation and testing are carried out using a widely used NoSQL key/value database, Riak, and a real (open) and big data from the Council of London for public transportation of the London bus services

Consistency in scalable systems

[EN] While eventual consistency is the general consistency guarantee ensured in cloud environments, stronger guarantees are in fact achievable. We show how scalable and highly available systems can provide processor, causal, sequential and session consistency during normal functioning. Failures and network partitions negatively affect consistency and generate divergence. After the failure or the partition, reconciliation techniques allow the system to restore consistency.This work has been supported by EU FEDER and Spanish MICINN under research grants TIN2009-14460-C03-01 and TIN2010-17193.Ruiz Fuertes, MI.; Pallardó Lozoya, MR.; Muñoz-Escoí, FD. (2012). Consistency in scalable systems. En On the Move to Meaningful Internet Systems: OTM 2012. Springer Verlag (Germany). 7566:549-565. https://doi.org/10.1007/978-3-642-33615-7_7S5495657566Ahamad, M., Bazzi, R.A., John, R., Kohli, P., Neiger, G.: The power of processor consistency. In: Proceedings of the Fifth Annual ACM Symposium on Parallel Algorithms and Architectures, SPAA 1993, pp. 251–260. ACM, New York (1993), http://doi.acm.org/10.1145/165231.165264Alvarez, A., Arévalo, S., Cholvi, V., Fernández, A., Jiménez, E.: On the Interconnection of Message Passing Systems. Inf. Process. Lett. 105(6), 249–254 (2008)Amazon Web Services LLC: Amazon Simple Storage Service (S3). Website (March 2011), http://aws.amazon.com/s3/Baker, J., Bond, C., Corbett, J.C., Furman, J.J., Khorlin, A., Larson, J., Léon, J., Li, Y., Lloyd, A., Yushprakh, V.: Megastore: Providing Scalable, Highly Available Storage for interactive services. In: 5th Biennial Conf. on Innovative Data Systems Research (CIDR), Asilomar, CA, USA, pp. 223–234 (January 2011)Baldoni, R., Beraldi, R., Friedman, R., van Renesse, R.: The Hierarchical Daisy Architecture for Causal Delivery. Distributed Systems Engineering 6(2), 71–81 (1999)Bernstein, P.A., Hadzilacos, V., Goodman, N.: Concurrency Control and Recovery in Database Systems. Addison-Wesley (1987)Bernstein, P.A., Reid, C.W., Das, S.: Hyder - A Transactional Record Manager for Shared Flash. In: 5th Biennial Conf. on Innovative Data Systems Research (CIDR), Asilomar, CA, USA, pp. 9–20 (January 2011)Bershad, B.N., Zekauskas, M.J., Sawdon, W.A.: The Midway Distributed Shared Memory System. In: Proc. IEEE CompCon Conf. (1993)Brewer, E.A.: Towards Robust Distributed Systems (Abstract). In: Proc. ACM Symp. Princ. Distrib. Comput., p. 7 (2000)Budhiraja, N., Marzullo, K., Schneider, F.B., Toueg, S.: The Primary-Backup Approach. In: Mullender, S.J. (ed.) Distributed Systems, 2nd edn., ch. 8, pp. 199–216. Addison-Wesley, ACM Press (1993)Campbell, D.G., Kakivaya, G., Ellis, N.: Extreme Scale with Full SQL Language Support in Microsoft SQL Azure. In: Intnl. Conf. on Mngmnt. of Data (SIGMOD), pp. 1021–1024. ACM, New York (2010), http://doi.acm.org/10.1145/1807167.1807280Cholvi, V., Jiménez, E., Anta, A.F.: Interconnection of distributed memory models. J. Parallel Distrib. Comput. 69(3), 295–306 (2009)Cooper, B.F., Ramakrishnan, R., Srivastava, U., Silberstein, A., Bohannon, P., Jacobsen, H., Puz, N., Weaver, D., Yerneni, R.: PNUTS: Yahoo!’s hosted data serving platform. PVLDB 1(2), 1277–1288 (2008)Daudjee, K., Salem, K.: Lazy Database Replication with Ordering Guarantees. In: Proc. Int. Conf. Data Eng., pp. 424–435. IEEE-CS (2004)Daudjee, K., Salem, K.: Lazy Database Replication with Snapshot Isolation. In: Proc. Int. Conf. Very Large Data Bases, pp. 715–726. ACM (2006)DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon’s Highly Available Key-value Store. In: ACM Symp. Oper. Syst. Princ., pp. 205–220 (2007)Fernández, A., Jiménez, E., Cholvi, V.: On the interconnection of causal memory systems. J. Parallel Distrib. Comput. 64(4), 498–506 (2004)Gilbert, S., Lynch, N.A.: Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. ACM SIGACT News 33(2), 51–59 (2002)Goodman, J.R.: Cache Consistency and Sequential Consistency. Tech. Rep. 61, SCI Committee (March 1989)Gray, J., Helland, P., O’Neil, P.E., Shasha, D.: The Dangers of Replication and a Solution. In: Proc. ACM SIGMOD Int. Conf. Manage. Data, pp. 173–182. ACM (1996)Helland, P., Campbell, D.: Building on Quicksand. In: Proc. Bienn. Conf. Innov. Data Syst. Research (2009), www.crdrdb.orgHutto, P., Ahamad, M.: Slow Memory: Weakening Consistency to Enhance Concurrency in Distributed Shared Memories. In: Proceedings of the 10th International Conference on Distributed Computing Systems, pp. 302–311 (May 1990)Johnson, S., Jahanian, F., Shah, J.: The Inter-group Router Approach to Scalable Group Composition. In: ICDCS, pp. 4–14 (1999)Kraska, T., Hentschel, M., Alonso, G., Kossmann, D.: Consistency Rationing in the Cloud: Pay only when it matters. PVLDB 2(1), 253–264 (2009)Lamport, L.: How to Make a Multiprocessor Computer that Correctly Executes multiprocess programs. IEEE Trans. Computers 28(9), 690–691 (1979)Lipton, R.J., Sandberg, J.S.: Pram: A Scalable Shared Memory. Tech. Rep. CS-TR-180-88, Princeton University, Department of Computer Science (September 1988)Mosberger, D.: Memory Consistency Models. Operating Systems Review 27(1), 18–26 (1993)Ruiz-Fuertes, M.I., Muñoz-Escoí, F.D.: Refinement of the One-Copy Serializable Correctness Criterion. Tech. Rep. ITI-SIDI-2011/004, Instituto Tecnológico de Informática, Valencia, Spain (November 2011)Stonebraker, M., Madden, S., Abadi, D.J., Harizopoulos, S., Hachem, N., Helland, P.: The End of an Architectural Era (It’s Time for a Complete Rewrite). In: 33rd Intnl. Conf. on Very Large Data Bases (VLDB), pp. 1150–1160. ACM Press, Vienna (2007)Terry, D.B., Demers, A.J., Petersen, K., Spreitzer, M., Theimer, M., Welch, B.B.: Session Guarantees for Weakly Consistent Replicated Data. In: Proc. Int. Conf. Parallel Distrib. Inform. Syst., pp. 140–149. IEEE-CS (1994)Vogels, W.: Eventually Consistent. Communications of the ACM (CACM) 52(1), 40–44 (2009)VoltDB, Inc.: VoltDB technical overview: A high performance, scalable RDBMS for Big Data, high velocity OLTP and realtime analytics. Website (April 2012), http://voltdb.com/sites/default/files/PDFs/VoltDBTechnicalOverview_April_2012.pdfWiesmann, M., Schiper, A.: Comparison of Database Replication Techniques Based on Total Order Broadcast. IEEE T. Knowl. Data En. 17(4), 551–566 (2005

Scalable data management in distributed information systems

[EN] In the era of cloud computing and huge information systems, distributed applications should manage dynamic workloads; i.e., the amount of client requests per time unit may vary frequently and servers should rapidly adapt their computing efforts to those workloads. Cloud systems provide a solid basis for this kind of applications but most of the traditional relational database systems are unprepared to scale up with this kind of distributed systems. This paper surveys different techniques being used in modern SQL, NoSQL and NewSQL systems in order to increase the scalability and adaptability in the management of persistent data. © 2011 Springer-Verlag.This work has been supported by EU FEDER and Spanish MICINN under research grants TIN2009-14460-C03-01 and TIN2010-17193Pallardó Lozoya, MR.; Esparza Peidro, J.; García Escriva, JR.; Decker, H.; Muñoz Escoí, FD. (2011). Scalable data management in distributed information systems. Lecture Notes in Computer Science. 7046:208-217. https://doi.org/10.1007/978-3-642-25126-9_31S2082177046Helland, P.: Life beyond distributed transactions: an apostate’s opinion. In: 3rd Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA, pp. 132–141 (2007)Finkelstein, S., Jacobs, D., Brendle, R.: Principles for inconsistency. In: 4th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA (2009)Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.: Bigtable: A distributed storage system for structured data. In: 7th Symp. on Operat. Syst. Design and Implem. (OSDI), pp. 205–218. USENIX Assoc., Seattle (2006)Cooper, B.F., Baldeschwieler, E., Fonseca, R., Kistler, J.J., Narayan, P.P.S., Neerdaels, C., Negrin, T., Ramakrishnan, R., Silberstein, A., Srivastava, U., Stata, R.: Building a cloud for Yahoo! IEEE Data Eng. Bull. 32, 36–43 (2009)DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon’s highly available key-value store. In: 21st ACM Symp. on Operat. Syst. Princ. (SOSP), Stevenson, Washington, USA, pp. 205–220 (2007)Stonebraker, M., Madden, S., Abadi, D.J., Harizopoulos, S., Hachem, N., Helland, P.: The end of an architectural era (it’s time for a complete rewrite). In: 33rd Intnl. Conf. on Very Large Data Bases (VLDB), pp. 1150–1160. ACM Press, Vienna (2007)Lomet, D.B., Fekete, A., Weikum, G., Zwilling, M.J.: Unbundling transaction services in the cloud. In: 4th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA (2009)Campbell, D.G., Kakivaya, G., Ellis, N.: Extreme scale with full SQL language support in Microsoft SQL Azure. In: Intnl. Conf. on Mngmnt. of Data (SIGMOD), pp. 1021–1024. ACM, New York (2010)Levandoski, J.J., Lomet, D., Mokbel, M.F., Zhao, K.K.: Deuteronomy: Transaction support for cloud data. In: 5th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA, pp. 123–133 (2011)Helland, P., Campbell, D.: Building on quicksand. In: 4th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA (2009)Muñoz-Escoí, F.D., García-Escrivá, J.R., Pallardó-Lozoya, M.R., Esparza-Peidro, J.: Managing scalable persistent data. Technical Report ITI-SIDI-2011/003, Instituto Tecnológico de Informática, Universitat Politècnica de València, Spain (2011)Agrawal, D., El Abbadi, A., Antony, S., Das, S.: Data management challenges in cloud computing infrastructures. In: 6th Intnl. Wshop. on Databases in Networked Information Systems (DNIS), Aizu-Wakamatsu, Japan, pp. 1–10 (2010)Stonebraker, M.: The case for shared nothing. IEEE Database Eng. Bull. 9, 4–9 (1986)Alonso, G., Kossmann, D., Roscoe, T.: SwissBox: An architecture for data processing appliances. In: 5th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA, pp. 32–37 (2011)Baker, J., Bond, C., Corbett, J.C., Furman, J.J., Khorlin, A., Larson, J., Léon, J.M., Li, Y., Lloyd, A., Yushprakh, V.: Megastore: Providing scalable, highly available storage for interactive services. In: 5th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA, pp. 223–234 (2011)Curino, C., Jones, E.P.C., Popa, R.A., Malviya, N., Wu, E., Madden, S., Balakrishnan, H., Zeldovich, N.: Relational cloud: A database-as-a-service for the cloud. In: 5th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA, pp. 235–240 (2011)Das, S., Agrawal, D., El Abbadi, A.: ElasTraS: An elastic transactional data store in the cloud. CoRR abs/1008.3751 (2010)Vogels, W.: Eventually consistent. Commun. ACM 52, 40–44 (2009)Breitbart, Y., Korth, H.F.: Replication and consistency: being lazy helps sometimes. In: 16th ACM Symp. on Princ. of Database Syst., PODS 1997, pp. 173–184. ACM, New York (1997)Brantner, M., Florescu, D., Graf, D.A., Kossmann, D., Kraska, T.: Building a database on S3. In: Intnl. Conf. on Mngmnt. of Data (SIGMOD), pp. 251–264. ACM Press, Vancouver (2008)Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. Operating Systems Review 44, 35–40 (2010)Burrows, M.: The Chubby lock service for loosely-coupled distributed systems. In: 7th Symp. on Operat. Syst. Design and Implem. (OSDI), pp. 335–350. USENIX Assoc., Seattle (2006)Junqueira, F.P., Reed, B.: The life and times of a ZooKeeper. In: 28th Annual ACM Symp. on Princ. of Distrib. Comp. (PODC), p. 4. ACM Press, Calgary (2009)MacCormick, J., Murphy, N., Najork, M., Thekkath, C.A., Zhou, L.: Boxwood: Abstractions as the foundation for storage infrastructure. In: 6th Simp. on Operat. Syst. Design and Impl. (OSDI), pp. 105–120. USENIX Assoc., San Francisco (2004)Stonebraker, M., Cattell, R.: Ten rules for scalable performance in ”simple operation” datastores. Commun. ACM 54, 72–80 (2011)Amazon Web Services LLC: Amazon SimpleDB (2011), http://aws.amazon.com/simpledb/Lamport, L.: The part-time parliament. ACM Trans. Comput. Syst. 16, 133–169 (1998)Bernstein, P.A., Reid, C.W., Das, S.: Hyder - a transactional record manager for shared flash. In: 5th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA, pp. 9–20 (2011)Bonnet, P., Bouganim, L.: Flash device support for database management. In: 5th Biennial Conf. on Innov. Data Syst. Research (CIDR), Asilomar, CA, USA, pp. 1–8 (2011)Microsoft Corp.: Windows Azure: Microsoft’s cloud services platform (2011), http://www.microsoft.com/windowsazure/VoltDB, Inc.: VoltDB technical overview: Next generation open-source SQL database with ACID for fast-scaling OLTP applications (2010), Downloadable from: http://voltdb.com/_pdf/VoltDBTechnicalOverviewWhitePaper.pd

Segurança em serviços de banco de dados em nuvem: controles para acordos de níveis de serviços

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Ciência da Computação, Florianópolis, 2013.Computação em nuvem surgiu como meio para economia de recursos através do compartilhamento de estruturas em sistemas distribuídos. Dentre os diversos modelos de entrega de serviços em nuvem estão os bancos de dados. No entanto, em ambientes corporativos, a segurança das aplicações com bancos de dados em nuvem, torna-se uma preocupação. Desde 1997, trabalhos de pesquisa vêm sendo desenvolvidos com o objetivo de minimizar alguns dos diversos problemas de segurança apontados, principalmente os relativos aos requisitos de confidencialidade. Esta dissertação focaliza o problema de segurança que pode ser encontrado, quando se celebram acordos de níveis de serviço (SLA) e contratos de serviço para bancos de dados em nuvem. No sentido de averiguar a segurança e tratar riscos, é proposto a utilização de um framework conceitual construído com um conjunto de controles internos para orientar clientes e provedores no estabelecimento de níveis de segurança. Com a utilização deste framework, controles internos foram implantados em ambientes de laboratório para nuvens públicas e privadas. Estudos de caso e análise de vulnerabilidades foram realizados para verificação da segurança, permitindo obter importantes resultados, tais como: viabilizar a escolha de provedores de serviços que possuem os controle desejados; criar métricas para o monitoramento de serviços; adequar e utilizar o PCMONS (Private Cloud MONitoring System) para realizar o monitoramento; integrar controles aos contratos de serviços e fiscalizar acordos de níveis de serviços.Abstract : Cloud computing enables resource savings through IT infrastructure sharing in distributed systems. Databases are one of cloud service delivery models. However, application security in cloud databases is a concern in enterprise environments. Since 1997, scientific researches have been developed in or- der to minimize some of these security problems, especially those relating to confidentiality. This master thesis focuses on the security problem, which can be found when entering into service level agreements (SLA) and service contracts. In order to investigate the security levels and risk treatment, we have proposed the use of a conceptual framework built with a set of internal controls to guide clients and providers for establishing acceptable security levels. By using this framework, internal controls were implemented in laboratory environment for public and private clouds. Case studies and vulnerabilities analysis were executed in order to investigate the security assurance. Important results were achieved, such as: to guide the choice of service providers holding the desired controls; create metrics for monitoring services; adapt and use of PCMONS (Private Cloud MONitoring System) monitoring tool; include controls into service contracts and manage service level agreements

Referential Integrity in Cloud NoSQL Databases

Cloud computing delivers on-demand access to essential computing services providing benefits such as reduced maintenance, lower costs, global access, and others. One of its important and prominent services is Database as a Service (DaaS) which includes cloud Database Management Systems (DBMSs). Cloud DBMSs commonly adopt the key-value data model and are called Not only SQL (NoSQL) DBMSs. These provide cloud suitable features like scalability, flexibility and robustness, but in order to provide these, features such as referential integrity are often sacrificed. In such cases, referential integrity is left to be dealt with by the applications instead of being handled by the cloud DBMSs. Thus, applications are required to either deal with inconsistency in the data (e.g. dangling references) or to incorporate the necessary logic to ensure that referential integrity is maintained. This thesis presents an Application Programming Interface (API) that serves as a middle layer between the applications and the cloud DBMS in order to maintain referential integrity. The API provides the necessary Create, Read, Update and Delete (CRUD) operations to be performed on the DBMS while ensuring that the referential integrity constraints are satisfied. These constraints are represented as metadata and four different approaches are provided to store it. Furthermore, the performance of these approaches is measured with different referential integrity constraints and evaluated upon a set of experiments in Apache Cassandra, a prominent cloud NoSQL DBMS. The results showed significant differences between the approaches in terms of performance. However, the final word on which one is better depends on the application demands as each approach presents different trade-offs