An Extended Evaluation of Schema Issues Advisor in the Azure SQL Database

The analysis of the telemetry data in Azure SQL database has revealed that most of the problems affecting the customers are due to the schema inconsistency errors. An assumption has been made that direct e-mail notifications sent to the customers about the current problems could significantly shorten time to resolve them. The prospective benefits are first validated by sending e-mail recommendations manually. Then, in an iterative way, the schema issue advisor that detects anomalies and automatically sends the appropriate notifications to the customers is implemented on top of Azure resources. Finally, an extended evaluation has confirmed the expected benefits of direct communications with the customers

CumuloNimbo: a cloud scalable multi-tier SQL database

This article presents an overview of the CumuloNimbo platform. CumuloNimbo is a framework for multi-tier applications that provides scalable and fault-tolerant processing of OLTP workloads. The main novelty of CumuloNimbo is that it provides a standard SQL interface and full transactional support without resorting to sharding and no need to know the workload in advance. Scalability is achieved by distributing request execution and transaction control across many compute nodes while data is persisted in a distributed data store. In this paper we present an overview of the platform

Serializability, not Serial: Concurrency Control and Availability in Multi-Datacenter Datastores

We present a framework for concurrency control and availability in multi-datacenter datastores. While we consider Google's Megastore as our motivating example, we define general abstractions for key components, making our solution extensible to any system that satisfies the abstraction properties. We first develop and analyze a transaction management and replication protocol based on a straightforward implementation of the Paxos algorithm. Our investigation reveals that this protocol acts as a concurrency prevention mechanism rather than a concurrency control mechanism. We then propose an enhanced protocol called Paxos with Combination and Promotion (Paxos-CP) that provides true transaction concurrency while requiring the same per instance message complexity as the basic Paxos protocol. Finally, we compare the performance of Paxos and Paxos-CP in a multi-datacenter experimental study, and we demonstrate that Paxos-CP results in significantly fewer aborted transactions than basic Paxos.Comment: VLDB201

CAP Theorem: Revision of its related consistency models

[EN] The CAP theorem states that only two of these properties can be simultaneously guaranteed in a distributed service: (i) consistency, (ii) availability, and (iii) network partition tolerance. This theorem was stated and proved assuming that "consistency" refers to atomic consistency. However, multiple consistency models exist and atomic consistency is located at the strongest edge of that spectrum. Many distributed services deployed in cloud platforms should be highly available and scalable. Network partitions may arise in those deployments and should be tolerated. One way of dealing with CAP constraints consists in relaxing consistency. Therefore, it is interesting to explore the set of consistency models not supported in an available and partition-tolerant service (CAP-constrained models). Other weaker consistency models could be maintained when scalable services are deployed in partitionable systems (CAP-free models). Three contributions arise: (1) multiple other CAP-constrained models are identified, (2) a borderline between CAP-constrained and CAP-free models is set, and (3) a hierarchy of consistency models depending on their strength and convergence is built.Muñoz-Escoí, FD.; Juan Marín, RD.; García Escriva, JR.; González De Mendívil Moreno, JR.; Bernabeu Aubán, JM. (2019). CAP Theorem: Revision of its related consistency models. The Computer Journal. 62(6):943-960. https://doi.org/10.1093/comjnl/bxy142S943960626Davidson, S. B., Garcia-Molina, H., & Skeen, D. (1985). Consistency in a partitioned network: a survey. ACM Computing Surveys, 17(3), 341-370. doi:10.1145/5505.5508Gilbert, S., & Lynch, N. (2002). Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51-59. doi:10.1145/564585.564601Muñoz-Escoí, F. D., & Bernabéu-Aubán, J. M. (2016). A survey on elasticity management in PaaS systems. Computing, 99(7), 617-656. doi:10.1007/s00607-016-0507-8Brewer, E. (2012). CAP twelve years later: How the «rules» have changed. Computer, 45(2), 23-29. doi:10.1109/mc.2012.37Attiya, H., Ellen, F., & Morrison, A. (2017). Limitations of Highly-Available Eventually-Consistent Data Stores. IEEE Transactions on Parallel and Distributed Systems, 28(1), 141-155. doi:10.1109/tpds.2016.2556669Viotti, P., & Vukolić, M. (2016). Consistency in Non-Transactional Distributed Storage Systems. ACM Computing Surveys, 49(1), 1-34. doi:10.1145/2926965Burckhardt, S. (2014). Principles of Eventual Consistency. Foundations and Trends® in Programming Languages, 1(1-2), 1-150. doi:10.1561/2500000011Herlihy, M. P., & Wing, J. M. (1990). Linearizability: a correctness condition for concurrent objects. ACM Transactions on Programming Languages and Systems, 12(3), 463-492. doi:10.1145/78969.78972Lamport. (1979). How to Make a Multiprocessor Computer That Correctly Executes Multiprocess Programs. IEEE Transactions on Computers, C-28(9), 690-691. doi:10.1109/tc.1979.1675439Ladin, R., Liskov, B., Shrira, L., & Ghemawat, S. (1992). Providing high availability using lazy replication. ACM Transactions on Computer Systems, 10(4), 360-391. doi:10.1145/138873.138877Yu, H., & Vahdat, A. (2002). Design and evaluation of a conit-based continuous consistency model for replicated services. ACM Transactions on Computer Systems, 20(3), 239-282. doi:10.1145/566340.566342Curino, C., Jones, E., Zhang, Y., & Madden, S. (2010). Schism. Proceedings of the VLDB Endowment, 3(1-2), 48-57. doi:10.14778/1920841.1920853Das, S., Agrawal, D., & El Abbadi, A. (2013). ElasTraS. ACM Transactions on Database Systems, 38(1), 1-45. doi:10.1145/2445583.2445588Chen, Z., Yang, S., Tan, S., He, L., Yin, H., & Zhang, G. (2014). A new fragment re-allocation strategy for NoSQL database systems. Frontiers of Computer Science, 9(1), 111-127. doi:10.1007/s11704-014-3480-4Kamal, J., Murshed, M., & Buyya, R. (2016). Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications. Future Generation Computer Systems, 56, 421-435. doi:10.1016/j.future.2015.09.024Elghamrawy, S. M., & Hassanien, A. E. (2017). A partitioning framework for Cassandra NoSQL database using Rendezvous hashing. The Journal of Supercomputing, 73(10), 4444-4465. doi:10.1007/s11227-017-2027-5Muñoz-Escoí, F. D., García-Escrivá, J.-R., Sendra-Roig, J. S., Bernabéu-Aubán, J. M., & González de Mendívil, J. R. (2018). Eventual Consistency: Origin and Support. Computing and Informatics, 37(5), 1037-1072. doi:10.4149/cai_2018_5_1037Fischer, M. J., Lynch, N. A., & Paterson, M. S. (1985). Impossibility of distributed consensus with one faulty process. Journal of the ACM, 32(2), 374-382. doi:10.1145/3149.21412

Mobile device reference apps to monitor and display biomedical information

Master of ScienceDepartment of Electrical and Computer EngineeringSteven WarrenSmart phones and other mobile technologies can be used to collect and display physiological information from subjects in various environments – clinical or otherwise. This thesis highlights software app reference designs that allow a smart phone to receive, process, and display biomedical data. Two research projects, described below and in the thesis body, guided this development. Android Studio was chosen to develop the phone application, after exploring multiple development options (including a cross-platform development tool), because it reduced the development time and the number of required programming languages. The first project, supported by the Kansas State University Johnson Cancer Research Center (JCRC), required a mobile device software application that could determine the hemoglobin level of a blood sample based on the most prevalent color in an image acquired by a phone camera, where the image is the result of a chemical reaction between the blood sample and a reagent. To calculate the hemoglobin level, a circular region of interest is identified from within the original image using image processing, and color information from that region of interest is input to a model that provides the hemoglobin level. The algorithm to identify the region of interest is promising but needs additional development to work properly at different image resolutions. The associated model also needs additional work, as described in the text. The second project, in collaboration with Heartspring, Wichita, KS, required a mobile application to display information from a sensor bed used to gather nighttime physiological data from severely disabled autistic children. In this case, a local data server broadcasts these data over a wireless network. The phone application gathers information about the bed over this wireless network and displays these data in user-friendly manner. This approach works well when sending basic information but experiences challenges when sending images. Future work for both project applications includes error handling and user interface improvements. For the JCRC application, a better way to account for image resolution changes needs to be developed, in addition to a means to determine whether the region of interest is valid. For the Heartspring application, future work should include improving image transmissions

Transactions and data management in NoSQL cloud databases

NoSQL databases have become the preferred option for storing and processing data in cloud computing as they are capable of providing high data availability, scalability and efficiency. But in order to achieve these attributes, NoSQL databases make certain trade-offs. First, NoSQL databases cannot guarantee strong consistency of data. They only guarantee a weaker consistency which is based on eventual consistency model. Second, NoSQL databases adopt a simple data model which makes it easy for data to be scaled across multiple nodes. Third, NoSQL databases do not support table joins and referential integrity which by implication, means they cannot implement complex queries. The combination of these factors implies that NoSQL databases cannot support transactions. Motivated by these crucial issues this thesis investigates into the transactions and data management in NoSQL databases. It presents a novel approach that implements transactional support for NoSQL databases in order to ensure stronger data consistency and provide appropriate level of performance. The novelty lies in the design of a Multi-Key transaction model that guarantees the standard properties of transactions in order to ensure stronger consistency and integrity of data. The model is implemented in a novel loosely-coupled architecture that separates the implementation of transactional logic from the underlying data thus ensuring transparency and abstraction in cloud and NoSQL databases. The proposed approach is validated through the development of a prototype system using real MongoDB system. An extended version of the standard Yahoo! Cloud Services Benchmark (YCSB) has been used in order to test and evaluate the proposed approach. Various experiments have been conducted and sets of results have been generated. The results show that the proposed approach meets the research objectives. It maintains stronger consistency of cloud data as well as appropriate level of reliability and performance

SLA-Based Performance Tuning Techniques for Cloud Databases

Today, cloud databases are widely used in many applications. The pay-per-use model of cloud databases enables on-demand access to reliable and configurable services (CPU, storage, networks, and software) that can be quickly provisioned and released with minimal management and cost for different categories of users (also called tenants). There is no need for users to set up the infrastructure or buy the software. Users without related technical background can easily manage the cloud database through the console provided by service providers, and they just need to pay to the cloud service provider only for the services they use through a service level agreement (SLA) that specifies the performance requirements and the pricing associated with the leased services. However, due to the resource sharing structure of the cloud, different tenants’ workloads compete for computing resource. This will affect tenants’ performance, especially during the workload peak time. So it is important for cloud database service providers to develop techniques that can tune the database in order to re-guarantee the SLA when a tenant’s SLA is violated. In this dissertation, two algorithms are presented in order to improve the cloud database’s performance in a multi-tenancy environment. The first algorithm is a memory buffer management algorithm called SLA-LRU and the second algorithm is a vertical database partitioning algorithm called AutoClustC. SLA-LRU takes SLA, buffer page’s frequency, buffer page’s recency, and buffer page’s value into account in order to perform buffer page replacement. The value of a buffer page represents the removal cost of this page and can be computed using the corresponding tenant’s SLA penalty function. Only the buffer pages whose tenants have the least SLA penalty cost increment will be considered by the SLA-LRU algorithm when a buffer page replacement action is taken place. AutoClustC estimates the tuning cost for resource provisioning and database partitioning, then selects the most cost saving tuning method to tune the database. If database partitioning is selected, the algorithm will use data mining to identify the database partitions accessed frequently together and will re-partition the database accordingly. The algorithm will then distribute the resulting partitions to the standby physical machines (PMs) that have the least overload score computed based on both the PMs’ communication cost and overload status. Comprehensive experiments were conducted in order to study the performance of SLA-LRU and AutoClustC using the TPC-H benchmark on both the public cloud (Amazon RDS) and private cloud. The experiment results show that SLA-LRU gives the best overall performance in terms of query response time and SLA penalty cost improvement ratio, compared to the existing memory buffer management algorithms; and AutoClustC is capable of identifying the most cost-saving cloud database tuning method with high accuracy from resource provisioning and database partitioning, and performing database re-partitioning dynamically to provide better query response time than the current partitioning configuration

Automatic intelligent database troubleshooting system in cloud environment

Oblast računarstva u klaudu se veoma intenzivno razvijala tokom prethodne decenije. Klaud okruženja i servisi koji takva okruženja pružaju postoje u različitim oblicima koji su i diskutovani u ovom radu. Prilikom njihove upotrebe uočeni su različiti problemi koji predstavljaju velike izazove za korisnike. Administratori i korisnici relacionih baza podataka su morali da rešavaju različite probleme i pre postojanja klaud platformi. Ažuriranje softvera, kao operativno zahtevan problem, pojavom relacionih baza podataka kao platforme u klaud okruženju postao je isključiva odgovornost pružaoca usluge. Nasuprot tome, problemi koji zahtevaju unapređivanje rada i optimizovanje relacione baze podataka postali su još izraženiji usled višeg nivoa apstrakcije koji donosi klaud okruženje i većeg broja relacionih baza koje je potrebno istovremeno održavati. S obzirom da se u klaud okruženju prikupljaju i čuvaju detaljne informacije o upotrebi servisa, u ovom radu je realizovan sistem koji na osnovu analize prikupljenih podataka olakšava korisnicima razumevanje funkcionisanja relacione baze podataka i pronalaženje uzroka problema koji se u radu sa njima mogu pojaviti. Nakon detaljnog pregleda uže i šire naučne oblasti, predložen je i definisan sistem koji uključuje dve vrste statističkih modela da bi se obezbedila i sveobuhvatnost i preciznost. Za donošenje konačnih odluka nad dobijenim podacima od strane statističkih modela o tome šta je uzrok a šta je posledica definisan je i ekspertski sistem. Opisan je i izgled infrastrukture koja je zasnovana na konceptu mikroservisa. Pored definisanog sistema, predstavljen je način organizacije tima sačinjenog od različitih aktera sa različitim odgovornostima. Konkretna implementacija sistema je izvršena u Azure platformi kompanije Microsoft. Implementirani sistem je potom podrobno testiran i evaluiran upotrebom realnog radnog opterecenja iz produkcionog okruženja Azure SQL relacione baze podataka tokom perioda od 6 meseci. Dobijeni rezultati su pokazali značajno unapređenje u pogledu performansi izvršavanja upita. Od pojedinačnih korisnika je dobijena i eksplicitna usmena i pismena potvrda o tome. Izvršena je i analiza dobijenih podataka o unapređenju korišćenja relacionih baza podataka svih korisnika platforme koji su se prijavili na ovaj sistem. Zaključak rada sadrži pravce i mogućnosti budućih istraživanja u ovoj oblasti.Development of the cloud computing area has grown immensely in the past decade. This work evaluates various types of cloud environments and services provided to clients. Various problems have been found in the use of the cloud and these present big challenges for the users. Users and administrators of the relational databases have encountered various problems even in times before the cloud existed. Problems such as software updates of relational databases services in s a cloud platform became the responsibility of the service provider. This was a significant improvement that reduced operational costs. However, problems with service improvement and query optimizations scaled to a higher level due to the number of the relational databases and the higher level of abstraction introduced by the cloud environment. In the cloud environment very detailed information about service usage are accumulated constantly. Here is proposed a system that, based on these data, allows users to understand how the relation database works and detects the source of the problem much easier. After a detailed analysis of related work, the system is carefully designed and elaborated. It includes two types of statistical data models to provide both recall and precision, and an expert system for making final decisions. The appropriate infrastructure is based on a microservice architecture. The project team organization was composed of several actors with different skillsets. The system is implemented within the Microsoft Azure platform. Some specific details of this implementation are also presented. The system was fully tested and evaluated using real data workload from the production environment of the Azure SQL relation database in a period of 6 months. The results have shown a significant improvement in the query execution performance. A response from the customers who used this service has shown that the user experience was significantly improved. The conclusion contains an overview of the project, suggests the ideas for improvement of the system, and discusses how the similar approach can be used in scientific areas

Reducing the Latency of Dependent Operations in Large-Scale Geo-Distributed Systems

Many applications rely on large-scale distributed systems for data management and computation. These distributed systems are complex and built from different networked services. Dependencies between these services can create a chain of dependent network I/O operations that have to be executed sequentially. This can result in high service latencies, especially when the chain consists of inter-datacenter operations. To address the latency problem of executing dependent network I/O operations, this thesis introduces new approaches and techniques to reduce the required number of operations that have to be executed sequentially for three system types. First, it addresses the high transaction completion time in geo-distributed database systems that have data sharded and replicated across different geographical regions. For a single transaction, most existing systems sequentially execute reads, writes, 2PC, and a replication protocol because of dependencies between these parts. This thesis looks at using a more restrictive transaction model in order to break dependencies and allow different parts to execute in parallel. Second, dependent network I/O operations also lead to high latency for performing leader-based state machine replication across a wide-area network. Fast Paxos introduces a fast path that bypasses the leader for request ordering. However, when concurrent requests arrive at replicas in different orders, the fast path may fail, and Fast Paxos has to fall back to a slow path. This thesis explores the use of network measurements to establish a global order for requests across replicas, allowing Fast Paxos to be effective for concurrent requests. Finally, this thesis proposes a general solution to reduce the latency impact of dependent operations in distributed systems through the use of speculative execution. For many systems, domain knowledge can be used to predict an operation’s result and speculatively execute subsequent operations, potentially allowing a chain of dependent operations to execute in parallel. This thesis introduces a framework that provides system-level support for performing speculative network I/O operations. These three approaches reduce the number of sequentially performed network I/O operations in different domains. Our performance evaluation shows that they can significantly reduce the latency of critical infrastructure services, allowing these services to be used by latency-sensitive applications