74 research outputs found
Supporting disconnection operations through cooperative hoarding
Mobile clients often need to operate while disconnected from the network due to limited battery life and network coverage. Hoarding supports this by fetching frequently accessed data into clients' local caches prior to disconnection. Existing work on hoarding have focused on improving data accessibility for individual mobile clients. However, due to storage limitations, mobile clients may not be able to hoard every data object they need. This leads to cache misses and disruption to clients' operations. In this paper, a new concept called cooperative hoarding is introduced to reduce the risks of cache misses for mobile clients. Cooperative hoarding takes advantage of group mobility behaviour, combined with peer cooperation in ad-hoc mode, to improve hoard performance. Two cooperative hoarding approaches are proposed that take into account access frequency, connection probability, and cache size of mobile clients so that hoarding can be performed cooperatively. Simulation results show that the proposed methods significantly improve cache hit ratio and provides better support for disconnected operations compared to existing schemes
Performance optimization problem in speculative prefetching
Speculative prefetching has been proposed to improve the response time of network access. Previous studies in speculative prefetching focus on building and evaluating access models for the purpose of access prediction. This paper investigates a complementary area which has been largely ignored, that of performance modeling. We analyze the performance of a prefetcher that has uncertain knowledge about future accesses. Our performance metric is the improvement in access time, for which we derive a formula in terms of resource parameters (time available and time required for prefetehing) and speculative parameters (probabilities for next access). We develop a prefetch algorithm to maximize the improvement in access time. The algorithm is based on finding the best solution to a stretch knapsack problem, using theoretically proven apparatus to reduce the search space. An integration between speculative prefetching and caching is also investigated
Contention management for distributed data replication
PhD ThesisOptimistic replication schemes provide distributed applications with access
to shared data at lower latencies and greater availability. This is
achieved by allowing clients to replicate shared data and execute actions
locally. A consequence of this scheme raises issues regarding shared data
consistency. Sometimes an action executed by a client may result in
shared data that may conflict and, as a consequence, may conflict with
subsequent actions that are caused by the conflicting action. This requires
a client to rollback to the action that caused the conflicting data,
and to execute some exception handling. This can be achieved by relying
on the application layer to either ignore or handle shared data inconsistencies
when they are discovered during the reconciliation phase of an
optimistic protocol.
Inconsistency of shared data has an impact on the causality relationship
across client actions. In protocol design, it is desirable to preserve the
property of causality between different actions occurring across a distributed
application. Without application level knowledge, we assume
an action causes all the subsequent actions at the same client. With
application knowledge, we can significantly ease the protocol burden of
provisioning causal ordering, as we can identify which actions do not
cause other actions (even if they precede them). This, in turn, makes
possible the client’s ability to rollback to past actions and to change
them, without having to alter subsequent actions. Unfortunately, increased
instances of application level causal relations between actions
lead to a significant overhead in protocol. Therefore, minimizing the
rollback associated with conflicting actions, while preserving causality,
is seen as desirable for lower exception handling in the application layer.
In this thesis, we present a framework that utilizes causality to create
a scheduler that can inform a contention management scheme to reduce
the rollback associated with the conflicting access of shared data.
Our framework uses a backoff contention management scheme to provide
causality preserving for those optimistic replication systems with high
causality requirements, without the need for application layer knowledge.
We present experiments which demonstrate that our framework reduces
clients’ rollback and, more importantly, that the overall throughput of
the system is improved when the contention management is used with
applications that require causality to be preserved across all actions
Rethinking Storage Management for Data Processing Pipelines in Cloud Data Centers
Data processing frameworks such as Apache Beam and Apache Spark are used for
a wide range of applications, from logs analysis to data preparation for DNN
training. It is thus unsurprising that there has been a large amount of work on
optimizing these frameworks, including their storage management. The shift to
cloud computing requires optimization across all pipelines concurrently running
across a cluster. In this paper, we look at one specific instance of this
problem: placement of I/O-intensive temporary intermediate data on SSD and HDD.
Efficient data placement is challenging since I/O density is usually unknown at
the time data needs to be placed. Additionally, external factors such as load
variability, job preemption, or job priorities can impact job completion times,
which ultimately affect the I/O density of the temporary files in the workload.
In this paper, we envision that machine learning can be used to solve this
problem. We analyze production logs from Google's data centers for a range of
data processing pipelines. Our analysis shows that I/O density may be
predictable. This suggests that learning-based strategies, if crafted
carefully, could extract predictive features for I/O density of temporary files
involved in various transformations, which could be used to improve the
efficiency of storage management in data processing pipelines
Scalable system for smart urban transport management
Efficient management of smart transport systems requires the integration of various sensing technologies, as well as fast processing of a high volume of heterogeneous data, in order to perform smart analytics of urban networks in real time. However, dynamic response that relies on intelligent demand-side transport management is particularly challenging due to the increasing flow of transmitted sensor data. In this work, a novel smart service-driven, adaptable middleware architecture is proposed to acquire, store, manipulate, and integrate information from heterogeneous data sources in order to deliver smart analytics aimed at supporting strategic decision-making. The architecture offers adaptive and scalable data integration services for acquiring and processing dynamic data, delivering fast response time, and offering data mining and machine learning models for real-time prediction, combined with advanced visualisation techniques. The proposed solution has been implemented and validated, demonstrating its ability to provide real-time performance on the existing, operational, and large-scale bus network of a European capital city
Recommended from our members
Provenance as First Class Cloud Data
Digital provenance is meta-data that describes the ancestry or history of a digital object. Most work on provenance focuses on how provenance increases the value of data to consumers. However, provenance is also valuable to storage providers. For example, provenance can provide hints on access patterns, detect anomalous behavior, and provide enhanced user search capabilities. As the next generation storage providers, cloud vendors are in the unique position to capitalize on this opportunity to incorporate provenance as a fundamental storage system primitive. To date, cloud offerings have not yet done so. We provide motivation for providers to treat provenance as first class data in the cloud and based on our experience with provenance in a local storage system, suggest a set of requirements that make provenance feasible and attractive.Engineering and Applied Science
On I/O Performance and Cost Efficiency of Cloud Storage: A Client\u27s Perspective
Cloud storage has gained increasing popularity in the past few years. In cloud storage, data are stored in the service provider’s data centers; users access data via the network and pay the fees based on the service usage. For such a new storage model, our prior wisdom and optimization schemes on conventional storage may not remain valid nor applicable to the emerging cloud storage.
In this dissertation, we focus on understanding and optimizing the I/O performance and cost efficiency of cloud storage from a client’s perspective. We first conduct a comprehensive study to gain insight into the I/O performance behaviors of cloud storage from the client side. Through extensive experiments, we have obtained several critical findings and useful implications for system optimization. We then design a client cache framework, called Pacaca, to further improve end-to-end performance of cloud storage. Pacaca seamlessly integrates parallelized prefetching and cost-aware caching by utilizing the parallelism potential and object correlations of cloud storage. In addition to improving system performance, we have also made efforts to reduce the monetary cost of using cloud storage services by proposing a latency- and cost-aware client caching scheme, called GDS-LC, which can achieve two optimization goals for using cloud storage services: low access latency and low monetary cost. Our experimental results show that our proposed client-side solutions significantly outperform traditional methods. Our study contributes to inspiring the community to reconsider system optimization methods in the cloud environment, especially for the purpose of integrating cloud storage into the current storage stack as a primary storage layer
QoS-Aware Resource Management for Multi-phase Serverless Workflows with Aquatope
Multi-stage serverless applications, i.e., workflows with many computation
and I/O stages, are becoming increasingly representative of FaaS platforms.
Despite their advantages in terms of fine-grained scalability and modular
development, these applications are subject to suboptimal performance, resource
inefficiency, and high costs to a larger degree than previous simple serverless
functions.
We present Aquatope, a QoS-and-uncertainty-aware resource scheduler for
end-to-end serverless workflows that takes into account the inherent
uncertainty present in FaaS platforms, and improves performance predictability
and resource efficiency. Aquatope uses a set of scalable and validated Bayesian
models to create pre-warmed containers ahead of function invocations, and to
allocate appropriate resources at function granularity to meet a complex
workflow's end-to-end QoS, while minimizing resource cost. Across a diverse set
of analytics and interactive multi-stage serverless workloads, Aquatope
significantly outperforms prior systems, reducing QoS violations by 5x, and
cost by 34% on average and up to 52% compared to other QoS-meeting methods
A Test Driven Approach to Develop Web-Based Machine Learning Applications
The purpose of this thesis is to propose the design and architecture of a testable, scalable, and ef-cient web-based application that models and implements machine learning applications in cancer prediction. There are various components that form the architecture of our web-based application including server, database, programming language, web framework, and front-end design. There are also other factors associated with our application such as testability, scalability, performance, and design pattern. Our main focus in this thesis is on the testability of the system while consid- ering the importance of other factors as well.
The data set for our application is a subset of the Surveillance, Epidemiology, and End Results (SEER) Program of the National Cancer Institute. The application is implemented with Python as the back-end programming language, Django as the web framework, Sqlite as the database, and the built-in server of the Django framework. The front layer of the application is built using HTML, CSS and various JavaScript libraries.
Our Implementation and Installation is augmented with testing phase that include unit and functional testing. There are other layers such as deploying, caching, security, and scaling that will be briefly discussed
- …