Why (and How) Networks Should Run Themselves

The proliferation of networked devices, systems, and applications that we depend on every day makes managing networks more important than ever. The increasing security, availability, and performance demands of these applications suggest that these increasingly difficult network management problems be solved in real time, across a complex web of interacting protocols and systems. Alas, just as the importance of network management has increased, the network has grown so complex that it is seemingly unmanageable. In this new era, network management requires a fundamentally new approach. Instead of optimizations based on closed-form analysis of individual protocols, network operators need data-driven, machine-learning-based models of end-to-end and application performance based on high-level policy goals and a holistic view of the underlying components. Instead of anomaly detection algorithms that operate on offline analysis of network traces, operators need classification and detection algorithms that can make real-time, closed-loop decisions. Networks should learn to drive themselves. This paper explores this concept, discussing how we might attain this ambitious goal by more closely coupling measurement with real-time control and by relying on learning for inference and prediction about a networked application or system, as opposed to closed-form analysis of individual protocols

Cloud Migration: A Case Study of Migrating an Enterprise IT System to IaaS

This case study illustrates the potential benefits and risks associated with the migration of an IT system in the oil & gas industry from an in-house data center to Amazon EC2 from a broad variety of stakeholder perspectives across the enterprise, thus transcending the typical, yet narrow, financial and technical analysis offered by providers. Our results show that the system infrastructure in the case study would have cost 37% less over 5 years on EC2, and using cloud computing could have potentially eliminated 21% of the support calls for this system. These findings seem significant enough to call for a migration of the system to the cloud but our stakeholder impact analysis revealed that there are significant risks associated with this. Whilst the benefits of using the cloud are attractive, we argue that it is important that enterprise decision-makers consider the overall organizational implications of the changes brought about with cloud computing to avoid implementing local optimizations at the cost of organization-wide performance.Comment: Submitted to IEEE CLOUD 201

funcX: A Federated Function Serving Fabric for Science

Exploding data volumes and velocities, new computational methods and platforms, and ubiquitous connectivity demand new approaches to computation in the sciences. These new approaches must enable computation to be mobile, so that, for example, it can occur near data, be triggered by events (e.g., arrival of new data), be offloaded to specialized accelerators, or run remotely where resources are available. They also require new design approaches in which monolithic applications can be decomposed into smaller components, that may in turn be executed separately and on the most suitable resources. To address these needs we present funcX---a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. funcX's endpoint software can transform existing clouds, clusters, and supercomputers into function serving systems, while funcX's cloud-hosted service provides transparent, secure, and reliable function execution across a federated ecosystem of endpoints. We motivate the need for funcX with several scientific case studies, present our prototype design and implementation, show optimizations that deliver throughput in excess of 1 million functions per second, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than more than 130000 concurrent workers.Comment: Accepted to ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC 2020). arXiv admin note: substantial text overlap with arXiv:1908.0490

Big Data and the Internet of Things

Advances in sensing and computing capabilities are making it possible to embed increasing computing power in small devices. This has enabled the sensing devices not just to passively capture data at very high resolution but also to take sophisticated actions in response. Combined with advances in communication, this is resulting in an ecosystem of highly interconnected devices referred to as the Internet of Things - IoT. In conjunction, the advances in machine learning have allowed building models on this ever increasing amounts of data. Consequently, devices all the way from heavy assets such as aircraft engines to wearables such as health monitors can all now not only generate massive amounts of data but can draw back on aggregate analytics to "improve" their performance over time. Big data analytics has been identified as a key enabler for the IoT. In this chapter, we discuss various avenues of the IoT where big data analytics either is already making a significant impact or is on the cusp of doing so. We also discuss social implications and areas of concern.Comment: 33 pages. draft of upcoming book chapter in Japkowicz and Stefanowski (eds.) Big Data Analysis: New algorithms for a new society, Springer Series on Studies in Big Data, to appea

Program Transformations for Asynchronous and Batched Query Submission

The performance of database/Web-service backed applications can be significantly improved by asynchronous submission of queries/requests well ahead of the point where the results are needed, so that results are likely to have been fetched already when they are actually needed. However, manually writing applications to exploit asynchronous query submission is tedious and error-prone. In this paper we address the issue of automatically transforming a program written assuming synchronous query submission, to one that exploits asynchronous query submission. Our program transformation method is based on data flow analysis and is framed as a set of transformation rules. Our rules can handle query executions within loops, unlike some of the earlier work in this area. We also present a novel approach that, at runtime, can combine multiple asynchronous requests into batches, thereby achieving the benefits of batching in addition to that of asynchronous submission. We have built a tool that implements our transformation techniques on Java programs that use JDBC calls; our tool can be extended to handle Web service calls. We have carried out a detailed experimental study on several real-life applications, which shows the effectiveness of the proposed rewrite techniques, both in terms of their applicability and the performance gains achieved.Comment: 14 page

ACCOUNTING AND FINANCIAL STATEMENTS AUTO ANALYSIS SYSTEM

This project was motivated by the need to revolutionize the generation of financial statements and financial analysis process thus speeding up business decision making. The research questions were: 1) How can machine learning increase the speed of financial statement preparation and automate financial statements analysis? 2) How can businesses balance the benefits of automating financial analysis with potential concerns around privacy, data security, and bias? 3) Can the Java J2EE framework provide a reliable running environment for machine learning? The findings were: 1) Machine learning can significantly increase the accuracy and speed of financial analysis. Using machine learning algorithms, financial data can be processed and analyzed in real-time, allowing for quicker and more precise financial analysis. Machine learning models can identify patterns and trends in financial data that may not be easily detectable by humans, leading to more accurate financial statements and analysis. Additionally, machine learning can automate repetitive tasks in the financial analysis process, saving time and resources for businesses. 2) Businesses need to carefully balance the benefits of automating financial analysis with potential concerns around privacy, data security, and bias. While machine learning can offer significant advantages in terms of accuracy and speed, it also requires handling sensitive financial data. Therefore, it is crucial for businesses to implement robust data security measures to protect against potential data breaches and ensure compliance with privacy regulations. Additionally, businesses need to be mindful of potential biases in machine learning algorithms, as biased algorithms can result in biased financial analysis. Regular audits and monitoring of machine learning models should be conducted to address and mitigate any potential biases. 3) The Java J2EE framework can provide a reliable running environment for machine learning. Java J2EE (Java 2 Platform, Enterprise Edition) is a widely used and mature framework for developing enterprise applications, including machine learning applications. It offers scalability, reliability, and security features that are essential for running machine learning algorithms in a production environment. Java J2EE provides robust support for distributed computing, allowing for efficient processing of large financial datasets. Furthermore, it offers a wide range of libraries and tools for implementing machine learning algorithms, making it a viable choice for running machine learning applications in the financial industry. The conclusions were: 1) Machine learning has the potential to significantly increase the accuracy and speed of financial analysis, thereby revolutionizing the generation of financial statements and the financial analysis process. Various machine learning algorithms, such as decision trees, random forests, and deep learning algorithms, can be utilized to identify patterns, trends, and hidden risks in financial data, leading to more informed and efficient business decision making. 2) Businesses need to carefully balance the benefits of automating financial analysis with potential concerns around privacy, data security, and bias. While machine learning can offer significant advantages in terms of accuracy and speed, there are ethical considerations that need to be addressed, such as ensuring data privacy, implementing effective data security measures, and mitigating biases in machine learning algorithms used in financial analysis. Businesses should adopt a responsible approach to machine learning implementation, considering the potential risks and benefits. 3) The Java J2EE framework can provide a reliable running environment for machine learning applications, but further research is needed to evaluate the performance and scalability of machine learning models in this framework. Identifying potential optimizations for running machine learning applications at scale in the Java J2EE framework can lead to more efficient and effective implementation of machine learning in financial analysis and decision-making processes. Further research in this area can contribute to the development of robust and scalable machine learning applications for financial analysis in the business domain. Areas for further study include: 1) Exploring different machine learning algorithms and techniques to further improve the accuracy and speed of financial analysis. 2) Conducting research on the impact of machine learning on financial decision making and business performance. 3) Investigating methods for addressing and mitigating biases in machine learning algorithms used in financial analysis. 4) Evaluating the effectiveness of different data security measures in protecting sensitive financial data in machine learning applications. 5) Studying the performance and scalability of machine learning models in the Java J2EE framework and identifying potential optimizations for running machine learning applications at scale