CloudJet4BigData: Streamlining Big Data via an Accelerated Socket Interface

Big data needs to feed users with fresh processing results and cloud platforms can be used to speed up big data applications. This paper describes a new data communication protocol (CloudJet) for long distance and large volume big data accessing operations to alleviate the large latencies encountered in sharing big data resources in the clouds. It encapsulates a dynamic multi-stream/multi-path engine at the socket level, which conforms to Portable Operating System Interface (POSIX) and thereby can accelerate any POSIX-compatible applications across IP based networks. It was demonstrated that CloudJet accelerates typical big data applications such as very large database (VLDB), data mining, media streaming and office applications by up to tenfold in real-world tests

Verification of Lost Data Packets and Regularizing Packets Transmission

Security in the network remains a major challenge which is highly susceptible to maliciousness. The routers especially are a major threat to the network. They can be malicious enough to disrupt the transmission of the data in the form of packets. In this paper, along with the detection of a malicious router, the transmission of packets is regularized to maximum extent possible. A Conditional Packet Buffering (CPB) algorithm is used to increase the through put of the router

Many is beautiful : commoditization as a source of disruptive innovation

Thesis (S.M.M.O.T.)--Massachusetts Institute of Technology, Sloan School of Management, Management of Technology Program, 2003.Includes bibliographical references (leaves 44-45).This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.The expression "disruptive technology" is now firmly embedded in the modern business lexicon. The mental model summarized by this concise phrase has great explanatory power for ex-post analysis of many revolutionary changes in business. Unfortunately, this paradigm can rarely be applied prescriptively. The classic formulation of a "disruptive technology" sheds little light on potential sources of innovation. This thesis seeks to extend this analysis by suggesting that many important disruptive technologies arise from commodities. The sudden availability of a high performance factor input at a low price often enables innovation in adjacent market segments. The thesis suggests main five reasons that commodities spur innovation: ** The emergence of a commodity collapses competition to the single dimension of price. Sudden changes in factor prices create new opportunities for supply driven innovation. Low prices enable innovators to substitute quantity for quality. ** The price / performance curve of a commodity creates an attractor that promotes demand aggregation. ** Commodities emerge after the establishment of a dominant design. Commodities have defined and stable interfaces. Well developed tool sets and experienced developer communities are available to work with commodities, decreasing the price of experimentation. ** Distributed architectures based on large number of simple, redundant components offer more predictable performance. Systems based on a small number of high performance components will have a higher standard deviation for uptime than high granularity systems based on large numbers of low power components. ** Distributed architectures are much more flexible than low granularity systems. Large integrated facilities often provide cost advantages when operating at the Minimum Efficient Scale of production. However, distributed architectures that can efficiently change production levels over time may be a superior solution based on the ability to adapt to changing market demand patterns. The evolution of third generation bus architectures in personal computers provides a comprehensive example of commodity based disruption, incorporating all five forces.by Richard Ellert Willey.S.M.M.O.T

Automated Network Diagnosis Prevent Problems

Software that performs well in one environment may be unusably slow in another, and determining the root cause is time-consuming and error-prone, even in environments in which all the data may be available. End users have an even more difficult time trying to diagnose system performance, since both software and network problems. Diagnosing performance degradation in distributed systems is a complex and difficult task.The source of performance stalls in a distributed system can be automatically detected and diagnosed with very limited information the dependency graph of data flows through the system, and a few counters common to almost all data processing systems. An automated approach for diagnosing performance stalls in networked systems. Flow Diagnoser requires as little as two bits of information per module to make a diagnosis: one to indicate whether the module is actively processing data, and one to indicate whether the module is waiting on its dependents. Flow Diagnoser is implemented in two distinct environments: an individual host�s networking stack, and a distributed streams processing system

Network Factors Influencing Packet Loss in Online Games

In real-time communications it is often vital that data arrive at its destination in a timely fashion. Whether it is the user experience of online games, or the reliability of tele-surgery, a reliable, consistent and predictable communications channel between source and destination is important. However, the Internet as we know it was designed to ensure that data will arrive at the desired destination instead of being designed for predictable, low-latency communication. Data traveling from point to point on the Internet is comprised of smaller packages known as packets. As these packets traverse the Internet, they encounter routers or similar devices that will often queue the packets before sending them toward their destination. Queued packets introduces a delay that depends greatly on the router configuration and the number of other packets that exist on the network. In times of high demand, packets may be discarded by the router or even lost in transmission. Protocols exist that retransmit lost packets, but these protocols introduce additional overhead and delays - costs that may be prohibitive in some applications. Being able to predict when packets may be delayed or lost could allow applications to compensate for unreliable data channels. In this thesis I investigate the effects of cross traffic and router configuration on a low bandwidth traffic stream such as that which is common in games. The experiments investigate the effects of cross traffic packet size, bit-rate, inter-packet timing and protocol used. The experiments also investigate router configurations including queue management type and the number of queues. These experiments are compared to real-world data and a mitigation strategy, where n previous packets are bundled with each new packet, is applied to both the simulated data and the real-world captures. The experiments indicate that most of the parameters explored had an impact on the packet loss. However, the real world data and simulated data differ and would require additional work to attempt to apply the lessons learned to real world applications. The mitigation strategy appeared to work well, allowing 90\% of all runs to complete without data loss. However, the mitigation strategy was implemented analytically and the actual implementation and testing has been left for future work

Automating Performance Diagnosis in Networked Systems

Diagnosing performance degradation in distributed systems is a complex and difficult task. Software that performs well in one environment may be unusably slow in another, and determining the root cause is time-consuming and error-prone, even in environments in which all the data may be available. End users have an even more difficult time trying to diagnose system performance, since both software and network problems have the same symptom: a stalled application. The central thesis of this dissertation is that the source of performance stalls in a distributed system can be automatically detected and diagnosed with very limited information: the dependency graph of data flows through the system, and a few counters common to almost all data processing systems. This dissertation presents FlowDiagnoser, an automated approach for diagnosing performance stalls in networked systems. FlowDiagnoser requires as little as two bits of information per module to make a diagnosis: one to indicate whether the module is actively processing data, and one to indicate whether the module is waiting on its dependents. To support this thesis, FlowDiagnoser is implemented in two distinct environments: an individual host's networking stack, and a distributed streams processing system. In controlled experiments using real applications, FlowDiagnoser correctly diagnoses 99% of networking-related stalls due to application, connection-specific, or network-wide performance problems, with a false positive rate under 3%. The prototype system for diagnosing messaging stalls in a commercial streams processing system correctly finds 93% of message-processing stalls, with a false positive rate of 2%

Transport protocols for multi hop wireless networks

EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Efficient techniques for end-to-end bandwidth estimation: performance evaluations and scalable deployment

Several applications, services, and protocols are conjectured to benefit from the knowledge of the end-to-end available bandwidth on a given Internet path. Unfortunately, despite the availability of several bandwidth estimation techniques, there has been only a limited adoption of these in contemporary applications. We identify two issues that contribute to this state of affairs. First, there is a lack of comprehensive evaluations that can help application developers in calibrating the relative performance of these tools--this is especially limiting since the performance of these tools depends on algorithmic, implementation, as well as temporal aspects of probing for available bandwidth. Second, most existing bandwidth estimation tools impose a large probing overhead on the paths over which bandwidth is measured. This can be a significant deterrent for deploying these tools in distributed infrastructures that need to measure bandwidth on several paths periodically. In this dissertation, we address the two issues raised above by making the following contributions: We conduct the first comprehensive black-box evaluation of a large suite of prominent available bandwidth estimation tools on a high-speed network. In this evaluation,we also illustrate the impact that technological and implementation limitations can have on the performance of bandwidth-estimation tools. We conduct the first comprehensive evaluation of available bandwidth estimation algorithms, independent of systemic and implementation biases. In this evaluation, we also illustrate the impact temporal factor such as measurement timescales have on the observed relative performance of bandwidth-estimation tools. We demonstrate that temporal properties can significantly impact the AB estimation process. We redesign the interfaces of existing bandwidth-estimation tools to allow temporal parameters to be explicitly specified and controlled. We design AB inference schemes which can be used to scalably and collaboratively infer the available bandwidth for a large set of end-to-end paths. These schemes allow an operator to select the desired operating point in the trade-off between accuracy and overhead of AB estimation. We further demonstrate that in order to monitor the bandwidth on all paths of a network we do not need access to per-hop bandwidth estimates and can simply rely on end-to-end bandwidth estimates

Impact of the Shodan Computer Search Engine on Internet-facing Industrial Control System Devices

The Shodan computer search engine crawls the Internet attempting to identify any connected device. Using Shodan, researchers identified thousands of Internet-facing devices associated with industrial controls systems (ICS). This research examines the impact of Shodan on ICS security, evaluating Shodan\u27s ability to identify Internet-connected ICS devices and assess if targeted attacks occur as a result of Shodan identification. In addition, this research evaluates the ability to limit device exposure to Shodan through service banner manipulation. Shodan\u27s impact was evaluated by deploying four high-interaction, unsolicited honeypots over a 55 day period, each configured to represent Allen-Bradley programmable logic controllers (PLC). All four honeypots were successfully indexed and identifiable via the Shodan web interface in less than 19 days. Despite being indexed, there was no increased network activity or targeted ICS attacks. Although results indicate Shodan is an effective reconnaissance tool, results contrast claims of its use to broadly identify and target Internet-facing ICS devices. Additionally, the service banner for two PLCs were modified to evaluate the impact on Shodan indexing capabilities. Findings demonstrated service banner manipulation successfully limited device exposure from Shodan queries

A Brave New World: Studies on the Deployment and Security of the Emerging IPv6 Internet.

Recent IPv4 address exhaustion events are ushering in a new era of rapid transition to the next generation Internet protocol---IPv6. Via Internet-scale experiments and data analysis, this dissertation characterizes the adoption and security of the emerging IPv6 network. The work includes three studies, each the largest of its kind, examining various facets of the new network protocol's deployment, routing maturity, and security. The first study provides an analysis of ten years of IPv6 deployment data, including quantifying twelve metrics across ten global-scale datasets, and affording a holistic understanding of the state and recent progress of the IPv6 transition. Based on cross-dataset analysis of relative global adoption rates and across features of the protocol, we find evidence of a marked shift in the pace and nature of adoption in recent years and observe that higher-level metrics of adoption lag lower-level metrics. Next, a network telescope study covering the IPv6 address space of the majority of allocated networks provides insight into the early state of IPv6 routing. Our analyses suggest that routing of average IPv6 prefixes is less stable than that of IPv4. This instability is responsible for the majority of the captured misdirected IPv6 traffic. Observed dark (unallocated destination) IPv6 traffic shows substantial differences from the unwanted traffic seen in IPv4---in both character and scale. Finally, a third study examines the state of IPv6 network security policy. We tested a sample of 25 thousand routers and 520 thousand servers against sets of TCP and UDP ports commonly targeted by attackers. We found systemic discrepancies between intended security policy---as codified in IPv4---and deployed IPv6 policy. Such lapses in ensuring that the IPv6 network is properly managed and secured are leaving thousands of important devices more vulnerable to attack than before IPv6 was enabled. Taken together, findings from our three studies suggest that IPv6 has reached a level and pace of adoption, and shows patterns of use, that indicates serious production employment of the protocol on a broad scale. However, weaker IPv6 routing and security are evident, and these are leaving early dual-stack networks less robust than the IPv4 networks they augment.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/120689/1/jczyz_1.pd