Distributed Data Streaming Algorithms for Network Anomaly Detection

Network attacks and anomalies such as DDoS attacks, service outages, email spamming are happening everyday, causing various problems for users such as financial loss, inconvenience due to service unavailability, personal information leakage and so on. Different methods have been studied and developed to tackle these network attacks, and among them data streaming algorithms are quite powerful, useful and flexible schemes that have many applications in network attack detection and identification. Data streaming algorithms usually use limited space to store aggregated information and report certain properties of the traffic in short and constant time. There are several challenges for designing data streaming algorithms. Firstly, network traffic is usually distributed and monitored at different locations, and it is often desirable to aggregate the distributed monitoring information together to detect attacks which might be low-profile at a single location; thus data streaming algorithms have to support data merging without loss of information. Secondly, network traffic is usually in high-speed and large-volume; data streaming algorithms have to process data fast and smart to save space and time. Thirdly, sometimes only detection is not useful enough and identification of targets make more sense, in which case data streaming algorithms have to be concise and reversible. In this dissertation, we study three different types of data streaming algorithms: hot item identification, distinct element counting and superspreader identification. We propose new algorithms to solve these problems and evaluate them with both theoretical analysis and experiments to show their effectiveness and improvements upon previous methods

Regulation and impact of the phytoplankton assemblage composition in the Southern Ocean

Southern Ocean waters have been identified as critical regions because of their potential to impact global climate as they play an integral role in oceanic overturning and circulation. They are also an area of deep and intermediate water formation and can potentially modulate atmospheric CO 2 concentrations. This project focuses on two regions that are seasonally limited by biologically available iron, the open HNLC Southern Ocean and the Ross Sea. The objective of this project was to determine how iron impacts the regulation of photosystem II and how the presence of particular phytoplankton species affects the ability of satellites to estimate biomass from remote sensing of ocean color. Two pulse amplitude modulated fluorometers were used to examine the photochemical efficiency of whole phytoplankton assemblages and single cells in both iron enrichment experiments and during a natural phytoplankton bloom in the Ross Sea from 2001--04. There were no significant differences in the photochemical recoveries of diatoms during the Southern Ocean Iron Experiment, with exception of Asteromphalus sp. (a centric diatom). The kinetics of increase from iron stress suggested that they occurred independently of cell surface area. A relationship between diatom abundance and SeaWiFS overestimation of chlorophyll a in the eastern Ross Sea was found in 2001--2, but this trend was not observed in other years when diatoms dominated. Under most circumstances (phytoplankton composition and size distribution), it appears that with a linear post-calibration correction we could utilize SeaWiFS for phytoplankton biomass estimates in the Ross Sea. However, we could not explain the degree that SeaWiFS over- or underestimated the in situ chlorophyll a with taxonomic composition or phytoplankton size distribution. There was distinct interannual variability in the Ross Sea over the course of our three year study. In February 2003 there was a clear secondary bloom dominated by diatoms, a feature previously unreported in the central Ross Sea. Intrusions of modified circumpolar deep water regulated the timing and magnitude of the second diatom bloom in 2003--2004 by infusing surface waters with additional iron. We could not conclusively explain the spatial variability in the phytoplankton assemblage with the fluorescence-based light utilization parameters or the mixed layer depth

Top-k aggregation queries in large-scale distributed systems

Distributed top-k query processing has recently become an essential functionality in a large number of emerging application classes like Internet traffic monitoring and Peer-to-Peer Web search. This work addresses efficient algorithms for distributed top-k queries in wide-area networks where the index lists for the attribute values (or text terms) of a query are distributed across a number of data peers. More precisely, in this thesis, we make the following distributions: We present the family of KLEE algorithms that are a fundamental building-block towards efficient top-k query processing in distributed systems. We present means to model score distributions and show how these score models can be used to reason about parameter values that play an important role in the overall performance of KLEE. We present GRASS, a family of novel algorithms based on three optimization techniques significantly increased overall performance of KLEE and related algorithms. We present probabilistic guarantees for the result quality. Moreover, we present Minerva1, a distributed search engine. Minerva offers a highly distributed (in both the data dimension and the computational dimension), scalable, and efficient solution toward the development of internet-scale search engines.Top-k Anfragen spielen eine große Rolle in einer Vielzahl von Anwendungen, insbesondere im Bereich von Informationssystemen, bei denen eine kleine, sorgfältig ausgewählte Teilmenge der Ergebnisse den Benutzern präsentiert werden soll. Beispiele hierfür sind Suchmaschinen wie Google, Yahoo oder MSN. Obwohl die Forschung in diesem Bereich in den letzten Jahren große Fortschritte gemacht hat, haben Top-k-Anfragen in verteilten Systemen, bei denen die Daten auf verschiedenen Rechnern verteilt sind, vergleichsweise wenig Aufmerksamkeit erlangt. In dieser Arbeit beschäftigen wir uns mit der effizienten Verarbeitung eben dieser Anfragen. Die Hauptbeiträge gliedern sich wie folgt. Wir präsentieren KLEE, eine Familie neuartiger Top-k-Algorithmen. Wir entwickeln Modelle mit denen Datenverteilungen beschrieben werden können. Diese Modelle sind die Grundlage für eine Schätzung diverser Parameter, die einen großen Einfluss auf die Performanz von KLEE und anderen ähnlichen Algorithmen haben. Wir präsentieren GRASS, eine Familie von Algorithmen, basierend auf drei neuartigen Optimierungstechniken, mit denen die Performanz von KLEE und ähnlichen Algorithmen verbessert wird. Wir präsentieren probabilistische Garantien für die Ergebnisgüte. Wir präsentieren Minerva, eine neuartige verteilte Peer-to-Peer-Suchmaschine

Tracing hydrological millennial-scale cycles in the late Quaternary of the Cariaco Basin and the southern Gulf of Cádiz using coccoliths and dinoflagellate cysts

Climate fluctuations in tropical to subtropical regions are coupled to hydrological changes. The reconstruction of these hydrological changes during rapid late Quaternary climate oscillations is crucial for climate change study. The present study demonstrates that coccoliths and dinoflagellate cysts are very useful as paleoclimatological indicators for the elucidation or a better understanding of the late Quaternary regional and global palaeoclimate. This study also reveals new insights in coccolithophore and dinoflagellate palaeoecology. At the outset, two fundamental studies were undertaken to strengthen dinoflagellate cysts as a proxy for paleoecological studies. Since so-called ‘standard’ palynological processing methods are still very variable and inflict damage on organic-walled microfossils to a certain extent, the effect on the determination of dinoflagellate cyst concentrations needed to be sorted out. Furthermore, since there were indications that process length variation of Lingulodinium machaerophorum is related to salinity, there was a need to assess its use for quantitative palaeosalinity reconstruction, which is of critical importance for better understanding of global climate change. Two locations were chosen for a high-resolution micropalaeontological study of hydrological millennial-scale cycles during Late Quaternary times: the Cariaco Basin, an anoxic basin offshore Venezuela and the Southern Gulf of Cádiz, offshore Morocco. Because of the high sedimentation rates, both sites contain a relatively undisturbed Late Quaternary climate record. Both record rapid, large climatic oscillations related to major hydrological changes caused by respectively the intertropical convergence zone (ITCZ) and the North Atlantic Oscillation (NAO). Results of the micropalaeontological analysis indicate that the changes in the dominance of river influx or upwelling cause important variations in plankton productivity in both locations. These variations are reflected in both assemblage changes and absolute abundance variation. Apart from the productivity reconstruction, changes in assemblage or morphology reflect changes in temperature and salinity. The seasonal and multi-year changes between upwelling and river dominated ecosystems, related in both regions to climatological shifts of respectively the ITCZ and NAO, can be extended to millennial-scale cycles and results in specific productivities, temperatures and salinities. Since millennial-scale shifts of both phenomena occur during the same periods (e.g. Younger Dryas), future work should elucidate the precise timing of both phenomena by comparing detailed multi-proxy records from north-south transects

Surface and deep hydrography across the mid-pleistocene transition; multi-proxy paleoceanographic reconstructions from the Southwest Indian Ocean

Studies of the Last Glacial Maximum have demonstrated a vital role for the carbon cycle in regulating climate on glacial-interglacial timescales, in particular emphasising the effects of storage or exhalation of CO2 from the world’s oceans. However the utility of proposed mechanisms over longer timescales, such as Early Pleistocene glacial cycles, remains unclear. The aim of this thesis is to investigate how changes in surface hydrography, deep ocean ventilation and flow, and marine biogeochemistry evolved across the Pleistocene, focusing on the Southwest Indian Ocean. New multi-proxy paleoceanographic records are generated from sediments cores in the Southwest Indian Ocean, at the boundary between the Subantarctic Southern Ocean and the subtropical Indian and Atlantic Oceans. These records, from International Ocean Discovery Program (IODP) Site U1475, span the Early to Late Pleistocene and include: Measurements of benthic C. wuellerstorfi and the deep-dwelling planktic G. truncatulinoides foraminifera stable isotopes; the accumulation of ice-rafted debris (IRD) deposited by Antarctic icebergs; and the ‘Sortable Silt’ mean grain-size proxy for near-bottom flow speed. The records are combined with published data from a nearby core (MD02-2588) and presented on the ‘Agulhas Plateau composite’. The results show that during glacial intervals, IRD deposition and near-bottom flow speeds were high, deep chemical ventilation was reduced, and the mode-water to deep-water chemical gradient was stronger relative to interglacial times. The glacial Agulhas Plateau was characterised by a northerly expansion of Southern Ocean waters accompanied by a stronger or more proximal Antarctic Circumpolar Current, a stronger chemocline, and a reduction in northern-sourced deep water presence. In these records, the Mid-Pleistocene Transition begins around 1.25 Ma with a stepwise increase in IRD deposition and higher amplitude cycles in near-bottom flow speed. The records also demonstrate that key interglacial periods are characterised by exceptionally sluggish deep-flow and a breakdown in vertical carbon stratification. Finally, the temporal phasing of the records describe a sequence of events whereby Southern Ocean surface conditions appear to lead deep circulation and global ice volume into ice ages; potential mechanisms to explain this phasing are outlined, involving shifts in the distribution of Antarctic iceberg melt