891 research outputs found
Asynchronous iterative computations with Web information retrieval structures: The PageRank case
There are several ideas being used today for Web information retrieval, and
specifically in Web search engines. The PageRank algorithm is one of those that
introduce a content-neutral ranking function over Web pages. This ranking is
applied to the set of pages returned by the Google search engine in response to
posting a search query. PageRank is based in part on two simple common sense
concepts: (i)A page is important if many important pages include links to it.
(ii)A page containing many links has reduced impact on the importance of the
pages it links to. In this paper we focus on asynchronous iterative schemes to
compute PageRank over large sets of Web pages. The elimination of the
synchronizing phases is expected to be advantageous on heterogeneous platforms.
The motivation for a possible move to such large scale distributed platforms
lies in the size of matrices representing Web structure. In orders of
magnitude: pages with nonzero elements and bytes
just to store a small percentage of the Web (the already crawled); distributed
memory machines are necessary for such computations. The present research is
part of our general objective, to explore the potential of asynchronous
computational models as an underlying framework for very large scale
computations over the Grid. The area of ``internet algorithmics'' appears to
offer many occasions for computations of unprecedent dimensionality that would
be good candidates for this framework.Comment: 8 pages to appear at ParCo2005 Conference Proceeding
A Brief History of Web Crawlers
Web crawlers visit internet applications, collect data, and learn about new
web pages from visited pages. Web crawlers have a long and interesting history.
Early web crawlers collected statistics about the web. In addition to
collecting statistics about the web and indexing the applications for search
engines, modern crawlers can be used to perform accessibility and vulnerability
checks on the application. Quick expansion of the web, and the complexity added
to web applications have made the process of crawling a very challenging one.
Throughout the history of web crawling many researchers and industrial groups
addressed different issues and challenges that web crawlers face. Different
solutions have been proposed to reduce the time and cost of crawling.
Performing an exhaustive crawl is a challenging question. Additionally
capturing the model of a modern web application and extracting data from it
automatically is another open question. What follows is a brief history of
different technique and algorithms used from the early days of crawling up to
the recent days. We introduce criteria to evaluate the relative performance of
web crawlers. Based on these criteria we plot the evolution of web crawlers and
compare their performanc
Let Your CyberAlter Ego Share Information and Manage Spam
Almost all of us have multiple cyberspace identities, and these {\em
cyber}alter egos are networked together to form a vast cyberspace social
network. This network is distinct from the world-wide-web (WWW), which is being
queried and mined to the tune of billions of dollars everyday, and until
recently, has gone largely unexplored. Empirically, the cyberspace social
networks have been found to possess many of the same complex features that
characterize its real counterparts, including scale-free degree distributions,
low diameter, and extensive connectivity. We show that these topological
features make the latent networks particularly suitable for explorations and
management via local-only messaging protocols. {\em Cyber}alter egos can
communicate via their direct links (i.e., using only their own address books)
and set up a highly decentralized and scalable message passing network that can
allow large-scale sharing of information and data. As one particular example of
such collaborative systems, we provide a design of a spam filtering system, and
our large-scale simulations show that the system achieves a spam detection rate
close to 100%, while the false positive rate is kept around zero. This system
has several advantages over other recent proposals (i) It uses an already
existing network, created by the same social dynamics that govern our daily
lives, and no dedicated peer-to-peer (P2P) systems or centralized server-based
systems need be constructed; (ii) It utilizes a percolation search algorithm
that makes the query-generated traffic scalable; (iii) The network has a built
in trust system (just as in social networks) that can be used to thwart
malicious attacks; iv) It can be implemented right now as a plugin to popular
email programs, such as MS Outlook, Eudora, and Sendmail.Comment: 13 pages, 10 figure
On the Efficacy of Live DDoS Detection with Hadoop
Distributed Denial of Service flooding attacks are one of the biggest
challenges to the availability of online services today. These DDoS attacks
overwhelm the victim with huge volume of traffic and render it incapable of
performing normal communication or crashes it completely. If there are delays
in detecting the flooding attacks, nothing much can be done except to manually
disconnect the victim and fix the problem. With the rapid increase of DDoS
volume and frequency, the current DDoS detection technologies are challenged to
deal with huge attack volume in reasonable and affordable response time.
In this paper, we propose HADEC, a Hadoop based Live DDoS Detection framework
to tackle efficient analysis of flooding attacks by harnessing MapReduce and
HDFS. We implemented a counter-based DDoS detection algorithm for four major
flooding attacks (TCP-SYN, HTTP GET, UDP and ICMP) in MapReduce, consisting of
map and reduce functions. We deployed a testbed to evaluate the performance of
HADEC framework for live DDoS detection. Based on the experiments we showed
that HADEC is capable of processing and detecting DDoS attacks in affordable
time
- …