1 research outputs found
WebParF: A Web partitioning framework for Parallel Crawlers
With the ever proliferating size and scale of the WWW [1] efficient ways of
exploring content are of increasing importance. How can we efficiently retrieve
information from it through crawling? And in this era of tera and multi-core
processors, we ought to think of multi-threaded processes as a serving
solution. So, even better how can we improve the crawling performance by using
parallel crawlers that work independently? The paper devotes to the fundamental
development in the field of parallel crawlers [4] highlighting the advantages
and challenges arising from its design. The paper also focuses on the aspect of
URL distribution among the various parallel crawling processes or threads and
ordering the URLs within each distributed set of URLs. How to distribute URLs
from the URL frontier to the various concurrently executing crawling process
threads is an orthogonal problem. The paper provides a solution to the problem
by designing a framework WebParF that partitions the URL frontier into a
several URL queues while considering the various design issues.Comment: 8pages, 7 figures, ISSN : 0975-3397 Vol.5 no.8, 201