1 research outputs found
The Implementation of Hadoop-based Crawler System and Graphlite-based PageRank-Calculation In Search Engine
Nowadays, the size of the Internet is experiencing rapid growth. As of
December 2014, the number of global Internet websites has more than 1 billion
and all kinds of information resources are integrated together on the Internet,
however,the search engine is to be a necessary tool for all users to retrieve
useful information from vast amounts of web data. Generally speaking, a
complete search engine includes the crawler system, index building systems,
sorting systems and retrieval system. At present there are many open source
implementation of search engine, such as lucene, solr, katta, elasticsearch,
solandra and so on. The crawler system and sorting system is indispensable for
any kind of search engine and in order to guarantee its efficiency, the former
needs to update crawled vast amounts of data and the latter requires real-time
to build index on newly crawled web pages and calculae its corresponding
PageRank value. It is unlikely to accomplish such huge computation tasks
depending on a single hardware implementation of the crawler system and sorting
system,from which aspect, the distributed cluster technology is brought to the
front. In this paper, we use the hadoop Map - Reduce computing framework to
implement a distributed crawler system, and use the GraphLite, a distributed
synchronous graph-computing framework, to achieve the real-time computation in
getting the PageRank value of the new crawled web page.Comment: 8 pages, 9 figure