Very Large Scale Information Retrieval

Abstract. This chapter is based on a series of five lectures presented at the EL

Efficiency Considerations in Very Large Information Retrieval Servers

It is estimated that the World Wide Web now contains more than twenty million different content areas, presented on more than 320 million web pages, and one million web servers--- and it is doubling every nine months [16, 17]. To combat this, Moore's law suggests that computational resource will continue to double every eighteen months. This suggests that--- although both curves are exponential---we may be losing ground in the quest to search and find useful information. We briefly review some of the algorithms known to be useful for very large scale information retrieval and suggest opportunities for improvement in this area. Without research in this area, the web and other electronic information sources will become large haystacks in which no one will be able to find any needles. 1 Introduction Information Retrieval (IR) is devoted to finding "relevant" documents. Simple pattern matching is one approach, but many other techniques exist. Unlike most algorithms found in Comp..