A Scalable and Fault Tolerant Architecture for Distributed Web Resource Discovery

Abstract

Most Internet search engines are built on a centralised design, and will therefore not cope with the expected future growth in information and query volume. A distributed approach to Internet resource discovery seems necessary. Many distributed designs have been proposed, but their scaleability is largely unknown. We propose a distributed design based on the use of classified metadata, which can be proven to be extremely scaleable, and has interesting fault-tolerant properties. We will show the principles of this design, called the Content- Sensitive Infrastructure (CSI), and demonstrate its properties through formal analysis and simulation