PEER-To-PEER SEARCH ENGINE:THE ARAIGNEE PROJECTByPierre PoignantDipl6me d'Ing6nieur, Ecole Polytechnique, France, 2001Submitted to the Department of Civil and Environmental EngineeringIn Partial Fulfillment of the Requirements for the Degree ofMaster of Science in Civil and Environmental Engineeringat theMassachusetts Institute of TechnologyJune 2003@ 2003 Massachusetts Institute of TechnologyAll Rights ReservedSignature of Author......Certified by..................................................Department of Civil and Environmental EngineeringMay 9, 2003...................... I.........................Steven R. LermanPrYessor of Civil and Environmental EngineeringThesis Supervisor.................................Oral BuyukozturkProfessor of Civil and Environmental EngineeringChairman, Departmental Committee on Graduate StudiesAccepted by.........BARKERMASSACHUSETTS INSTITUTEOF TECHNOLOGYJUN 0 2 2003LIBRARIES1AbstractFast Peer-To-Peer Search Engine The Araignee ProjectByPierre PoignantSubmitted to the Department of Civil and Environmental EngineeringOn May 9, 2003In Partial Fulfillment of the Requirements for the Degree ofMaster of Science in Civil and Environmental EngineeringAccording to a survey by Yankee Group, 50% of the users search for data everyday. As a result, search engines are of utmost importance to Internet users. Currently,conducting an online search is a very frustrating experience as one is often unable to findexactly what one is looking for from the thousands of webpages. This can be attributed tothe inadequacies of existing search tools such as Internet directories and search engines.In particular, existing search tools lack a "collective memory" functionality that allow auser who is searching for the same item as a previous user to make use of the latter'ssearch results.I propose creating a new tool, "Araignee" for Internet users to simplify theirsearch by developing a new methodology of web archiving that will capitalize on theexperience of every user and personalize the search. This proposed tool has the potentialto create a new kind of collaboration between web users.Thesis Supervisor: Steven R. LermanTitle: Professor2AcknowledgementsFirst I would like to thank my advisor Professor Steve Lerman for the manyenriching discussions we have had and for his invaluable advice in the writing of thisthesis. More generally, I would like to thank the MIT Center for Educational ComputingInitiatives and the TEAL Project for their financial support that enabled me to further mystudies at MIT.I would also like to thank: My family for their constant support, Helene foralways smiling, Sophie for being so lively, Alex for our (in every sense) excitingconversations, Ardoin for being my roommate, Brice for being Brice, the winter for beingsuch a new experience, Lolo for being the master techos, Julien, Yann and Matthieu forthe olaloo spirit, My cat, The music for keeping me out of the noisy world and finallyadversity for being such a good teacher.And most importantly I would like to give a special thank to my fabulousgirlfriend Eeleen for always being a source of inspiration. She made my life at MIT anunforgettable adventure.Now, most people do not stop to read the acknowledgements. Since you did,here's a little bonus, come to Paris and you will get a free and nice French dinner fromme!3Table of contents1. The M arket................................................................................................................ 81.1. Internet - A huge m arket ................................................................................. 81.2. Search Engines: A feature the Internet can't do without ................................. 91.3. Peer-To-Peer ................................................................................................. 101.4. Conclusion from the M arket Research ............................................................. 122. The Services Provided by A raignee.................................................................... 132.1. The core services........................................................................................... 132.2. Com pany and Com m unity Applications........................................................ 152.2.1. Com panies............................................................................................. 152.2.2. Com m unities.......................................................................................... 152.3. Single U ser A pplications ............................................................................... 152.3.1. A personal space ................................................................................... 162.3.2. A personalized propositions space........................................................ 162.3.3. The best of the web............................................................................... 162.3.4. A search engine...................................................................................... 162.3.5. A content filter...................................................................................... 172.4. Potential of A raignee ..................................................................................... 173. The Com petitors.................................................................................................. 183.1. The Search Providers ...................................................................................... 183.2. Search Engines............................................................................................... 193.2.1. The Technology ...................................................................................... 193.2.2. The Incum bents...................................................................................... 213.3. W eb D irectories ............................................................................................ 233.4. N ew concepts ................................................................................................. 233.4.1. Peer-To-Peer.......................................................................................... 233.4.2. Linguistic Analysis ............................................................................... 233.4.3. Graphical U ser Interfaces ..................................................................... 2443.5. The N ew com ers ............................................................................................ 243.5.1. Exalead.................................................................................................. 243.5.2. M apstan................................................................................................. 253.5.3. K artoo ................................................................................................... 273.5.4. A ltaV ista Peer-To-Peer .......................................................................... 283.5.5. H um an Links........................................................................................... 283.5.6. O thers................................................................................................... 293.6. Conclusion ..................................................................................................... 304. The Technology Behind A raignee ...................................................................... 314.1. O verview ........................................................................................................... 314.2. The Proxy Server .......................................................................................... 324.2.1. The Proxy............................................................................................... 324.2.2. The W eb Server...................................................................................... 334.2.3. The Search Com ponent........................................................................ 334.3. The Proxy Server Architecture ..................................................................... 344.3.1. The H TTP proxy server ........................................................................ 354.3.2. The request server ................................................................................. 364.3.3. The filter technology............................................................................. 384.3.4. The transducerfilter............................................................................. 384.4. The spiderfilter............................................................................................. 384.4.1. Lex - A Lexical Analyzer Generator ................................................... 384.4.2. The D atabase......................................................................................... 394.5. The Peer-To-Peer Server................................................................................ 404.5.1. The JXTA Fram ework.......................................................................... 404.5.2. The Peer-To-Peer Structure ................................................................... 414.5.3. The Com m unication Protocol............................................................... 434.5.4. The PipeServer...................................................................................... 444.5.5. The M essage Store.................................................................................. 454.5.6. Message Routing Protocol ................................ 454.6. D eploym ent of the Softw are .......................................................................... 465. D iscussion................................................................................................................. 4855.1. Scalability ..................................................................................................... 485.1.1. Size of the D atabase............................................................................... 485.1.2. The Routing Protocol............................................................................. 485.1.3. Conclusion............................................................................................ 495.2. Reliability ...................................................................................................... 495.2.1. Crash Resistance ................................................................................... 495.2.2. Database Integrity................................................................................. 505.2.3. Results Accuracy ................................................................................... 505.3. Privacy .............................................................................................................. 515.3.1. Routing Protocol................................................................................... 515.3.2. Privacy Statem ent................................................................................. 515.3.3. Open Source .................................................................... 525.4. Profiling / Personalization............................................................................. 525.4.1. Collaborative Filtering .......................................................................... 535.5. Hum an Com puter Interaction ........................................................................ 546. Conclusion............................................................................................................... 557. References ................................................................................ 566List of figuresFigure 1: Top 15 Countries Ranked by number of Internet Users.................................. 8Figure 2: Broadband and dialup subscribers in the US.................................................... 9Figure 3: Online Activities in Europe (as %of internet users daily)............................. 10Figure 4: Architecture Of Peer-To-Peer Technology ................................................... 11Figure 5: Use of peer-to-peer in the US........................................................................ 11Figure 6:Graphical illustration of the concept of Araignee .......................................... 14Figure 7: Web of Search Providers (Source: Salomon Smith Barney).......................... 18Figure 8: M ap of the Internet ........................................................................................ 20Figure 9: Search Engine Audience Home & Work Users January 2003...................... 21Figure 10: Average Minutes Spent Searching Per Visitor January. 2003 .................... 22Figure 11: Exalead Graphical User Interface............................................................... 25Figure 12: Mapstan Graphical User Interface............................................................... 26Figure 13: Kartoo Graphical User Interface ................................................................. 27Figure 14: Human Links Graphical User Interface...................................................... 29Figure 15: Araignee Software Architecture................................................................. 31Figure 16: The proxy server architecture...................................................................... 34Figure 17: Example of an indexed Tree Structure ........................................................ 40Figure 18: The JXTA architecture (JXTA Programmer Guide).................................... 41Figure 19: The GUI Server Architecture ..................................................................... 42Figure 20: R outing Exam ple........................................................................................ 46Figure 21: How Collaborative Filtering Works ............................................................ 53Figure 22: Araignee Recommender System................................................................. 5471. The Market1.1. Internet - A huge marketToday, based on the findings in Figure 1, it is estimated that there are over 665million Internet users worldwide. By 2005, it is expected that this figure will reach 1billion. Of these, over 80% will be English speaking.USJapan64.8China54.5Gennany-30.4UK27.2South KOMe26.9Italy20.9Canada17.8France16.7India16.6Brazil=15.8Russia13.5AustraliaU10.5SpainN10.4TaiwanN9.5Note Top 15 total=496.0; Wordwide total=665.9Source: eTFOrecasts, December 2002Figure 1: Top 15 Countries Ranked by number of Internet UsersCurrently, most of the Internet users in the US are connected via the traditional phone-line modem. Broadband Internet is however increasing in popularity among the Internetusers. With reference to Figure 2, whilst only 20% of the Internet users in the US to-datehave broadband connections, this figure is estimated to climb beyond 50% by 2005. In8fact, it is expected that broadband connection will gradually become the standard form ofInternet access in many countries.20012002Unn44.144.5N Broadband U Dial-upNote: *includes residential and business DSL and cable modemsubscribersSource: Morgan Stanley, company reports, October 2002Figure 2: Broadband and dialup subscribers in the US1.2. Search Engines: A feature the Internet can't do withoutBased on the findings in Figure 3, 50% of Internet users use a search engine daily.Given the information overload that currently exists, it is unsurprising that searching forinformation on the Internet is one of the main activities of Internet users.9E-M ail ------------Search enginesInstant messaging25.0%Research products and services140%Stock quotes11.0%Online banking- 10.0%Directory services- 10.0%Download music8.0%Look for jobs04.5%Online gamingM4.0%Source: Yankee Group, December 2002Figure 3: Online Activities in Europe (as %of internet users daily)1.3. Peer-To-PeerBy supporting real-time communications, visibility between end-users, andresource sharing among PCs, peer-to-peer systems has expanded Web functionalitysignificantly. A very good overview of peer-to-peer systems and applications can befound in [Miller 2001].There are typically three different implementations of peer-to-peer technology.These include (i) autonomous networking that is completely decentralized (e.g. Gnutellaand Freenet), (ii) distributed computing that is tightly organized around a hub (e.g. SETI)and (iii) coordinated computing that has a hybrid architecture (e.g. Napster and AIM).The proposed new tool, Araignee, will fall under the last category.10Centra rerwDistributed Computingex: S.E.T.I.Coordinated ComputingNapster. AIMAutonomous NetworksFreenet, GnutellaFigure 4: Architecture Of Peer-To-Peer TechnologyCurrently, enterprises solutions implementing peer-to-peer are not widespread.However, Frost And Sullivan have forecasted a very steep growth in the useof the peer-to-peer technology in the next 5 years as indicated in Figure 5 below:- Online householdsBroadband suscribersPeer-to-Peer users2000 2001-U-2002 2003Figure 5: Use of peer-to-peer in2004 2005Source : Forrester Researchthe USFrom Figure 5, the diffusion of peer-to-peer usage highly correlates to thediffusion of broadband Internet. This is largely due to the fact that peer-to-peer systemsthrive on an environment where the user is connected to the Internet 24x7, and broadbandconnection helps create the conditions for the development of such an environment.11706050403020100_Conclusion from the Market ResearchBoth the importance of search tools to Internet users and the predicted widespreadadoption of broadband technology create excellent conditions for a peer-to-peer searchengine. In addition, with peer-to-peer technology becoming a familiar concept to many(due in part to the proliferation of applications such as Napster), there is clearly a spot inthe market for an innovative peer-to-peer search engine.The subsequent sections detail the functionalities of Araignee as well as itscompetitors. This review serves to illustrate the originality of Araignee.121.4.2. The Services Provided by Araignee2.1. The core servicesThe number of webpages is growing at an exponential pace. However, of these,only a small fraction are what many users view as "quality" webpages. Currently, webdirectories are unable to go through all the websites, while search engines such as Googledo not take into account human assessment of websites. These inadequacies thereforelead to search results that are often less than satisfactory. The proposed search engine onthe other hand can overcome these shortcomings and provide an invaluable resource tousers.With Araignee, the user installs a software on his personal computer, which willthen be used to monitor the user's surfing activities. In particular, the software records thepages visited by the user as well as the frequency with which the user visits these pages.Hence, this tool makes use of the frequency with which an individual visits a particularwebsite to indirectly infer the quality of the website. In other words, the more frequentlya website is accessed, the more likely it is a quality website. This is a reasonableassumption because the more frequently a website is visited by the user, the more likelythis website satisfies the user's needs and, by extension, is a quality website.Using the data collected on a user's surfing activities, Araignee can create profilesfor each of its users. Profiles for users can be created based on their interests by studyingthe web sites they visit as well as the keywords that appear. The more a user surfs, themore precise will be his profile. Eventually, clusters of users with similar profiles can beformed.Araignee also offer users the opportunity to benefit from a group experience byallowing users to search in group-databases. This means that someone who wishes toknow more about C# will be able to know which websites are most frequently visited byothers interested in the topic. Similarly, users will be able to join groups to share theirsearch results. For example companies can group their employees such that whensomeone from a particular group needs information on topic X, he will be able to quicklyknow which websites other members in his group frequently visit sites pertaining to topic13X. Not only can this help to reduce the search time significantly, it can also help toenhance knowledge sharing in the company.A first user searches about a theme. After a few useless websites, he finally finds the useful information.A second user searches about the same theme. Araignee capitalizes on the experience of the first user and isable to immediately provide the relevant websites to the second user.Figure 6:Graphical illustration of the concept of Araignee14Company and Community Applications2.2.1. CompaniesAraignee can be very useful to companies. Nowadays companies need to put inmore effort to capitalize on the knowledge of employees, and Araignee can serve as aninvaluable resource in this regard. Indeed, Araignee enables the employees of a companyto benefit from the web searches of all the other employees. Araignee also enablesemployees to participate in the capitalization of knowledge without adding to theirworkload. In this respect, Araignee can be considered to be a very useful knowledgemanagement tool. Furthermore, Araignee can provide information on the searches carriedout by employees on the company's Intranet. By using these statistics to makeimprovements to its Intranet, a company can then optimize its information system.2.2.2. CommunitiesAraignee services are also very well designed for communities sharing the sameinterests. Indeed the search engine will be very efficient if the users have the same kindof profiles. By profile, we refer to the interests and surfing habits of the user. In thecurrent state of the Araignee project, the user is assumed to choose his own interests andhence create his own profile. However, as we will examine in section 5.4, Araignee willalso have the capability to create a user's profile from scratch by comparing his surfhabits with other users' surf habits.2.3. Single User ApplicationsAraignee is based on a revolutionary principle, i.e. to enable a web user to benefitfrom the experience and the searches of other web users.Through analyzing the surfing habits of a user, Araignee can pinpoint a user'sinterests and favorite websites. Personalized webpages corresponding to the user'sexpectations could then be created automatically for every user by tapping on the152.2.experiences / searches of other Araignee users who share similar interests or profiles asthe user.The following sections detail the various applications that will prove useful to anAraignee user.2.3.1. A personal spaceLinks to the websites most frequently visited by a user would be displayed in thisspace. These links can be classified according to the user's interest or some other mannerdepending on the user's preference. This would allow a user to access his favoritewebsites much more quickly without having to tediously create bookmarks on hisbrowser.2.3.2. A personalized propositions spaceThe links of the websites most frequently used by other web users for the subjectslisted in the user's personal space would be displayed in this space. Thanks to Araignee'sability to generate a user's profile based on his surfing activities, users would be able tosee the favorite websites of other web users with similar profiles as themselves, a la"knowledge sharing" in the true sense of its word. Forum propositions and newsletters ofpossible interest to the user can also be displayed in this section.2.3.3. The best of the webThis section would display websites that are most often used by Internet users andhence deemed to be successful or popular websites. Websites with the greatest trafficgrowth could also be displayed in this section, so as to inform users the "in" thing on theweb.2.3.4. A search engineBy enabling users to share their search result without any action from their part,Araignee creates a new kind of passive collaboration. Armed with the ability to analyzethe surfing data of users, Araignee can display the pages that are most visited and hencedeemed to be most relevant to someone conducting a search on a particular subject. Asmore and more users get on the system, the results will be correspondingly more refined,16leading to the creation of a dynamic search engine that continuously gets better using thecumulative data about the surfing choices of its users.2.3.5. A content filterAs Araignee functions as a proxy and analyzes the content of a webpage, it cantherefore act as a content filter and block out undesirable webpages contained in a list.This list can be stored on a central server and be updated continuously by usersthemselves. As such, Araignee can even double up as a parental control filter or even anadvertisement blocker.2.4. Potential of AraigneeThe greatest benefit Araignee yields is that of the community search engine. Asindicated in Section 2.1, Araignee is a statistics-driven tool which makes use of thefrequency with which an individual visits a particular website to infer the quality of thewebsite. This information is then used to improve the surf experience of an individual orof other individuals through a variety of ways. Future collaborative content managementsystems are expected to be modeled in this manner.173. The CompetitorsAraignee is a community tool centered on web searches. Its use and design aredifferent from a typical search engine. Nevertheless, some of the functionalities offeredby Araignee do overlap with those offered by search engines and web directories. It istherefore necessary to understand the history and development of these other tools.3.1. The Search ProvidersA user makes use of a search provider to search for information or websites on theInternet. The search provider in turn extracts the search results from paid listings (wherecompanies pay to be listed), from editorial listings (where the editors of the searchprovider index the various websites) or from traditional search engine (where robotscrawl the Internet to look for the required information).MSN AOL SearohYabool Eartkt~nkSupplies --------- Receives Paid Search ListingsSupplies Reces Editorial Search ResultsSupplies---- -- ----- Receives Directory ListingsFigure 7: Web of Search Providers (Source: Salomon Smith Barney)The various search providers exchange listings using their respective search methods.Among them, Google, Teoma, Inktomi are the traditional search engines; Overture18provides paid listings whilst DMOZ is a traditional web directory. In spite of the complexmarket relations amongst the various search providers, the search results yielded cannonetheless be divided into two main categories, namely search engine results anddirectory results. In the former case, the results are automatically generated; as for thelatter case, the results are indexed manually. Araignee goes one step further bycombining both of these approaches, i.e. automatic indexation and human judgment. Thefollowing sections examine the current search engine and directories market and showwhere Araignee fits into the current scheme of things.3.2. Search Engines3.2.1. The TechnologyA search engine functions with two main components, a crawler and an indexer.The crawler surfs the web from website to website by following the links between thewebsites. The pages detected are then passed to the indexer. Using statistical analysis, theindexer analyses the content of the page, such as the language used. The words in thewebpage are then classified, with greater importance given to certain words such as thosecontained in the title or those repeated many times in the webpage.With this technique, a lot of problems can arise, for instance spain indexing bywebmasters. In such situations, "rogue" webmasters who are able to decipher thealgorithms used to classify and rank webpages, will, in their attempts to push up theranking of their webpage, make modifications to the content of the webpage so that itends up highly ranked by the search engine. In actual fact however, such webpages aretotally irrelevant to the user. More details about the algorithms used by search engines toclassify and rank webpages can be found in [Marckini 2001]. Although search engineshave developed numerous methods to overcome problems such as spain indexing,ultimately, search engines can only do a statistical analysis of words. The search enginescannot for instance analyze or grade the content of the webpage. Hence, as the number ofwebpages on the Internet increases, the accuracy of the results generated by search19engines will almost certainly decrease. Araignee overcomes this problem by making useof the human factor in the indexing of the webpages.Another problem that can arise is related to the inherent repartition of the Internetaccording to the links between the webpages. This results in search engines not beingable to automatically go through all the available webpages.In44 MiDendrit252 Millions5riokns 44milio~nsPaGesS 12 MAllmnsFigure 8: Map of the InternetThe 200-plus million webpages on the Internet can be divided among fivecategories [S&V 2000]. The 56 million heart pages are interconnected and linked to oneanother. There are 44 million pages that point to the heart, and 44 million pages that arelinked from the heart. Another 12 million are disconnected pages that belong mostly toIntranets.Whilst crawlers can easily index the heart, the rest of the Internet is difficult toreach. Hence, human intervention is needed to index most of these other pages. Araigneecan provide a seamless solution to this problem.203.2.2. The Incumbents0% 10% 20% 30%(oogle 29.5%Yahoo! 89MSN 27.6%AOL 18.4%Ask Jeeves 9.9%Overture 4.8%InifoSpace 4.5%Netscape 4.4%AltaVista 4.0%Lycos 2.4%EartfLink 2.0% Source: NielsenlNetRatingsfor SearchEngineWatch.cornLookSmart 1.7%Figure 9: Search Engine Audience Home & Work Users January 2003AltaVista was the first famous search engine of the web. Unfortunately, itsleadership position has since been eroded, and the likes of Google looks set to be the next"big thing" in the field of search engines. MSN on the other hand is the search engine ofMicrosoft, the most commonly used operating system today. By building this searchengine as part of the Internet platform, Microsoft has managed to maintain MSN'sfoothold in the world of search engines.Google was launched in 1998 amid little fanfare and with very little funding. ButInternet users have been rapidly seduced by its speed and simplicity. The principle ofGoogle is to detect the number of links pointing to another page. This allows Google tocompute the popularity of a site and to classify it in its list of results without any humanintervention. In fact, such is the attraction of Google's technology that they have beenable to sell their technology to web directories like Yahoo!.21Today, Google can be said to be the market leader, with their technology beingused by 60% of the users (if we include the use of Google on Yahoo!). However, oneproblem associated with the Google technology is fast becoming evident - with the everincreasing number of webpages that need to be indexed, the indexation process isbecoming very complex. This has slowed down the performance of the Google searchengine considerably as illustrated in Figure 8.10 20 30-I-I--11.111.010.29.68.7-M7.4M7.0-5.2E3.528.5Source: Nielsen/lNetRatingsfor SearchEngineWatch.com4038.0Figure 10: Average Minutes Spent Searching Per Visitor January. 2003A further weakness associated with Google is that given its current leadershipposition, it would be less receptive to making changes in its technology than a newcomerto the scene, as the associated risks would be far too high. This could potentially stifleany technological evolution.220GoogleYahoo!MSNAOLAsk JeevesOvertureInfospaceNeLscapeAltavistaLycosEarthLinkLookSmart3.3. Web DirectoriesWeb directories can be said to be the indirect competitors of Araignee. Users caneither search using a traditional search engine or the directory resources. However whilstweb directories are very popular, they are being used as search tools less and less often.This is because the web site indexation is done by humans, and this cannot keep up withthe exponential growth of the Internet content. In fact, companies offering webdirectories such as Yahoo understand this limitation and are instead adopting a strategy ofdiversifying their operations to offer a huge array of services so as to attract the users tostay on the directory pages.3.4. New conceptsOverall, the search engine market has been static for the past few years, with thegreatest contribution made by Google. However, more and more offerings are emergingin this market that could change everything in the next few years. In particular, the latesttechnologies developed in other fields and which simplify and speed up searchessignificantly are being used for new search engines. These are detailed below.3.4.1. Peer-To-PeerThis technique enables users to share files stored on different computers and istherefore well-designed for information, documents or files search. A handful of newsearch engines currently use or partially use this technique to share files or webaddresses. In fact, this technique has been made extremely popular by the prolific musicalfiles exchange via Napster.3.4.2. Linguistic AnalysisSearch engines use key word matching to find the pages corresponding to a user'squery. But this approach is sometimes inefficient. Indeed words can have differentmeanings; they can be used in different contexts, and only with a precise analysis of the23context can we determine if the page will satisfy the user. Some search engines thereforemake use of linguistic analysis for greater efficiency. For instance, if one sends out aquery for web pages containing the words "recycling cars", he would likely be interestedin pages containing the words "recycle car" as well. Linguistics search engines analyzethe etymology of the words in the query and display pages containing all the relevantcombination of the results instead of just what the user typed in. This can thus help tosave a lot of time for the user.3.4.3. Graphical User InterfacesSearch engines and web directories present their results in the same way: the userquery is entered in a form field, and the answer is given as a list of web sites classified byaccuracy one after another. Some recent search engines organize their results in analtogether different format: the web sites are presented graphically on a map. The searchis therefore more intuitive and can be refined easily since the links between the differentwebsites are displayed. This translates to a more convivial, less discouraging and fastersearch for the user.3.5. The Newcomers3.5.1. ExaleadWebsite: http://www.exalead.com/Exalead is a new search engine developed by Francois Bourdoncle and a teamfrom Ecole Polytechnique. This search engine integrates a linguistic module that enablesa greater flexibility during the indexation of a web page. Exalead recognizes words orverbs with different forms, regardless of the orthography or the conjugation. Indeed thesearch engine detects the root of the keyword and is therefore able to find pertinent pageseven if they do not include the matching word.The strategy of Exalead is to develop partnerships with existing portals or searchengines. The incorporation of Exalead technology on AOL.fr led to a growth from 1% to6% of the market for the portal search engine.24'A 7a 8 ' UM k t it ! t..' LMS* amd 1k we uk"S INV j A" t $ Soittsin. Uen MueIIei sdstm. R Ve ~e maraeuswnneu saat nsraae:Eme e ense aY ,J .V41 rw Y *I hI*th vcp i, Fu s Ew E SOU Sn F Th&iL WN 12TC UD &O CKVET OtF 9OW WAEO vEYMO 8ENAtC*NTO NtiAL SPOTSMAII9* imd E.um* Cam kpwa S-'% F(e60 it * A **w t*a"Xn Wilai 5) Ako O*M$s 117 W SiWI r 1YV dNSKC V Tht TeK t.rt OW JM.Wqe 9.. $n*$* I n W,. f WA4*V*pA. Er44eshT* f #% Wrmotte.4d V.AVI miied _TWOVY~~ ~ ~ ~ 1Tor o ~otc"O~~f*S yKLph4D 4. NT k PO" A Wt41* A i r.-41*miM 4*54(m Tht* m "*IIWebOU oi "fl TEn ARE.4 COMENTLYM. IMIAVthtskI 4f tAW"_WAIl2VbU 4* *Ad*M~Ii W+(4+1 tA WI <
www.google.com 80 / Google 17 21This page can therefore be accessed through every web browser. A possible nextstep would be to implement this with XML.* A Web server query: The server can answer some page queries. The filenethelp.h provides basic support for this functionality. The main problem here issecurity since access to critical documents like that below should not be granted.GET //etc/passwd HTTP/1.0The solution here is simply to disallow files with paths that start with "/" or "../",or which contain the string "I../".374.3.3. The filter technologyThe proxy is scalable, with the virtual class proxyjilter allowing filters to beadded to the service. Every request is transmitted to the filters and every filter treats eachrequest independently.4.3.4. The transducer_filterThe transducer analyses the request and checks that it is correct and legitimate i.e.the user is not trying to perform an illegal operation such as accessing a banned website.The transducer then uses the different tools of the proxy to return the answercorresponding to the request.4.4. The spider_filterThis filter is in charge of indexing all the content going through the proxy (at themoment only HTML or text). The class indextree and the subclasses are used for this.The data extracted from the various pages are stored in a database.4.4.1. Lex - A Lexical Analyzer GeneratorBackgroundLex is a program generator designed for lexical processing of character inputstreams. The user specifies regular expressions in the code given to Lex. Lex recognizesthese expressions in an input stream and partitions the input stream into strings matchingthe expressions. Lex associates the regular expressions and the program fragments of thecode. As each expression appears in the input to the program written by Lex, thecorresponding fragment is executed.The user supplies additional code beyond expression matching needed forcompleting his tasks. The program which recognizes the expressions is generated in C.Thus, a high-level expression language is provided to write the string expressions to bematched while the user's freedom to write actions is unimpaired. This avoids forcing the38user who wishes to use a string manipulation language for input analysis to writeprocessing programs in the same and often-inappropriate string handling language.Lex Source CodeThe general format of Lex source is:{definitions}{rules}(user subroutines}The following example generates a histogram of the length of the word of an input file:int lengs [100];[a-z]+ lengs[yyeng] ++;\n ;%%yywrap() {int i;printf( "Length No. words\n");for(i=0; i<100; i++)if (lengs [i] > 0) printf( "%5d%10d\n",i,lengs [i]);return (1);In the above example, the method yywrapo is a C method called by Lex. Usingthis tool we generate a code that parses HTML. We can therefore detect different HTMLtags and differentiate the content of the page from its layout. Every time a word isencountered, this word is added to the indexed tree.4.4.2. The DatabaseThe database in Araignee is composed of two structures: a list of WebPages and atree of words. Every time a new page is looked up, it is stored in the WebPages list andassigned a new id. The software then parses the page using the code created by Lex.Every time Lex encounters a new word, it adds the word to the tree. The tree has anindexed structure, optimized for look up.39words: MAD LA IM CE I0USid of pages: 102, 102, 102,34, 10, 899,46, ... 1001... 64Figure 17: Example of an indexed Tree StructureA Web Page is indexed only once and only the most recent visit is indexed. For eachpage, the following information is stored: the host, the port, the page, the title (extractedfrom the HTML) and the number of times the user has visited the page.4.5. The Peer-To-Peer Server4.5.1. The JXTA FrameworkJXTA was a project initiated by Sun in 2000. This project came under the ApacheLicense in April 2001 and has gained a huge community of developers to date. JXTA is aframework for peer-to-peer software based on the use of XML format for messages. Thegoal of JXTA is to enable the implementation of distributed network computing usingpeer-to-peer topology, and to develop basic building blocks and services that wouldenable innovative applications for peer groups.JXTA was chosen as the foundation for the first implementation of Araignee forthe following reasons:0 Apache License authorizes the use of the code in commercial products.40LJLJ" The peer-to-peer network created can be used across firewalls (because it usesHTTP)* Code for many platforms are available, including Pocket PC." The Java implementation is cross platform" Many implementations are planned for other languages (C, python etc.)" JTXA implements multiple services very useful to peer-to-peer networks. Forexample the Discovery Protocol locates advertisements from other peers. TheResolver Protocol locates peers, groups, pipes etc. The Information Protocolqueries the status of other peers, controls peer nodes etc.* A lot of open-source packages are available e.g. JXTA search (data exchange) orjnushare (file exchange protocol)IXTAApplications JXTA Community App licaons JXTAApplicationsShellJXTA~~~ Sun .indexnSeviesXTA Community Servimes JXTA -Sftrchi NorServices - i shng CommandsJXTA Peer Groups Peer Pipes Peer MonitoringCore SecurityFigure 18: The JXTA architecture (JXTA Programmer Guide)All the JXTA projects (services and applications) are based on the JXTA corecode. Araignee's peer-to-peer system is also based on the JXTA core but the inspirationfor many the concepts and code implementation comes from open source projectsprovided by Sun.4.5.2. The Peer-To-Peer StructureFigure 19 shows the different components of the JXTA server.41GUI (Flash, php, cgi, win32.-.)XML request XML answEGUI. Server (port. 8081)Routing Mode Message Stock Proxy Request ClientSearchRequest SearchAnswerBroadcast Receive UnicastJXTA PipeServer (port 9700, 9701)JXTA PeersFigure 19: The GUI Server ArchitectureThe user uses a GUI to query the local server (i.e. the GUI server). The GUIserver then uses the JXTA PipeServer to query the peer-to-peer network.The query sent to the GUI has the following format:\OThis is a very straightforward use of XML. The request is a XML tag with twoparameters, (i) the keywords stored in the request field and (ii) a profile field that will42Ireventually be used to hold the profile of the user so that queries can subsequently berestricted to the databases of users with similar profiles.The results arrive with the following format:\O\OThe results are in XML format. A result starts with an "ans" tag, followed by an"item" tag. Every "item" tag received gives a different website that corresponds to thequery. (The fields of the item tag are used to connect the page.) At the end of the list ofitems, it is followed by another "ans" to indicate the end of the result.4.5.3. The Communication ProtocolThe communication protocol used is a simplified version of the Cristian Protocol[Cristian 1996].MultiHopThe peer can parameterize the number of hops of each request before launchingthe request. As the routing protocol is stored on every peer, this parameter can thereforedepend on the speed of the Internet connection.MultiHubHubs centralize the request. The results obtained from their neighbors are thengrouped before an answer is sent to the peer which launched the request.Request StructureThe requests have a very simple format:43structure : ARAIGNEEREQ_1.0 request requestorigin nbhops id-reqrequest req:urlencodedrequest[,profil:urlencodedprofilI]id-req idmachine-idmessage or md5 hash(id-req)Every request starts with the tag ARAIGNEE._REQ_1.0. This is followed by theorigin of the request (which can be set to anything for privacy reasons), followed by thenumber of hops the request has to do on the network (this number will decreasedeverytime the request reaches a hub) and finally an unique identifier (idjreq) that is usedto tag the message.response : ARAIGNEE_ANS_1.0 request id-reqlist of resultsThe answers have a similar format. We only add the list of results behind thedescription of the message.The main advantage of this format lies in the simplicity of the parser.4.5.4. The PipeServerThe PipeServer is the entry point of all the communications related to the JXTAframework and therefore receives all the requests. It saves the idjreq (uniqueidentification of the request) for a limited time so that if a message comes back withinthat span of time, it will not be processed again. The PipeServer then acts as a hub for themessage, forwarding the message to neighbors. As this continues, the number of hops isconsequently decreased. When the PipeServer receives an answer, it adds the answer tothose previously received so that a base of answers gets built up gradually. An answer isbounced back to its emitter when answers have arrived from all the emitters that werequeried or when the message's time out is reached. This mechanism would thereforegreatly benefit if a cache were implemented so that the answers to a query previouslyposed can be easily extracted from the cache instead of having to repeat the entireprocess.444.5.5. The Message StoreThe message store remembers the previous message, gathers the answers andlaunches a cleaning thread every 10 ms to erase the timed out messages.4.5.6. Message Routing ProtocolThe routing protocol is very simple. Every peer sends the message to n neighbors(where n is a parameter). If it does not know enough neighbors, it will issue a request fornew neighbors; this will be done asynchronously via the JXTA API. The id of a messageis stored for a few seconds in the message store. If a message with the same id arrivesagain during this span of time, it is ignored. However, even though the duplicate messageis ignored, an empty answer will be sent because the peers which sent out this "repeat"message are awaiting a reply to the message. If no answer is sent, then a timed out isused. The interesting point here is that all the routing rules are stored locally. While thisallows for a very flexible implementation, unfortunately, it also creates a lot ofredundancy. This issue will be addressed in Section 5.1.2.The protocol is illustrated in Figure 20. Peer 4 first sends a request to all hisneighbors (i.e. Peers 2, 3 and 5). Peers 2, 3 and 5 then forward the request to theirneighbors and wait for their answers. Peer 1 answers Peer 2, and having answered Peer 2,sends an empty answer to Peer 3. Similarly, having received the query already previouslyfrom Peer 4, Peer 2 returns an empty answer to Peer 5. Finally, Peers 2, 3 and 5 answerPeer 4.request requestPeer 3 Per4Peer 545IF e e r Peer 2Peer 3 Per Peer 5Peer 1 Peer 2Peer 3 Peer 4 Peer 5Figure 20: Routing Example4.6. Deployment of the Software46The proxy server has been implemented and tested on a Solaris machine. Thepeer-to-peer application on the other hand, having been written in Java, can be executedfrom any operating system supporting Java.Several rounds of testing using various machines have been conducted, and thetest results show the software to run as intended.475. DiscussionIn this section, we will examine the critical success factors of the Araigneeproject. For this thesis, only the initial development of the Araignee software has beenaddressed. However, before Araignee can be deployed commercially, a number of otherissues will need to be followed up on. Future work pertaining to the Araignee projectshould address the following issues.5.1. ScalabilityThus far, Araignee has only been deployed using a handful of computers. Thescalability of the software has thus not yet been tested. We can nonetheless identify anumber of issues that are likely to arise when the software is deployed with a widernetwork.5.1.1. Size of the DatabaseThe design of the current Araignee database is very simple. If the number of sitesexceeds say 1000 per peer, it should be recoded. This upper limit of 1000 pages will notbe easily reached especially if the user only goes through the proxy for particularsearches. However, this might not be the case in a network where there is only one proxyserver for many peers, for instance a company network. The filter technology in whichthe proxy server has been written can however enable this change to take place easily.5.1.2. The Routing ProtocolThe routing rules have been stored locally. This means that a peer does not haveto connect to a central server in order to know how to route the messages. Further, forreasons of privacy, the path of the message is not stored inside the message. Whilst thiscould create quite a number of redundant messages on the network, the messages are in48general very small (typically a few bytes), hence there is little risk that the network wouldbe flooded.Nonetheless, grouping peers according to regions and allowing only onecommunication per message between regions can resolve this issue, and the JXTAframework can easily handle this.5.1.3. ConclusionIn general, we expect the peer-to-peer networks created in Araignee will berelatively small. In fact, Araignee is designed to help Internet users in very specificsearches. The database on each terminal will therefore store only a limited number ofwebpages so a group of peers possessing the same interests will unlikely be huge. Weforesee that Araignee will mainly be used for small communities or enterprise networks,so the issue of scalability should not pose a significant problem.5.2. Reliability5.2.1. Crash ResistanceAs the Araignee software is a server, the issue of reliability is of criticalimportance. The proxy needs to be running all the time; otherwise the user cannot browsethe Internet. In fact, the restarting of the server will have to be made easy so that in theevent that the software crashes, the server can be easily restarted.As the JXTA server is based on Sun's code, there is therefore a huge communityof developers which can be relied upon to report bugs and crashes. In fact, the JXTAframework has been tested extensively by this community, which has also developedmany applications using the JXTA framework. As such, the JXTA framework appears tobe relatively reliable.495.2.2. Database IntegrityIssues of integrity and reliability will undoubtedly arise when it comes todatabases. The former issue can be easily resolved if the database is local, by simplylocking the entire database when someone is writing into it. Such a move will affect theperformance only minimally. If, however, a company or a group decides to use the proxyfor more than one computer, Araignee should be rewritten with a better database.The trickier issue is what happens when the software crashes. Currently, thedatabase is stored on the disk every time 50 pages have been visited. A related issue thatneeds to be further considered is what happens should a peer decide to leave the network.In such a scenario, will all the data then be lost? To better cater to these two issues, acentral server should eventually be implemented to store some of the data.5.2.3. Results AccuracyThrough searching the database of users belonging to specific interest groups,Araignee generates more accurate search results for a user. However, any user is almostcertainly not going to surf the Internet for just one purpose (and the same one at that)over an extended period of time. The software should therefore provide a user with theflexibility to change his group setting while surfing, for instance via an applicationaccessed in the system tray. This will require some collaboration from the user.For greater accuracy of the search results, the time spent by a user on a page canalso be included as a parameter. Due to the nature of the HTTP protocol however, itmight prove difficult to assess accurately the time spent by a user on a specific page.A further method to improve the accuracy of the search results is to let usersjudge for themselves the various pages either by book-marking it through Araignee or bygrading the pages. The more highly graded webpages would correspondingly get higherweight in the display of the results. Once again this would require co-operation fromusers. Araignee however has been designed to be seamless for its users. Suchcollaboration required from users might hence compromise on this ideology.A key factor in judging the relevance of a particular webpage to a particular topicis the familiarity of the person with the subject at hand. A history professor's evaluation50of a webpage is in general deemed more valuable than a student's evaluation of the samewebpage. By this reasoning, the webpages accessed by the professor should have aheavier weight in the display of the results. Though straightforward, this fact is quitedifficult to exploit in the Araignee software. In companies or small communities,individuals whose evaluation ought to carry a heavier weight can be identified relativelyeasily. In larger groups of people where the people do not know one another, this is muchmore difficult. One solution is to give higher weight to the pages evaluated or accessedby the more active participants. The flip side of this solution however is that it might leadto more spam.5.3. PrivacyPrivacy is a crucial issue in Araignee especially since the software monitorseverything the user does on the Internet. The recent public outcries against spywaresoftware have shown the public's increasing reluctance to use such software. The uniquething about Araignee is that there is a function which allows the user to enable or disablethe software according to his preference, in order to protect his privacy. Nevertheless, weneed to be more specific about what else can be done to protect an individual's privacy.This is detailed in the following sections.5.3.1. Routing ProtocolThe protocol ensures privacy of requests. When a user wants to query Araignee,he queries the local JXTA PipeServer through the GUI server. The PipeServer then actsas a hub for this request. It is therefore impossible to track who issued a particular requeston the network.5.3.2. Privacy Statement51Whilst a high level of integrity is required to ensure that data extracted fromusers' surfing activities will be treated with the highest confidence and not be divulged, itshould also be supplemented by a software agreement with a clear privacy statement, toinform and raise awareness amongst users of the potential risks of privacy infringementarising from use of the Araignee software. Together, these can help bring aboutresponsible monitoring and enable users to benefit from a peer-to-peer search enginerather than having to take chances or being overly worried that using Araignee willinfringe upon their privacy.5.3.3. Open SourceFor the moment, Araignee is not an open source project. However, once the firstversion of Araignee is launched, it is suggested that the code be under the GPL license.Not only would this enable the project to grow with a growing community of developers,the increased transparency would also help create trust with the public by assuring themthat Araignee is not a spyware.5.4. Profiling / PersonalizationAraignee is a very useful tool for gathering statistics on a user. However thesestatistics will not be of much use if they are not analyzed. As indicated in Section 5.2.3,whilst we want to give a user the flexibility of choosing his own profile, at the same time,we want the use of Araignee to be as seamless as possible for a user. It would thereforebe ideal if a user's profile can be detected through his surf activities. A lot of work hasbeen done in the field of collaborative filtering. Some of the ideas pertaining tocollaborative filtering could perhaps be applicable in Araignee, to analyze the statisticsobtained from monitoring a user's surf activities. This is discussed in detail in thefollowing sections.525.4.1. Collaborative FilteringCollaborative Filtering is a technique for providing recommendations and/orcritiques based on statistical match of peoples' taste. The Collaborative Filtering Systemmaintains a database of users as well as the items rated by the users, for instancewebpages. The system then compares the vectors of ratings of different users and groupspeople with similar opinions together ([Basu 1998], [Breese, 1998], [Chaptini 2003]).Should a user from a particular group seek advice on a particular issue, advice will thenbe provided to him based on the experience of others in his group. The rationale behindthis is that since the different users in a particular group have more or less similarpreferences / opinions, it is likely that the advice will correlate with the opinion of theuser.Individuals PreferencesSynthesis OfPreferencesIndividual PreferenceRecomrnandation based-on collaborative filteringFigure 21: How Collaborative Filtering WorksAraignee can make great use of this technique. The search engine can query acentral service that dishes out advice according to the surf habits of the user. The centralservice then reroutes the search engine to the community that would match the user'sprofile the most. This allows Araignee to offer to a user webpages which others withsimilar profiles have found useful.53Araignee userWeb Browser Araignee searchAraignee Proxy - JXTA ServerWorld Wi de WebAraignee PeersFigure 22: Araignee Recommender SystemGUIQuery ? + profileCommunity5.5. Human Computer InteractionFor a community search engine, the more users that subscribe to it, the moreeffective it will be. The interface of the Araignee will hence need to be very friendly inorder to enhance the usability of the product to encourage uptake of the application. Inparticular, having a good interface in place will allow users to pick up the applicationeasily as opposed to leaving users frustrated if the user-interface is poorly designed.54CentralAraigneeProfiieManagerONr%.6. ConclusionGiven the information overload nowadays, Internet search tools are becoming anindispensable tool to users. The Internet search tool market used to be dominated byGoogle. However, as Internet users increasingly demand for more accurate search results,Google's position looks set to be toppled by the new generation of search engines such asAraignee which adds a human factor to the classification of results. Such new searchengines offer the best of both worlds by combining the accuracy of directories with thebreadth of results of automatic search engines. Furthermore, the predicted widespreadadoption of broadband technology in the near future (with its 24x7 connection) will helpcreate a fertile environment for the deployment of such peer-to-peer search engines.In this thesis, we have taken a first crack at laying the groundwork for thedevelopment of the Araignee software. Whilst we have deployed different technologies inthe process, much work still needs to be done to reach a stage before the Araigneesoftware can be commercialized. In particular, it should be noted that this thesis is notintended to be an ending piece that aims to give the last word on peer-to-peer searchengines. Rather, we hope it will be a seminal piece that can stir researchers into launchingfurther investigations into this area. Peer-to-peer search engines can then become thekiller application which many have predicted they would be.557. References[Basu 1998] Basu, Hirsh, and Cohen "Recommendation as Classification: Using Socialand Content-based Information in Recommendation", Recommender System Workshop'98 pp. 11-15.1998[Breese, 1998] Breese,J. S., Heckerman D., and Kadie C. "Empirical Analysis ofPredictive Algorithms for Collaborative Filtering". Proceedings of the 14th Conferenceon Uncertainty in Artificial Intelligence, pp. 43-52. 1998[Brooks 1995] Charles Brooks, Murray S. Mazer, Scott Meeks, and Jim Miller,"Application-Specific Proxy Servers as HTTP Stream Transducers", MIT/OSF ResearchInstitute, December 1995 http://www.w3.org/Conferences/WWW4/Papers/56/[Chaptini 2003] Chaptini Bassam, << Reccomender Systems and Collaborative Filtering:Litterature Review", Unpublished February 2003.[Cristian 1996] F. Cristian, "Synchronous and Asynchronous Group Communication",Communications of the ACM, vol. 39, 1996[Fielding 1996] R. Fielding, UC Irvine, H. Frystyk, "Hypertext Transfer Protocol --HTTP/1.0", MIT Laboratory for Computer Science, May 1996http://www.ietf.org/rfc/rfc1945.txt?number=1945[Gradecki 2002] Joe Gradecki, "Building Java peer-to-peer applications", Wiley Pub.,2002.[JXTA 2001] Sun Microsystems, Inc., "Project JXTA Programmer's Guide", 2001,http://www.jxta.org/docs/jxtaprogguide final.pdf[Marckini 2001] Fredrick Marckini, "Search engine positioning" Wordware Pub., 2001http://www.books24x7.com/marc.asp?isbn=155622804X[Miller 2001] Michael Miller, "Discovering P2P" San Francisco Calif SYBEX, 2001,http://www.books24x7.com/marc.asp?isbn=0782140181[Paxson 1995] Vern Paxson, "Flex, A fast scanner generator", March 1995,http://www.gnu.org/manual/flex-2.5.4/html mono/flex.html[S&V 2000] Science et Vie, "Les secrets des moteurs de recherche", Science et Vie NO998, November 2000[Searchenginewatch 2003] Internet.com, "Search Engine Watch Reports", January 2003http://www.searchenginewatch.com[STL 1994] Silicon Graphics, "Standard Template Library Programmer's Guide", 1994http://www.sgi.com/tech/stl/56

Araignee Project

Abstract

Similar works

Full text

Available Versions

DSpace@MIT