This dissertation is being presented as a partial requirement for the degree of Master of Software System Technology in the University of Sheffield. The thesis on title “Information Retrieval From The World Wide Web” is undertaken from May to September 1995. The aim of this project is to design a system which uses the Web searching technology. When this system is completed, it can be implemented to do indexing and searching any interested document that stored in the Web. This application may give some benefits to its users especially to the Department of Computer Science, University of Sheffield. Ideally, this system is developed by using one of the software engineering approaches called “Incremental delivery strategy”. The project began with the feasibility study of the techniques used in Information Retrieval and also the components involved in the World Wide Web and continued with the process of requirement analysis. The designer chose to use the data flow diagram method in order to show the design of this system. After the design was done, the system was implemented by using the C programming language and aided by the World Wide Web Library (or known as libwww). This dissertation describes the development stages of this system. Problems faced and

suggestions to recover were presented as discussions. And other relevant information was attached as appendixes

Fauziah, Baharum

English

Universiti Utara Malaysia: UUM eTheses

rmation R&ftwalthe World Wide WebFauziah Baharom29 September 1995.This dissertation is a part requirement forthe MSc in Software System Technology,Department of Computer Science,University of SheffleI11.. . UNIT KOLEKSl  KHAS*.‘”\DECLARATION(To he completed by the student and inserted into the dissertc~tion  immediately @et-  thecontents page)All sentences or passages quoted in this thesis from other people’s work have beensp~if~ally  acknowledged by clear cross referencin g to author, work and page(s). Iunderstand that failure to do this amounts to plagiarism and will be considered grounds tofailure in this thesis and the degree examination as a whole.FALIZ\hH  0AHAROM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .._Date aqJoq/W. . . . . . . . . . . . _. . . ._.  . . . , . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .iAbstractThis dissertation is being presented as a partial requirement for the degree of Master ofSoftware System Technology in the University of Sheffield. The thesis on title“Information Retrieval From The World Wide Web” is undertaken from May to-:-September 1995.The aim of this project is to design a system which uses the Web searching technology.When this system is completed , it can be implemented to do indexing and searching anyinterested document that stored in the Web. This application may give some benefits to itsusers especially to the Department of Computer Science, University of Sheffield.Ideally, this system is developed by usin,(J one of the software engineering approachescalled “Incremental delivery strategy”. The project began with the feasibility study of thetechniques used in Information Retrieval and also the components involved in the WorldWide Web and continued with the process of requirement analysis. The designerzhose  touse the data flow diagram method in order to show the design of this system. After thedesign was done, the system was implemented by using the C programming language andaided by the World Wide Web Library (or known as Zibcvw~v).This dissertation describes the development stages of this system. Problems faced andsuggestions to recover were presented as discussions. And other relevant information wasattached as appendixes.f.  .i: xDedicationDedicated tb my parents,Hajjah Embon binti Mat Isa andAllahyarham Haji Baharom bin Nik Soh.AcknowledgementsI would like to express thanks to my supervisor, Dr. Robert J. Gaizauskas, for hisconsistent support and guidance throughout my dissertation work. Despite his busyschedules, Dr. Robert J. Gaizauskas has given me his time for comments and suggestionson this dissertation.Y -A lots of thanks to Mr. Hamish Cunningham for his help especially in technical problems.Special thanks to the Universiti Utara Malaysia for their financial support. My sincerethanks also goes to the Department of Computer Science, University of Sheffield staffsand all my friends, who always given me ideas and supports to finish this dissertation.To my sisters and brothers, without whose help and unending generosity none of thiswould have been conceivable, thank you.: chI1-.) ‘.“‘..‘. ,,.,,  ‘,.C_. L I, ,(/ -7. . ..., . . , ,,~ ,,_ :..A*  “,. ,.,  .- *  , , , .1,  - ,. _._, .* ,,; >.: PContentsABSTRACTDEDICATIONACKNOWLEDGEMENTSCONTENTSCHAPTER I : INTRODUCTION.1.0 Introduction.. ........................................................................................................ 11 .1 Description Of The Problem. ................................................................................ 11.2 The Aim Of The Project. ..................................................................................... .21 .3 Objectives ............................................................................................................ .21 .4 Scopes.. ............................................................................................................... .21 .5 Project Constraints. ............................................................................................. .31 .6 Structure Of This Thesis.. ............................................................................. -. ..... 3CHAPTER 2 : BACKGROUND.2.0 Introduction.. ........................................................................................................ 52.1 Discussion Of Information Retrieval And Its Techniques.. ..................................... 52.1.1 Searching.. ..................................................................................................... 62.1.1.1  Boolean Search ....................................................................................... 62.1.2 File Structure. ............................................................................................... .72.1.2.1 Linear List File.. ..................................................................................... .72.1.2.2 Ordered Sequential File. ......................................................................... .72.1.2.3 Indexed File. .......................................................................................... .72.1.3 Wild Card. T.................................................................................... ..... .I:. ...... .92.2 The World Wide Web.. ....................................................................................... 1 02.2.1 Problems With The World Wide Web. ......................................................... 1 02.2.2 The Future Of The Web. .............................................................................. 112.3 The Wide Area Information Servers .................................................................... 1 22.3.1 How To Access WAIS From Web ?. ............................................................ 1 32.3.1.1 Direct WAIS Access. ............................................................................ 1 32.3.1.2 Web-to-WAIS Gateway. ....................................................................... 1 32.3.1.3 Customised WAIS Gateway.................................................................. 1 32.3.1.4 WAISgate. ............................................................................................ 1 42.3.2 w&index.. ................................................................................................... 1 42.4 Robots.. .............................................................................................................. 1 52.4.1 Yahoo.......................................................................................................... 1 62.4.2 Lycos........................................................................................................... 1 72.4.3 World Wide Web Worm. ............................................................................ I’72.4.4 WebCrawler ................................................................................................. 1 72.5 Libwww............................................................................... ..I.._ ......................... 1 82.5.1  Control And Data Flow Of The fibwww. ...................................................... 1 92.5.2 Core Modules. ............................................................................................ .21ContentsCHAPTER 3 : SOFTWARE ENGINEERING.3.0 Introduction . .. . . . . . 33. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..-3. I Software Life Cycle Models . . . . . . . . . . . . . . . . . . . .._............................_...._...._.._....._..........  23- 34.1 Waterfall Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 43.1.2 V Model. _. . . . . . . . ._.  . 35. . . . . . . . . . . .._........................................................................~3.1.3 Prototyping. . . . . . . . . . . .._..  . . . . . . . . . .._...................  . . . . . . . . . . . . . . . . ..__..._....  . . . . . ._.._....  . . ._.  ._..  263.2 Delivery Strategies.. _. . . . . , . . . . . . . . . ._.  . . . . . . . . . . . . . . . . ._.  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .273.3 The Model Adopted. . .._.._...................................._..............................................  28CHAPTER 4 : SYSTEM REQUIREMENT.4.0 Introduction.. ...................................................................................................... 3 04.1 Review Of The System . ...................................................................................... 304.2 Resources Requirement ...................................................................................... 3 34.2.1 Software ...................................................................................................... 334.2.2 Hardware ..................................................................................................... 33l CHAPTER 5 : SYSTEk?  DESIGN.5.0 Introduction....................................................................................................... .3445.1 System Modelling. .............................................................................................. 345.1.1  Data Flow Diagram...................................................................................... 345.2 The Design Of The System. ................................................................................ 357 CHAPTER 6 : IMPLEMENTATION AND TESTING..6.0 Introduction........................................................................................................ 396.1 Implementation................................................................................................... 396.1.1 Functions Description. ................................................................................ .396.1.1.1 Function Request-type.. ....................................................................... .396.1.1.2 Function Receive-Req. ........................................................................ .396.1.1.3 Function Validation.. ............................................................................ .396.1.1.4 Function load the document. ................................................................ .406.1.1.5  Function HTParse ................................................................................. 4 06.1.1.6 Function Generate-Index...................................................................... 4 06.1.1.7 Function Check-S top-list. .................................................................. .4 16.1.2 Description Of Data Structures Used. .......................................................... 416.1.2.1 Request Structure. ................................................................................ 416.1.2.2 Anchor Structure .................................................................................. 4 2Contents.-.,- .. 6.2  Testing. .............................................................................................................. 42- 6.2.1  Testing Objectives. ...................................................................................... 426.2.3  Type Of Testino3 ........................................................................................... 436.2.3.1 White Box Testinm3. ............................................................................... 436.2.3.2 Black Box Testino3.....................,.......................,................................... 43~~-  --  ~2.4 .Sbftware  Testing Step ~~  ~~  :-~~-  -- . ~. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 ~6.2.4.1 Unit Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446.2.4.2 Integration Testino~..........“....“.““““““““““..,.......................,.............. 446.2.4.3 Validation Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45_~.__  624.3 System Testing . . . . . . . . . . . . . . . . . . . . L  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .._...  ,..456.3 Testing Admitted. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45CHAPTER 7 : DISClXSXON.7.0 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467.1 Difficulties Encountered. ,...................................................................................  467.2 Further Works. . . . .._.._....  . . . . . . . . . . . . . . . . . . . . . . ,_.  ,_....  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ,., . . . . . . . . . . . ,477.3 Experience Obtained.. . . . . . . . . . . . . . . . . , . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47CHAPTER 8 : CONCLUSION.8.0 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48REFERENCES.APPENDICES.Appendix A : Project Diary.Appendix B : Glossary Of Terms.Appendix C : Stop List.Chapter 1. Introduction.1.0 Introduction.i II1 The aim of this project is to understand the Web search technology and use it to designand develop an information retrieval tool.1.1 Description Of The Problem.Today there are many types-of media that can be used to provide information through theworld. And with the help of expertise in science and technology, now people-can easilyget information from network semices. This network’s information service is known asWorld Wide Web. This is a rival network of the Internet which has similar functions.Users can switch from one to the other smoothly without noticing the difference.As we know, the World Wide Web is a huge information system on the Internet that usesa standard protocol ( HTTP ) and a standard for describing the structure of the documents(HTML). In general, the information retrieval system from the World Wide Web can bedivided into two main activities which are indexing and searching.This dissertation is aimed to design a system to retrieve information from the World Wider Web by referring to the concept and the design of the existing information retrieval systemlike Mosaic, Netscape and Gopher. The system will be developed by using a high level. language. At the, moment, this dissertation tasks will cover on the development ofindexing process. In general, the dissertation work carried out involved :l Understanding the components of World Wide Web ( also called WWW or the Web )and Wide Area Information Servers ( WAIS ) and how they are work.l To generate a client which can serve an indexing request. Basically the indexingrequest is provided by a user. Before executing the request, the client must identifythe type of that indexing request either only for accessing a document from the localfile system or loading a document from a remote site.l To find the linkage of all documents. When the first document is accessedsuccessfully, the client then will try to look for embedded links in it. If the documentdo not containing any parse, then the process of loading a document will be stoppedotherwise it will be continued until all the linkage documents was loaded by the client.l To send the loaded document to the indexing engine.l To create an indexing engine. This program will be used to generate an index files anddoing others task which related with this process like to generate the synonyms tableand to find stop word in the indexing request.Infornlatiolz  Retrieval from the World Wide Web iwe J.iThe contents of the thesis is for internal user only ..l*..IReferences[Rijs79][Beoh  11[Salt831[Press71[Salt891[Your891[Liu94][Eage93][Hand951[Tur195][Gaiz94][Cowl94][Laff94]Van Rijsbergen, C.J., Information Retrieval , Butterworth & Co(Publisher) Ltd., 1979.Beohm,  B., Software Engineering Economics , Prentice-Hall, 1981.Salton, G. and McGill, M.J., Introduction to Modern InformationRetrieval , McGraw-Hill, Inc., 1983.Pressman, R. S., Sofnvare  Engineering A Practitioner’s Approach ,McGraw Hill, Inc., 1987.Salton, G., Automatic Test Processing : the traizsfornzation,  analysis,and retrieval of infomation  by computer ) Addison-Wesley, Inc., 1989.Yourdon,  E., Modern Stmctured Analysis , Prentice-Hall International,1989.Liu, C., Peek, J., and et. al, Managing Internet Information Services ,O-Reilly  & Associates, Inc., 1994.Eager, B., Usilzg  the World Wide Web , Que Corporation, 1994.Handley,  M., Crowcroft, J., The World Wide Web , UCL Press, 1995.Turlington,  S. R., Walking The World Wide Web , Ventana Press, 1995.Gaizauskas, R., Notes Formal Method , 1994-l 995.Cowling, T., Notes Sofhzlare  Project Management , 1994-  1995.Lafferty, H., Notes Structured S)*stem Analysis and Design System  ,1994-1995.Itlformatiorl  Retrievaf  frot?l  the  M’orid  Il%ie  Web page 49*....cI’Hypertext References.Wwll http://www.tamu.edu/global-info/searcheng-intro.ht~[Hypr’] http://www.w3.org/hypertex~~~ibra~~serlArchitecturefDesignModel.htrnlD-b@4 http://w\vw.w3.orglhypertext~ibra~~serlArchitecturelControlF1ow.htn-JWvpr41Fbpr51http:/fw~~w.w3.org/h\lpertext/WWW/ibra~~~ser/ArchitecturelThreads. htmlhttp://~‘~w.\v3.org;hypertex~ILibrary~ser/ArchitecturelDataStructure.htmlL’Wpr61 http://\veb.nexor.co.uk/mak/doc/robots/

Information Retrieval From The World Wide Web

http://etd.uum.edu.my/595/2/Fauziah_Baharom_95.pdf

Information Retrieval From The World Wide Web

Abstract

Similar works

Full text

Available Versions

Universiti Utara Malaysia: UUM eTheses