Multimodal dialogue systems provide multiple modalities in the form of speech, mouse clicking, drawing or touch that can enhance human-computer interaction. However, one of the drawbacks of the existing multimodal systems is that they are highly domain-speciﬁc and they do not allow information to be shared across different providers. In this paper, we propose a semantic multimodal system, called Semantic Restaurant Finder, for the Semantic Web in which the restaurant information in different city/country/language are constructed as ontologies to allow the information to be sharable. From the Semantic Restaurant Finder, users can make use of the semantic restaurant knowledge distributed from different locations on the Internet to ﬁnd the desired restaurants

He, Yulan

Hui, Siu Cheung

Quan, Thanh Tho

English

Open Research Online

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsA multimodal restaurant finder for semantic webConference or Workshop ItemHow to cite:He, Yulan; Quan, Thanh Tho and Hui, Siu Cheung (2007). A multimodal restaurant finder for semantic web.In: 4th International Conference on Computing and Telecommunication Technologies (RIVF 2006), 12-16 Feb 2006,Ho Chi Minh city, Vietnam.For guidance on citations see FAQs.c© 2007 The AuthorsVersion: Not SetCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukINTERNALTIONAL IEEE CONFERENCE ON COMPUTER SCIENCES - RIVF’06 1A Multimodal Restaurant Finder forSemantic WebYulan He, Thanh Tho Quan and Siu Cheung HuiSchool of Computer EngineeringNanyang Technological UniversitySingaporeEmail:{asylhe,pa0218164b,asschui}@ntu.edu.sgAbstract— Multimodal dialogue systemsprovide multiple modalities in the form ofspeech, mouse clicking, drawing or touchthat can enhance human-computer interac-tion. However, one of the drawbacks of theexisting multimodal systems is that they arehighly domain-specific and they do not allowinformation to be shared across differentproviders. In this paper, we propose a se-mantic multimodal system, called SemanticRestaurant Finder, for the Semantic Webin which the restaurant information in dif-ferent city/country/language are constructedas ontologies to allow the information tobe sharable. From the Semantic RestaurantFinder, users can make use of the semanticrestaurant knowledge distributed from dif-ferent locations on the Internet to find thedesired restaurants.Index Terms— Multimodal dialogue sys-tems, Semantic Web, OntologyI. INTRODUCTIONConversation or dialogue is the mostnatural way for people to exchange ideas,convey information, and express feelings,etc. Therefore, speech interfaces offer greatpotential for enhancing human-computerinteraction (HCI). In recent years, lots ofresearch work have been done to enhancethe speech communication by using com-plimentary information from gestures. Thefirst known multimodal interface was builtin 1980 by Bolt [1] in which shapes couldbe created, moved, copied, removed, andnamed using the combination of speechand pointing to resolve deictic referencessuch as that and there. Cheyer [2] describesa multimodal map-based application for atravel planning domain by combining hand-writing, gesture and speech. More recently,Johnston et al. [3]–[5] developed a multi-modal system called MATCH (MultimodalAccess to City Help) which provides amobile multimodal speech-pen interface torestaurant and subway information for NewYork City. Pele´ et al. [6] developed a virtualhuman called Nesto which is an interactive3D embodied conversational agent.One of the drawbacks of the exist-ing multimodal systems is that they arehighly domain-specific and they do notallow information to be shared across dif-ferent providers. In this paper, we pro-pose a semantic multimodal dialogue sys-tem, called Semantic Restaurant Finder(SRF), using Semantic Web services inwhich the restaurant information in differ-ent city/country/language are constructedas ontologies to allow the information tobe sharable.The rest of the paper is organized asfollows. Section II describes the currentdevelopment of Semantic Web and Se-mantic Web services. Section III discussesthe system support for the multimodal Se-mantic Restaurant Finder. Section IV givesthe Semantic Web architecture for SRF.The approach for automatic generation ofrestaurant ontology is then discussed insection V followed by the performanceanalysis presented in section VI. Finally,section VII concludes the paper.II. SEMANTIC WEB AND SERVICESVast information are available on theWeb. The Semantic Web extends the abil-ity of the World Wide Web by develop-ing standards and tools that allow mean-ing to be added to the content of webpages.Ontology is a formal, explicit spec-ification of a shared conceptualization. ItINTERNALTIONAL IEEE CONFERENCE ON COMPUTER SCIENCES - RIVF’06 2is the key enabling technology for the Se-mantic Web which provides a frameworkfor the sharing and reuse of data on theWeb.Web Services [7] have recently becomean important feature in the Web commu-nity since they provide a new level ofinteroperability between applications. WebServices are based on agent technology[8]. A Web Service is identified by a URI(Universal Resource Identifier) [9], whosepublic interfaces and bindings are definedand described using XML-based servicedescription languages. Typical Web Ser-vice techniques such as Web Services De-scription Services (WSDS) [10] or Univer-sal Description, Discovery and Integration(UDDI) [11] provide technical informationon service accessibility (including the ser-vice address and the service port), but itlacks the description on service capability.In addition, functions offered by typicalWeb Services are accessible, but not fullyunderstandable by other programs.This problem can be tackled by the Se-mantic Web. The Semantic Web uses ontol-ogy as a standard for knowledge presenta-tion. Therefore, knowledge can be shared,reused and exchanged on the SemanticWeb. Based on OWL (Web Ontology Lan-guage) [12], an ontology language descrip-tion language, OWL-S (former DAML-S)[13] was introduced to describe Web Ser-vices in a semantic manner.III. MULTIMODAL DIALOGUE SUPPORTFigure 1 depicts the multimodal dialoguesupport for Semantic Restaurant Finder.Users can interact with the SFR systemin natural ways such as speech, typing,or mouse clicking and drawing to enquirerestaurant information. For example, userscan request a list of restaurants by saying”show me seafood or general restaurantsnear Orchard Road”. An example of restau-rant finding result is displayed in Figure 2.The multimodal Semantic RestaurantFinding system consists of eight maincomponents. The Open Agent Architecture(OAA2) [14], [15] is used to facilitate mul-tiple asynchronous communications amongvarious components. The components arebriefly discussed as follows.• Speech Recognizer. Implemented us-ing the ATK toolkit [16], which is thereal-time API for HTK [17].• Speech Semantic Parser. Implementedusing the Hidden Vector State (HVS)model [18], [19]. This is illustrated inFigure 3 which shows the sequence ofHVS stack states corresponding to thegiven parse tree. The details of howthis is done are given in [19].• Gesture Recognizer. The gesture rec-ognizer used is a sample C++ Win32application called GestureApp [20].• Gesture Semantic Parser. The gesturesemantic parser can be rather treatedas an interpreter to map the recognizedraw gesture or symbol into a semanticpartial frame.• Multimodal Input Fuser. Multimodalinput fusing is done by using thetime correlates between speech events(words) and the pointing, drawing ormouse clicking events.• Dialogue Manager. The dialoguemanager controls the interaction withthe user.• Response Generator. The responsegenerator produces an appropriate re-sponse according the the instructionsgiven by the dialogue manager.• Speech Synthesizer. The Festival 2.0speech synthesizer [21] is used totransform text into speech.IV. SEMANTIC WEB ARCHITECTUREFOR SEMANTIC RESTAURANT FINDERFigure 4 shows the system architec-ture of SRF using Semantic Web services.There might be several restaurant ontolo-gies available on the Semantic Web. Wewill discuss the ontology generation pro-cess in section V. The Semantic Web archi-tecture consists of three major components:Web Service Provider, Matchmaking Agentand Web Service Requester.There might be many Web ServiceProviders on the Web. Each Web ServiceProvider accesses the restaurant ontologyto retrieve restaurant-related informationfor a given user request. These Web ServiceProviders must register its services withthe Matchmaking Agent which serves as aregistry and look-up service repository byINTERNALTIONAL IEEE CONFERENCE ON COMPUTER SCIENCES - RIVF’06 3Mouse Clickingor DrawingRestaurantOntologiesTextResponseSpeechRecognizerSpeechSynthesizerResponseGeneratorSpeechSemanticParserDialogueManagerMultimodalInput FuserPartialFrameSystemActionIntegratedInputSpeechOutputDisplayOutputRestaurantInformationSpeechInputGestureRecognizerGesture GestureSemanticParserPartialFrameFig. 1. Multimodal dialogue support for SRF.Fig. 2. An example of restaurant finding result.DUMMYSSSELOCATIONshow  meseafood  or generalrestaurantsOrchard Road sent_endRESTAURANTsent_startLOCATION_RELATIVEnearSSRESTAURANTLOCATION_RELATIVELOCATIONSSRESTAURANTLOCATION_RELATIVESSRESTAURANTSSDUMMYSS SSSEFig. 3. Example of a parse tree and its vector state equivalent.Web ServiceRequesterWeb ServiceProviderMatchmaking AgentWeb ServiceProviderWeb ServiceRequesterInternetRestaurant DomainRestaurantDatabaseRestaurant OntologyRestaurant DomainRestaurantDatabaseRestaurant OntologyHand heldcomputerHand heldcomputerMultimodalInputMultimodalInputUser UserFig. 4. Semantic Web architecture for Semantic Restaurant Finder.INTERNALTIONAL IEEE CONFERENCE ON COMPUTER SCIENCES - RIVF’06 4sending a profile file. The information inthe profile is described using the ontology-based service description language OWL-S. The Web Service Requester connectsthe client Web Browser of the SemanticRestaurant Finder system with the restau-rant ontology. It creates a profile upon re-ceipt of the user’s request. Such a profile isthen sent as a request to the MatchmakingAgent to locate the appropriate Web ServiceProvider to answer the request.V. RESTAURANT ONTOLOGYGENERATIONRestaurant information can be extractedfrom the Web and stored in a database,known as the Restaurant Database. Then,the information on the Restaurant Databasewill be converted into ontology formalism.This is shown in Figure 5.A. Knowledge DiscoveryThe Knowledge Discovery process dis-covers hidden knowledge from the Restau-rant Database using a clustering technique.In this research, we have applied one of themost effective clustering techniques knownas Kohonen Self-Organizing Map (KSOM)Neural Network [22] for knowledge dis-covery. Figure 6 shows the KnowledgeDiscovery process.• Feature Selection - Information onrestaurants’ street addresses, regionallocations and cuisine types are ex-tracted as feature factors.• Preprocessing - Information on theextracted features (i.e. the restaurantaddresses, locations and cuisine types)is converted into representative binarystrings.• Transformation - The restaurantrecords are converted into restaurantvectors.• KSOM Clustering - This step clus-ters the restaurant vectors using theKSOM neural network. The trainingalgorithm for the KSOM network isgiven in [22]. After training, a setof clusters for restaurants (or restau-rant vectors) are generated, and theyare referred to as restaurant clusters.Information on restaurant clusters isthen stored in the Restaurant ClusterDatabase.B. Ontology GenerationThe Ontology Generation process cre-ates the Restaurant Ontology to representthe knowledge on restaurants, which can beobtained from the Restaurant Database andRestaurant Cluster Database. The OntologyGeneration process is shown in Figure 7.Figure 8 shows the generated RestaurantOntology.Class Mapping maps the schematicand discovered knowledge into ontology’sschematic and discovered classes.Schematic Relation Generation gener-ates relations between schematic classes.Composed Relation Generation gener-ates the relations between the schematicclasses and discovered classes.Instances Generation generates instancesfor the schematic and discovered classesbased on the data available on the Restau-rant Cluster Database and RestaurantDatabase.VI. PERFORMANCE ANALYSISTo evaluate the performance of the pro-posed approach, an experiment has beenconducted. In the experiment, a RestaurantDatabase is constructed. The database con-tains 2100 restaurants in Singapore, whichare located over 66 locations based onMRT (Mass Transit Railway) stations andprovides a variety of 76 cuisine types.As discussed earlier, KSOM neural net-work is adopted for the proposed sys-tem.For comparison purpose, a supervisedneural network, Learning Vector Quantiza-tion Version 3 (or LVQ3) and the k-nearestneighbor (kNN) technique, are also usedin the experiment for evaluation. There aretwo popular variations of the kNN tech-nique. The first variation, denoted as kNN1,is based on vector’s normalized Euclideandistance to find the matches closest to theinput string, while the second variation,denoted as kNN2, uses the fuzzy-trigramtechnique.Table I gives the retrieval accuracy ofdifferent techniques. The accuracy of thetwo neural networks KSOM and LVQ3are better than those of the kNN tech-niques. And LVQ3 has achieved better per-formance than KSOM. This is expected assupervised learning techniques often per-form better than an equivalent unsupervisedINTERNALTIONAL IEEE CONFERENCE ON COMPUTER SCIENCES - RIVF’06 5KnowledgeDiscoveryRestaurantDatabaseRestaurantClusterDatabaseOntologyGeneration RestaurantOntologyFig. 5. Restaurant ontology generation.RestaurantDatabaseFeatureSelectionPre-processing TransformationKSOMClusteringRestaurantClustersDatabaseFig. 6. Knowledge Discovery process.Ontology Classes withComposed RelationsSchematic RelationGenerationClass MappingComposed RelationGenerationOntology Classes Ontology Classes withSchematic RelationsInstancesGenerationRestaurantOntologyRestaurantDatabaseRestaurantClusterDatabaseDatabasesFig. 7. Ontology Generation process.RestaurantClusterRestaurantbelong-toAddresshasSchematic ClassesDiscovered ClassLocationlocated-atCuisineserveFig. 8. Restaurant Ontology.one when applied to the same domain.However, as LVQ3 always require humaninterpretation in training, it will encounterdifficulties when training a large dataset.Based on the clustering results fromKSOM, the Restaurant Ontology is thengenerated. The generated ontology can berepresented using an ontology descriptionlanguage that enables it to be sharable overthe Semantic Web by Web-based applica-tions.VII. CONCLUSIONIn this paper, we have proposed theSemantic Restaurant Finder system, inwhich restaurant information in differentcity/country/language are constructed asontologies for knowledge sharing. As such,users can make use of the semantic restau-rant knowledge distributed from differ-INTERNALTIONAL IEEE CONFERENCE ON COMPUTER SCIENCES - RIVF’06 6TABLE IPERFORMANCE ON RETRIEVAL ACCURACY.Retrieval Technique Retrieval AccuracykNN1 81.4%kNN1 77.6%LVQ3 93.2%KSOM 90.3%ent locations on the Internet to find thedesired restaurants. To achieve this, thispaper has presented the multimodal dia-logue support and the semantic web ar-chitecture for Semantic Restaurant Finder.In the proposed architecture, Web ServiceRequester, Matchmaking Agent and WebService Provider are constructed to allowusers to access the multiple RestaurantOntologies available on the Semantic Webenvironment. A framework for ontologygeneration using the KSOM neural networkhas also been discussed. Finally, an exper-iment has been conducted to evaluate theperformance of the proposed system.REFERENCES[1] R. A. Bolt, “Put-that-there: Voice and gesture atthe graphics interface,” in Proceedings of the 7thannual conference on Computer graphics andinteractive techniques, 1980, pp. 262–270.[2] A. Cheyer and L. Julia, “Multimodal maps: Anagent-based approach,” in International Con-ference on Cooperative Multimodal Communi-cation (CMC/95), Eindhoven, The Netherlands,May 1995.[3] M. Johnston, S. Bangalore, and G. Vasireddy,“Match: Multimodal access to city help,”Madonna di Campiglio, Italy, 2001.[4] M. Johnston, S. Bangalore, G. Vasireddy,A. Stent, P. Ehlen, M. Walker, S. Whittaker,and P. Maloor, “Match: An architecture formultimodal dialogue system,” in Proceedings ofACL02, 2002, pp. 376–383.[5] M. Johnston and S. Bangalore, “Matchkiosk: Amultimodal interactive city guide,” in Proceed-ings of ACL04, Barcelona, Spain, July 2004, pp.222–225.[6] D. Pele´, G. Breton, F. Panaget, and S. Loyson,“Let’s find a restaurant with Nestor. a 3Dembodied conversational agent on the web!”in AAMAS Workshop on Embodied Conversa-tional Characters as Individuals, Melbourne,Australia, July 2003.[7] D. Austin, A. Barbin, C. Ferris, and S. Garg,Web Services Architecture Requirements, 2002,http://www.w3c.org/TR/wsa-reqs.[8] S. Franklin and A. Graesser, “Is it an agent, orjust a program?: A taxonomy for autonomousagents,” in Proceedings of the Third Interna-tional Workshop on Agent Theories, Architec-tures, and Languages, 1996.[9] T. Berners-Lee, Weaving the Web: The OriginalDesign and Ultimate Destiny of the World WideWeb. HarperBusiness Publishers, 2000.[10] E. Christensen, F. Curbera, G. Meredith,and S. Weerawarana, Web servicesdescription language (WSDL) 1.1, 2001,http://www.w3.org/TR/wsdl.[11] The UDDI Technical White Paper, 2000,http://www.uddi.org.[12] W3C, “Web Ontology Language overview,”availaible at http://www.w3.org/TR/owl-features/.[13] A. Ankolekar, M. Burstein, J. Hobbs, O. Las-sila, D. Martin, S. McIlraith, S. Narayanan,M. Paolucci, T. Payne, K. Sycara, and H. Zeng,“Daml-s: Semantic markup for web services,” inProceedings of the International Semantic WebWorking Symposium, Stanford, CA, 2001.[14] D. Martin, A. Cheyer, and D. Moran, “The openagent architecture: A framework for buildingdistributed software systems,” Applied ArtificialIntelligence, vol. 13, no. 1-2, pp. 91–128, 1999.[15] A. Cheyer and D. Martin, “The open agent ar-chitecture,” Journal of Autonomous Agents andMulti-Agent Systems, vol. 4, no. 1, pp. 143–148,2001.[16] C. U. E. Department, The ATKReal-Time API for HTK, 2004,http://htk.eng.cam.ac.uk/develop/atk.shtml.[17] C. U. E. Department, The Hidden Markov ModelToolkit (HTK), 2004, http://htk.eng.cam.ac.uk.[18] Y. He and S. Young, “Hidden vector statemodel for hierarchical semantic parsing,” in Pro-ceedings of the IEEE International Conferenceon Acoustics, Speeach and Signal Processing,Hongkong, 2003.[19] Y. He and S. Young, “Semantic processing us-ing the hidden vector state model,” ComputerSpeech and Language, vol. 19, no. 1, pp. 85–106, 2005.[20] K. Boukreev, Mouse Gestures Recognition,2001, http://www.codeguru.com/.[21] T. U. o. E. The Centre for SpeechTechnology Research, The FestivalSpeech Synthesis System, 2004,http://www.cstr.ed.ac.uk/projects/festival/.[22] T. Kohonen, Self-Organizing Maps. Springer,1995.

A multimodal restaurant finder for semantic web

Abstract

Similar works

Full text

Available Versions

Open Research Online