The research in this paper makes explicit why existing measures for response quality evaluation is not suitable for the ever-evolving field of question answering and following that, a short-term solution for evaluating response quality of heterogeneous systems is put forward. To demonstrate the challenges in evaluating systems of different nature, this research presents a black-box approach using a classification scheme and scoring mechanism to assess and rank three example systems

Goh, O.S.

Sahib, S.

Wong, W.

English

Research Repository

   MURDOCH RESEARCH REPOSITORY   http://dx.doi.org/10.1109/WI.2005.61       Wong, W., Sahib, S. and Goh, O.S. (2005) Evaluation of response quality for heterogeneous question answering systems. In: IEEE/WIC/ACM International Conference on Web Intelligence, WI 2005, 19 - 22 September, Compiegne Cedex, France, pp. 610-613.     http://researchrepository.murdoch.edu.au/14622/          Copyright © 2005 IEEE  Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.   1Evaluation of Response Quality for Heterogeneous Question Answering SystemsWilson Wong & Shahrin SahibFaculty of Information and Communication Technology, Kolej Universiti Teknikal KebangsaanMalaysia, Melaka, Malaysia{wilson, shahrinsahib}@kutkm.edu.myOng-Sing GohSchool of Information Technology, MurdochUniversity, Perth, Western Australia, 6150osgoh88@gmail.comAbstractThe research in this paper makes explicit why existing measures for response quality evaluation is not suitablefor the ever-evolving field of question answering andfollowing that, a short-term solution for evaluatingresponse quality of heterogeneous systems is put forward. To demonstrate the challenges in evaluating systems ofdifferent nature, this research presents a black-boxapproach using a classification scheme and scoringmechanism to assess and rank three example systems.1. IntroductionGenerally, question answering systems can be catego-rized into two groups based on the approach in each di-mension. The first is question answering based on shallow natural language processing and information retrieval andthe second approach is question answering based on natu-ral language understanding and reasoning. Table I summa-rizes the characteristics of the two approaches with re-spects to the dimensions in question answering. Some ofthe well known systems from the first approach are likeWebclopedia [1] and AnswerBus [2], while examples ofquestion answering systems from the second are likeWEBCOOP [3] in tourism, NaLURI [4] in Cyberlaw andSTART [5].Table I. Characteristics of the two approaches in ques-tion answeringDimension Shallow natural language proc-essing and information retrievalNatural language understanding and reasoningTechnique Syntax processing and informationretrievalSemantic analysis or higher, and reasoningSource Free-text documents Knowledge baseDomain Open-domain Domain-orientedResponse Extracted snippets Synthesized responsesQuestion Questions using wh-words Beyond wh-wordsEvaluation Information retrieval metrics N/AThe evaluation of question answering systems for non-dynamic responses has been largely reliant on the TRECcorpus. It is easy to evaluate systems in which there is aclearly defined answer, however, for most natural lan-guage questions there is no single correct answer [6].Evaluation can turn into a very subjective matter espe-cially when dealing with different types of natural lan-guage systems in different domains due several reasons:no baseline or comparable systems in certain domains,developing test questions is not easy, and dynamic natureof the responses, there is no right or wrong answer asthere are always responses to justify the absence of ananswer.2. Existing Metrics for Question AnsweringThe most notable evaluation for question answeringhas to be the question answering track in the TRECevaluation [7]. Evaluation in TREC assesses the quality of response in terms of precision and recall, and is well-suited for question answering systems based on shallownatural language processing and information retrieval likeAnswerBus. There are several inherent requirements thatmake such evaluation inappropriate for domain-orientedquestion answering systems based on understanding andreasoning: assessments should average over large corpusor query collection, assessments have to be binary whereanswers can only be classified as correct and incorrect and assessments would be heavily skewed by corpus, makingthe results not translatable from one domain to another.There are also other measures but are mostly designedfor general tasks related to natural language processinglike translation, database query, etc. [8] proposes that asimple number scale be established for the evaluation ofnatural language text processing systems. This metric is tobe based on the simple average of four things: size of thelexicon, the speed and accuracy of the parse and the over-all experience of the system. The author oversimplifiedmatters by equating the ability of understanding to meresentence parsing and as the computing strength increasesin terms of hardware and software, the factor of speed and accuracy can no longer be discriminative enough to sepa-rate one system from another. Proceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05) 0-7695-2415-X/05 $20.00 © 2005 IEEE2AnswerBusresponsesSTARTresponsesNaLURIresponsesO2_N responses that suggest possible spelling mistakeO1_Ninformative responses in the event of no answerBQ_Nresponses that provide accurate and direct answers to questionsLQ_A responses with completely irrele-vant informationBQ_A responses with some elements that meet the questions’ requirementsO1_A uninformative responses inthe event of no answersBQ_Sresponses that provide accurate and direct answers to questionsO2_Sresponses that suggest possible spelling mistakeO1_S uninformative responses in the event of no answersUnlike the previous, general model is provided by [9]that acts as a basis of a quantitative measure for evaluating how well a system can understand natural language. Buthow well a system can understand natural language onlyprovides for half of the actual ability required to generatehigh-quality responses. Hence, such general model isinadequate for more specific application of natural lan-guage understanding like question answering.[10] and [12] have also suggested a type of black-boxevaluation where we evaluate a system to see how good it is at producing the quality or desirable answers. [13] fur-ther characterize the black-box evaluation and suggestedthat systems can be evaluated on their answer providingability that includes measures for answer completeness,accuracy and relevancy. The authors also state thatevaluation measures should include more fine grainedscoring procedures to cater answers to different types ofquestion. The authors give examples of answers that areexplanations or summaries or biographies or comparativeevaluations cannot be meaningfully rated as simply rightor wrong. We consider this black-box approach as com-prehensive in assessing how well question answeringsystems produce responses required by users and howcapable are these systems in handling various types ofsituations and questions. Despite the merits of this evalua-tion approach, none of the authors provide further detailson the formal measures used for scoring and ranking thesystems under evaluation.3. Black-box Approach for EvaluationIn this paper, we present an innovative measure forevaluating response quality: a black-box approach through observation and classification with a scoring mechanism.This black-box approach is based on the work of [10],[12] and [13] as discussed in previous sections for evalu-ating response quality. We further refine this approach byproposing a response classification scheme and a scoringmechanism. To demonstrate this approach, we have se-lected three question answering systems that representdifferent level of response generation complexity namelyAnswerBus, START and NaLURI.To begin with, this black-box approach requires a setof questions that can sufficiently examines the responsegeneration strength of all systems under evaluation. Forthis purpose, we prepare 45 questions of various natureson the Cyberlaw domain. These questions will be used toprobe the systems and the actual responses are gatheredfor later use. Details of the questions and responses for thethree systems are available in [4].For this approach, we propose a classification schemethat consists of categories to encompass all possible typesof response from all systems under evaluation. Thisscheme consists of three category codes as shown in Table II and was designed based on the quality of responses asperceived by general users and is not tied down to anyimplementation detail of any systems. This makes thescheme generally applicable to all evaluations of questionanswering systems with different approaches.Table II: Categories in black-box approachcategory notation desc. purposeBQ_tgeneralLQ_tt is systemsinitialrepresents best and lowest qualityresponse for each systemdynamic Oj_tj is an integerrepresents other evaluation-specificcriteriaDynamic category allows evaluators to create as manynew categories as required by the types of systems underevaluation. The Oj_t category not only makes this scheme expandable but also dynamic because as technology pro-gresses, the response generation capability of systems may increase and in such cases, evaluators can define evalua-tion-specific categories. For this evaluation, we defineO1_t for quality of response in the event of no answer and O2_t for response that suggest possible spelling mistake.In this evaluation, the initials for AnswerBus, START and NaLURI are A, S and N respectively. Figure 1: Grouping of responses into categoriesNext, using these codes, the evaluators will try to ob-serve and classify each response into one of the catego-ries. The classification is done based on the manual ob-servation by evaluators who are guided by the criteria ofeach category. For example, if the evaluator comes acrossa response that is generated by system a and the responseappears to be an uninformative attempt to notify the userthat no valid answer can be found, then we can classifythat response as O1_a. This is to say that system a gener-ates uninformative response in the event of no answer.From the nature of the responses generated by the threesystems, we can group them into relevant categoriesProceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05) 0-7695-2415-X/05 $20.00 © 2005 IEEE3shown in Figure 1. After classification of the responses isdone, a scoring mechanism is used to determine responses from which system are of the best overall quality. A pair-wise relative comparison is performed and points areassigned based on superiority of responses of the samecategory. If there are n systems under evaluation, thenthere should be nC2 = k pairs. Let qi represents the pair ofsystem txi and tyi. To perform the scoring, a table is con-structed as shown in Table III where the column headerrepresents all the q1, q2,…, qk pairs. The row header willconsists of the two general categories BQ_t and LQ_t and other evaluation-specific categories Oj_t.Table III. Template for scoring mechanismq1 q2 … qkCategorytx1 ty1 tx2 ty2 txk ty1kBQ_tLQ_tOj_tTotalThen for every qi, we compare BQ_txi with BQ_tyi,LQ_txi with LQ_tyi and other Oj_txi with Oj_tyi. The rulesfor superiority comparison and assigning of score are asfollows:• if the description of the responses for txi is better than tyi under a particular category, then txi is assigned with 1 and tyi is assigned with 0 under the same category;• if the description of the responses for txi is inferiorcompared to tyi under a particular category, then txi isassigned with 0 and tyi is assigned with 1 under thesame category; and• if the description of the responses for txi is the sameas tyi under a particular category, then both txi and tyiare assigned with 0 under the same category.After filling up all the cells in the score table, summa-tion of scores for every txi and tyi under all categories isperformed. Here are a few examples to demonstrate theworking behind the scoring mechanism. The best qualityresponses of AnswerBus, BQ_A have the possibility ofcontaining irrelevant elements, whereas responses gener-ated by START are always correct and directly answer the questions. Due to this, the best quality responses fromSTART, which belongs to BQ_S, are a level higher thanthe best quality responses of AnswerBus, BQ_A. Hence,for the pair “START vs. AnswerBus”, START will beassigned with one point. In the case of ties, like othercategories O_1S and O_1A which demonstrate the samequality of responses in the event of no answers, no pointswill be given for either side of the pair “START vs. An-swerBus”. Consider another example where the responsesfrom O_2S, which attempt to alert the users of possiblespelling mistake, make START an additional level higherthan AnswerBus. This provides START with anotheradditional point in the pair “START vs. AnswerBus”. Thecomparison will be done on all the three systems, givingus three possible pairs. From Table IV, we can observethat AnswerBus has the total score of 0 + 0 = 0, NaLURIwith 3 + 1 = 4 and START with 0 + 2 = 2.Table IV. Scoring table for quality evaluation usingpair-wise relative comparisonAnswerBus vs. NaLURISTART vs. NaLURISTART vs. An-swerBusCategoryAnswer-BusNaLURISTARTNaLURISTARTAnswer-BusBQ_LQ_0 1 0 0 1 0O_1 0 1 0 1 0 0O_2 0 1 0 0 1 0Total 0 3 0 1 2 04. Implications and Validity of ResultsFrom the total scores of the three systems, NaLURIranked first with 4 points, followed by START with 2points and lastly, AnswerBus with 0 points. This makesthe quality of responses generated by NaLURI relativelybetter compare with START and AnswerBus. The condi-tion is to assume that the evaluators’ observations andclassifications are consistent throughout, and the set ofquestions used for evaluation is exhaustive enough totrigger all possible responses.Table V. Relation between quality of responses and components in question answeringCategories of responsesAnswerBus START NaLURIresponses with halfcorrect answers andthe rest are irrele-vant.through sentenceparsing and informationretrievaln/a n/aresponses that provide accu-rate/direct answersn/athrough natural language under-standing and reasoningthrough natural language under-standing and reasoningquality of re-sponses in the event of no answersuninformativedue to the lack of advanced reason-inguninformativedue to the lack of advanced reason-inginformative due to the use of ad-vanced reasoningresponses that suggest possiblespelling mistaken/athrough addi-tional linguisticfeaturethrough additionallinguistic featureThe approach of evaluating the response qualitythrough observation, classification and a scoring mecha-nism has revealed to us that the lack or addition of com-ponents has great impact on the response quality. Forinstance, one of the criteria that have contributed to theProceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05) 0-7695-2415-X/05 $20.00 © 2005 IEEE4higher score of NaLURI is the capability of the system ingenerating dynamic responses to suit the various anoma-lous situations. Useful responses can be dynamically gen-erated by NaLURI to cater the condition when no answers are available. This ability can be attributed to the inclu-sion of the two advanced reasoning components namelyexplanation on failure and dynamic answer generation.Table V neatly shows how each of the categories of re-sponses are achieved through the different approach to-wards question answering that implements diverse com-ponents in information retrieval, natural language under-standing and reasoning.After having concluded that NaLURI is comparativelybetter than the other two systems, skeptical thoughts mayarise. Firstly, thoughts may arise concerning to the domain of the question. People may question that the evaluation is inclined towards NaLURI because the question set isprepared in the same domain as NaLURI, which is Cyber-law. But, what is the domain of START and AnswerBus?“AnswerBus is an open-domain question answering…”[2] while START is capable of handling many domainsbased on the statement “our system answers millions ofnatural language questions about places, political andeconomic systems, people, dictionary definitions, andmuch more…” by [5]. Hence, the authors do not see anyproblem in START and AnswerBus handling Cyberlawquestions.Secondly, thoughts may arise concerning to the natureof the question. People may question that the evaluation is inequitable towards START and AnswerBus because thenature of the questions used to evaluate vary greatly andcover beyond wh-questions. But, we would like the read-ers to recall that the aim of this evaluation is to assess and rank systems of any approach based on the quality ofresponses generated. How can we rank these systems if we merely use wh-questions, knowing that given the presentstate of question answering technology, handling wh-questions is no more a challenge? Hence, benchmark forquestion answering systems has to progress with time byconsidering various state-of-the-art factors instead ofdwelling in the past.5. ConclusionsIn this paper, we have highlighted the rise of the needfor standard metrics to assess and measure the quality ofresponses produced by systems of different approachesand domain. Following this, we have presented a short-term solution for the evaluation of the quality of responsesin the form of a black-box approach through classificationand a scoring mechanism using pair-wise relative com-parison. To demonstrate the approach, we have also pre-sented the data and results obtained through an evaluation performed on three very different systems.References[1] Hermjakob, U., Parsing and Question Classificationfor Question Answering. In Proceedings of the ACLWorkshop on Open-Domain Question Answering, 2001.[2] Zheng, Z., AnswerBus Question Answering System. In Proceedings of the Conference on Human LanguageTechnology, 2002.[3] Benamara, F. & Saint-Dizier, P., Advanced Relaxationfor Cooperative Question Answering. In New Directionsin Question Answering. MIT Press, 2004.[4] Wong, W., Practical Approach to Knowledge-basedQuestion Answering with Natural LanguageUnderstanding and Advanced Reasoning. Thesis (MSc),Kolej Universiti Teknikal Kebangsaan Malaysia, 2005.[5] Katz, B., Felshin, S. & Lin, J., The START Multime-dia Information System: Current Technology and FutureDirections. In Proceedings of the International Workshop on Multimedia Information Systems, 2002.[6] Moldovan, D., Pasca, M., Surdeanu, M. & Harabagiu,S., Performance Issues and Error Analysis in an Open-Domain Question Answering System. In Proceedings ofthe 40th Annual Meeting of the Association for Computa-tional Linguistics, 2002.[7] Voorhees, E., Overview of TREC 2003. In Proceedings ofthe 12th Text Retrieval Conference, 2003.[8] Facemire, J., A Proposed Metric for the Evaluation ofNatural Language Systems. In Proceedings of the IEEEEnergy and Information Technologies in the Southeast,1989.[9] Guida, G. & Mauri, G., A Formal Basis for Perform-ance Evaluation of Natural Language Understanding Sys-tems. Computational Linguistics, 10(1):15-30, 1984.[10] Srivastava, A. & Rajaraman, V., A Vector Measurefor the Intelligence of a Question-Answering (Q-A) Sys-tem. IEEE Transactions on Systems_Man and Cybernet-ics, 25(5):814-823, 1995.[11] Allen, J., Natural Language Understanding. Benja-min/Cummins Publishing, 1995.[12] Nyberg, E. & Mitamura, T., Evaluating QA Systemson Multiple Dimensions. In Proceedings of the Workshopon QA Strategy and Resources, 2002.[13] Diekema, A., Yilmazel, O. & Liddy, E., Evaluation of Restricted Domain Question-Answering Systems. In Pro-ceedings of the ACL Workshop on Question Answering in Restricted Domains, 2004.Proceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05) 0-7695-2415-X/05 $20.00 © 2005 IEEE

Evaluation of response quality for heterogeneous question answering systems

Goh, Ong Sing

Universiti Teknikal Malaysia Melaka (UTeM) Repository

1Evaluation of Response Quality for Heterogeneous Question Answering SystemsWilson Wong & Shahrin SahibFaculty of Information and Communication Technology, Kolej Universiti Teknikal KebangsaanMalaysia, Melaka, Malaysia{wilson, shahrinsahib}@kutkm.edu.myOng-Sing GohSchool of Information Technology, MurdochUniversity, Perth, Western Australia, 6150osgoh88@gmail.comAbstractThe research in this paper makes explicit why existing measures for response quality evaluation is not suitablefor the ever-evolving field of question answering andfollowing that, a short-term solution for evaluatingresponse quality of heterogeneous systems is put forward. To demonstrate the challenges in evaluating systems ofdifferent nature, this research presents a black-boxapproach using a classification scheme and scoringmechanism to assess and rank three example systems.1. IntroductionGenerally, question answering systems can be catego-rized into two groups based on the approach in each di-mension. The first is question answering based on shallow natural language processing and information retrieval andthe second approach is question answering based on natu-ral language understanding and reasoning. Table I summa-rizes the characteristics of the two approaches with re-spects to the dimensions in question answering. Some ofthe well known systems from the first approach are likeWebclopedia [1] and AnswerBus [2], while examples ofquestion answering systems from the second are likeWEBCOOP [3] in tourism, NaLURI [4] in Cyberlaw andSTART [5].Table I. Characteristics of the two approaches in ques-tion answeringDimension Shallow natural language proc-essing and information retrievalNatural language understanding and reasoningTechnique Syntax processing and informationretrievalSemantic analysis or higher, and reasoningSource Free-text documents Knowledge baseDomain Open-domain Domain-orientedResponse Extracted snippets Synthesized responsesQuestion Questions using wh-words Beyond wh-wordsEvaluation Information retrieval metrics N/AThe evaluation of question answering systems for non-dynamic responses has been largely reliant on the TRECcorpus. It is easy to evaluate systems in which there is aclearly defined answer, however, for most natural lan-guage questions there is no single correct answer [6].Evaluation can turn into a very subjective matter espe-cially when dealing with different types of natural lan-guage systems in different domains due several reasons:no baseline or comparable systems in certain domains,developing test questions is not easy, and dynamic natureof the responses, there is no right or wrong answer asthere are always responses to justify the absence of ananswer.2. Existing Metrics for Question AnsweringThe most notable evaluation for question answeringhas to be the question answering track in the TRECevaluation [7]. Evaluation in TREC assesses the quality of response in terms of precision and recall, and is well-suited for question answering systems based on shallownatural language processing and information retrieval likeAnswerBus. There are several inherent requirements thatmake such evaluation inappropriate for domain-orientedquestion answering systems based on understanding andreasoning: assessments should average over large corpusor query collection, assessments have to be binary whereanswers can only be classified as correct and incorrect and assessments would be heavily skewed by corpus, makingthe results not translatable from one domain to another.There are also other measures but are mostly designedfor general tasks related to natural language processinglike translation, database query, etc. [8] proposes that asimple number scale be established for the evaluation ofnatural language text processing systems. This metric is tobe based on the simple average of four things: size of thelexicon, the speed and accuracy of the parse and the over-all experience of the system. The author oversimplifiedmatters by equating the ability of understanding to meresentence parsing and as the computing strength increasesin terms of hardware and software, the factor of speed and accuracy can no longer be discriminative enough to sepa-rate one system from another. Proceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05) 0-7695-2415-X/05 $20.00 © 2005 IEEE2AnswerBusresponsesSTARTresponsesNaLURIresponsesO2_N responses that suggest possible spelling mistakeO1_Ninformative responses in the event of no answerBQ_Nresponses that provide accurate and direct answers to questionsLQ_A responses with completely irrele-vant informationBQ_A responses with some elements that meet the questions’ requirementsO1_A uninformative responses inthe event of no answersBQ_Sresponses that provide accurate and direct answers to questionsO2_Sresponses that suggest possible spelling mistakeO1_S uninformative responses in the event of no answersUnlike the previous, general model is provided by [9]that acts as a basis of a quantitative measure for evaluating how well a system can understand natural language. Buthow well a system can understand natural language onlyprovides for half of the actual ability required to generatehigh-quality responses. Hence, such general model isinadequate for more specific application of natural lan-guage understanding like question answering.[10] and [12] have also suggested a type of black-boxevaluation where we evaluate a system to see how good it is at producing the quality or desirable answers. [13] fur-ther characterize the black-box evaluation and suggestedthat systems can be evaluated on their answer providingability that includes measures for answer completeness,accuracy and relevancy. The authors also state thatevaluation measures should include more fine grainedscoring procedures to cater answers to different types ofquestion. The authors give examples of answers that areexplanations or summaries or biographies or comparativeevaluations cannot be meaningfully rated as simply rightor wrong. We consider this black-box approach as com-prehensive in assessing how well question answeringsystems produce responses required by users and howcapable are these systems in handling various types ofsituations and questions. Despite the merits of this evalua-tion approach, none of the authors provide further detailson the formal measures used for scoring and ranking thesystems under evaluation.3. Black-box Approach for EvaluationIn this paper, we present an innovative measure forevaluating response quality: a black-box approach through observation and classification with a scoring mechanism.This black-box approach is based on the work of [10],[12] and [13] as discussed in previous sections for evalu-ating response quality. We further refine this approach byproposing a response classification scheme and a scoringmechanism. To demonstrate this approach, we have se-lected three question answering systems that representdifferent level of response generation complexity namelyAnswerBus, START and NaLURI.To begin with, this black-box approach requires a setof questions that can sufficiently examines the responsegeneration strength of all systems under evaluation. Forthis purpose, we prepare 45 questions of various natureson the Cyberlaw domain. These questions will be used toprobe the systems and the actual responses are gatheredfor later use. Details of the questions and responses for thethree systems are available in [4].For this approach, we propose a classification schemethat consists of categories to encompass all possible typesof response from all systems under evaluation. Thisscheme consists of three category codes as shown in Table II and was designed based on the quality of responses asperceived by general users and is not tied down to anyimplementation detail of any systems. This makes thescheme generally applicable to all evaluations of questionanswering systems with different approaches.Table II: Categories in black-box approachcategory notation desc. purposeBQ_tgeneralLQ_tt is systemsinitialrepresents best and lowest qualityresponse for each systemdynamic Oj_tj is an integerrepresents other evaluation-specificcriteriaDynamic category allows evaluators to create as manynew categories as required by the types of systems underevaluation. The Oj_t category not only makes this scheme expandable but also dynamic because as technology pro-gresses, the response generation capability of systems may increase and in such cases, evaluators can define evalua-tion-specific categories. For this evaluation, we defineO1_t for quality of response in the event of no answer and O2_t for response that suggest possible spelling mistake.In this evaluation, the initials for AnswerBus, START and NaLURI are A, S and N respectively. Figure 1: Grouping of responses into categoriesNext, using these codes, the evaluators will try to ob-serve and classify each response into one of the catego-ries. The classification is done based on the manual ob-servation by evaluators who are guided by the criteria ofeach category. For example, if the evaluator comes acrossa response that is generated by system a and the responseappears to be an uninformative attempt to notify the userthat no valid answer can be found, then we can classifythat response as O1_a. This is to say that system a gener-ates uninformative response in the event of no answer.From the nature of the responses generated by the threesystems, we can group them into relevant categoriesProceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05) 0-7695-2415-X/05 $20.00 © 2005 IEEE

http://eprints.utem.edu.my/id/eprint/12398/1/Evaluation_of_response_quality_for_heterogeneous_question_answering_systems_page_1-2.pdf

Evaluation of response quality for heterogeneous question answering systems

Abstract

Similar works

Full text

Available Versions

Research Repository

Universiti Teknikal Malaysia Melaka (UTeM) Repository