How to Evaluate your Question Answering System Every Day and Still Get Real Work Done

In this paper, we report on Qaviar, an experimental automated evaluation system for question answering applications. The goal of our research was to find an automatically calculated measure that correlates well with human judges' assessment of answer correctness in the context of question answering tasks. Qaviar judges the response by computing recall against the stemmed content words in the human-generated answer key. It counts the answer correct if it exceeds agiven recall threshold. We determined that the answer correctness predicted by Qaviar agreed with the human 93% to 95% of the time. 41 question-answering systems were ranked by both Qaviar and human assessors, and these rankings correlated with a Kendall's Tau measure of 0.920, compared to a correlation of 0.956 between human assessors on the same data.Comment: 6 pages, 3 figures, to appear in Proceedings of the Second International Conference on Language Resources and Evaluation (LREC 2000

Intégration des données d'un lexique syntaxique dans un analyseur syntaxique probabiliste

ISBN : 978-2-7453-2512-9International audienceThis article reports the evaluation of the integration of data from a syntactic-semantic lexicon, the Lexicon-Grammar of French, into a syntactic parser. We show that by changing the set of labels for verbs and predicational nouns, we can improve the performance on French of a non-lexicalized probabilistic parser

Information Extraction using Context-free Grammatical Inference from Positive Examples

Information extraction from textual data has various applications, such as semantic search. Learning from positive example have theoretical limitations, for many useful applications (including natural languages), substantial part of practical structure (CFG) can be captured by framework introduced in this paper. Our approach to automate identification of structural information is based on grammatical inference. This paper mainly introduces the Context-free Grammar learning from positive examples. We aim to extract Information from unstructured and semi-structured document using Grammatical Inference. DOI: 10.17762/ijritcc2321-8169.15064

Problems with evaluation of unsupervised empirical grammatical inference systems

Abstract. Empirical grammatical inference systems are practical systems that learn structure from sequences, in contrast to theoretical grammatical inference systems, which prove learnability of certain classes of grammars. All current empirical grammatical inference evaluation methods are problematic, i.e. dependency on language experts, appropriateness and quality of an underlying grammar of the data, and influence of the parameters of the evaluation metrics. Here, we propose a modification of an evaluation method to reduce the ambiguity of results

Evaluación de analizadores de constituyentes y de dependencias

El presente trabajo muestra la evaluación cuantitativa y cualitativa de un grupo de analizadores de constituyentes y de dependencias con el objetivo de ser usados en el desarrollo de una métrica automática basada en conocimiento para evaluar la salida de sistemas de traducción automática. Primero se describe la metodología seguida en ambos tipos de evaluación y a continuación se muestran los resultados obtenidos y las conclusiones alcanzadas.This work presents the quantitative and qualitative evaluation of a set of both constituency and dependency parsers which are to be used in the development of a knowledge-based automatic MT metric. Firstly, the methodology used in both types of evaluation is described; secondly, we show the results obtained, and finally we draw some conclusions.This work has been funded by the Spanish Government project KNOW, TIN2009-14715-C0403

Statistical parsing of morphologically rich languages (SPMRL): what, how and whither

The term Morphologically Rich Languages (MRLs) refers to languages in which significant information concerning syntactic units and relations is expressed at word-level. There is ample evidence that the application of readily available statistical parsing models to such languages is susceptible to serious performance degradation. The first workshop on statistical parsing of MRLs hosts a variety of contributions which show that despite language-specific idiosyncrasies, the problems associated with parsing MRLs cut across languages and parsing frameworks. In this paper we review the current state-of-affairs with respect to parsing MRLs and point out central challenges. We synthesize the contributions of researchers working on parsing Arabic, Basque, French, German, Hebrew, Hindi and Korean to point out shared solutions across languages. The overarching analysis suggests itself as a source of directions for future investigations

Constituency and Dependency Parsers Evaluation

This work presents the quantitative and qualitative evaluation of a set of both constituency and dependency parsers which are to be used in the development of a knowledgebased automatic MT metric. Firstly, the methodology used in both types of evaluation is described; secondly, we show the results obtained, and finally we draw some conclusions