Towards Understanding Spontaneous Speech: Word Accuracy vs. Concept Accuracy

In this paper we describe an approach to automatic evaluation of both the speech recognition and understanding capabilities of a spoken dialogue system for train time table information. We use word accuracy for recognition and concept accuracy for understanding performance judgement. Both measures are calculated by comparing these modules' output with a correct reference answer. We report evaluation results for a spontaneous speech corpus with about 10000 utterances. We observed a nearly linear relationship between word accuracy and concept accuracy.Comment: 4 pages PS, Latex2e source importing 2 eps figures, uses icslp.cls, caption.sty, psfig.sty; to appear in the Proceedings of the Fourth International Conference on Spoken Language Processing (ICSLP 96

Natural language processing

Beginning with the basic issues of NLP, this chapter aims to chart the major research activities in this area since the last ARIST Chapter in 1996 (Haas, 1996), including: (i) natural language text processing systems - text summarization, information extraction, information retrieval, etc., including domain-specific applications; (ii) natural language interfaces; (iii) NLP in the context of www and digital libraries ; and (iv) evaluation of NLP systems

Design and enhanced evaluation of a robust anaphor resolution algorithm

Syntactic coindexing restrictions are by now known to be of central importance to practical anaphor resolution approaches. Since, in particular due to structural ambiguity, the assumption of the availability of a unique syntactic reading proves to be unrealistic, robust anaphor resolution relies on techniques to overcome this deficiency. This paper describes the ROSANA approach, which generalizes the verification of coindexing restrictions in order to make it applicable to the deficient syntactic descriptions that are provided by a robust state-of-the-art parser. By a formal evaluation on two corpora that differ with respect to text genre and domain, it is shown that ROSANA achieves high-quality robust coreference resolution. Moreover, by an in-depth analysis, it is proven that the robust implementation of syntactic disjoint reference is nearly optimal. The study reveals that, compared with approaches that rely on shallow preprocessing, the largely nonheuristic disjoint reference algorithmization opens up the possibility/or a slight improvement. Furthermore, it is shown that more significant gains are to be expected elsewhere, particularly from a text-genre-specific choice of preference strategies. The performance study of the ROSANA system crucially rests on an enhanced evaluation methodology for coreference resolution systems, the development of which constitutes the second major contribution o/the paper. As a supplement to the model-theoretic scoring scheme that was developed for the Message Understanding Conference (MUC) evaluations, additional evaluation measures are defined that, on one hand, support the developer of anaphor resolution systems, and, on the other hand, shed light on application aspects of pronoun interpretation

Researching grammar learning strategies: Combining the macro- and micro-perspective

Udostępnienie publikacji Wydawnictwa Uniwersytetu Łódzkiego finansowane w ramach projektu „Doskonałość naukowa kluczem do doskonałości kształcenia”. Projekt realizowany jest ze środków Europejskiego Funduszu Społecznego w ramach Programu Operacyjnego Wiedza Edukacja Rozwój; nr umowy: POWER.03.05.00-00-Z092/17-00

Measuring and understanding patterns of change in intervention studies with children: implications for evidence-based practice

Purpose: Comparisons across studies of the effects of intervention are problematic. Such analyses raise both methodological and statistical challenges. A single data set was examined to investigate whether different established approaches to measuring change in children with specific language impairments alter the conclusions that can be drawn regarding the efficacy of an intervention. Methods: Measures of cognitive and language skills were collected at baseline and at six months following an intervention. Reliable and valid psychometric measures were used. Data from the intervention study were used to explore the patterns of results obtained using four different measures of change: change of diagnostic category, differential improvement across assessment measures, item specific changes and predictors of individual change. Results: Associations between different tests purporting to measure similar constructs were modest. The measures identified different children as impaired both at baseline and follow-up. No effect of intervention was evident when a categorical analysis of impairment was used. Both treatment and comparison children changed significantly across time on the majority of measures, providing evidence of development, but specific effects of the intensive intervention were evident using ANCOVAs. Item analysis indicated that one of the standardized language tests adopted in the evaluation was insensitive to change over a six month period. Change in individual children's performance was predicted by language level on entry to the project. Conclusion: The implications of the results are discussed in terms of the range of analytic approaches available to intervention researchers and the need to consider combinations of methods when analysing outcome data. †We would like to thank ICAN, the health trusts involved and the two research officers, Kerry Williams and Belinda Seeff, who collected the data. © 2007 Taylor & Francis Group, LLC