Evaluating automatic F-structure annotation for the Penn-II treebank

Abstract

Methodologies have been developed (van Genabith et al., 1999a,b; Sadler et al., 2000; Frank, 2000; van Genabith et al., 2001; Frank et al., 2002) for automatically annotating treebank resources with Lexical-Functional Grammar (LFG: Kaplan and Bresnan, 1982) fstructure information. Until recently, however, most of this work on automatic annotation has been applied only to limited datasets, so while it may have shown 'proof of concept', it has not been demonstrated that the techniques developed scale up to much larger data sets (Liakata and Pulman, 2002). More recent work (Cahill et al., 2002a,b) has presented efforts in evolving and scaling techniques established in these previous papers to the full Penn-ll Treebank (Marcus et al., 1994). In this paper, we present and assess a number of quantitative and qualitative evaluation methodologies which provide insights into the effectiveness of the techniques developed to derive automatically a set of f-structures for the more than 1,000,000 words and 49,000 sentences of Penn-II