We describe a baseline system for the VideoCLEF Vid2RSS task. The system uses an unaltered off-the-shelf Information Retrieval system. ASR content is indexed using

default stemming and stopping methods. The subject categories are populated by using the category label as a query on the collection, and assigning the retrieved items

to that particular category. We describe the results of the system and provide some high-level analysis of its performance

Jones, Gareth J.F.

Newman, Eamonn

English

DCU Online Research Access Service

DCU at VideoClef 2008Eamonn Newman and Gareth J.F. JonesCentre for Digital Video Processing, Dublin City University, Dublin 9, Ireland{enewman|gjones}@computing.dcu.iehttp://www.cdvp.dcu.ieAbstract. We describe a baseline system for the VideoCLEF Vid2RSStask in which videos are to be classified into thematic categories based ontheir content. The system uses an off-the-shelf Information Retrieval sys-tem. Speech transcripts generated using automated speech recognitionare indexed using default stemming and stopping methods. The cate-gories are populated by using the category theme (or label) as a queryon the collection, and assigning the retrieved items to that particularcategory. Run 4 of our system achieved the highest f-score in the task bymaximising recall. We discuss this in terms of the primary aims of thetask, i.e., automating video classification.Key words: Classification, Information Retrieval, Automatic SpeechRecognition1 IntroductionThe VideoClef Vid2RSS task required users to classify videos into one (or more)of a set of categories. Audio content consists primarily of Dutch with some em-bedded English content. The data provided consists of automatic speech recogni-tion (ASR) transcripts (generated independently using Dutch and English ASRsystems), shot boundary keyframes, and catalogue metadata (in Dutch). Eachcategory is then published as an RSS feed. The system described in this paperis based on an Information Retrieval approach. We built a standard free textindex using the ASR transcripts and associated metadata as the content.2 System DescriptionWe used the open source Lucene Search Engine technology [1] as the basetechnology for our system. Dutch-language content was stopped, stemmed andtokenised using Lucene’s built-in Dutch analyser, DutchAnalyzer1. English-language content was stopped and tokenised by the Lucene default tokeniser,StandardAnalyzer2. The StandardAnalyzer does not perform any stemmingof tokens.1 org.apache.lucene.analysis.nl.DutchAnalyzer2 org.apache.lucene.analysis.standard.StandardAnalyzer2.1 Run ConfigurationsFive separate runs were prepared and submitted to the task. The runs varied interms of both system configuration and the data which was used.1. Dutch ASR transcripts: In this run, we indexed the entire set of DutchASR transcripts (the FreeTextAnnotation elements). The index was queriedwith the labels in the order given in Table 1 and each item was classifiedinto a single category.2. English ASR transcripts: This is identical to Run 1, using English ASRtranscripts and translations of the category labels as queries.3. Dutch ASR with query expansion: We ran the same queries as Run 1,but added an additional step of query expansion in order to improve the recallof certain categories (some categories in earlier runs had nothing assignedto them). Because of this, items could be assigned to multiple categories.Queries were expanded by performing an initial query which consisted ofjust the category label. We take the first 10 retrieved documents and extractthe 5 most frequently occuring terms in each. We process this set of 50terms to remove any duplicates. The remaining terms are combined withthe original query to form the expanded query.4. English ASR with query expansion: This is identical in method toRun 3, but the data now consists of the English ASR transcripts, ratherthan the Dutch.5. Dutch metadata:We indexed the catalogue metadata which were suppliedin the data sets. Specifically, we used the description elements from themetadata documents. Once again, the Dutch category term labels were usedas queries, and the items were restricted to appear in one feed only.2.2 Category orderThe categories were ordered from most specific to least specific, as in Table 1. Foreach category, a query was made to the IR system using the category name as thequery keyword. All retrieved items were labelled as belonging to that category.The ordering meant that when an item was retrieved, it was placed into themost specific category possible. For our submitted runs, in Runs 1, 3, and 5, aretrieved item was placed only into the first category for which it was retrieved.In Runs 2 and 4, it was placed in all categories for which it was retrieved. Thisrestriction was imposed to improve the precision of the classification task, sincelabels such as “film” were very general and tended to capture most, if not all, ofthe items.3 ResultsIn Table 2 we present the retrieval scores attained by our system runs. Themetrics are defined in Section 2.2 of the Track Overview paper [2]. A directcomparison of Runs 1 and 2 suggests that the Dutch transcripts were moreDutch Englisharcheologie archeologyarchitectuur architecturechemie chemistrydansen danceschilderijen paintingswetenschappelijk onderzoek scientific researchbeeldende kunst visual artsgeschiedenis historyfilm filmmuziek musicTable 1. Category Label Ordermetric Run 1 Run 2 Run 3 Run 4 Run 5micro-average precision 0.50 0.32 0.16 0.17 0.83micro-average recall 0.35 0.21 0.91 0.72 0.18f-score micro-average 0.41 0.25 0.28 0.28 0.29macro-average precision 0.54 0.62 0.42 0.50 0.93macro-average recall 0.55 0.38 0.90 0.70 0.28f-score macro-average 0.54 0.47 0.58 0.59 0.43Table 2. Vid2RSS Scores for Runs 1 to 5useful in identifying the subject categories than the English ones. Indeed, theEnglish transcripts had the poorest f-scores at both micro and macro level. Thisis most likely attributable to the fact that the majority of the dialogue was inDutch and so contained less “noise” than the English counterparts. Processing ofthe ASR transcripts to identify the points at which the language changed wouldallow for the combination of transcripts (or the removal of erroneous segments)which would improve classification performance.Runs 3 and 4 used query expansion to add new keywords to the queries andallowed items to be placed in multiple categories. As we can see, this relaxationresulted in a large drop in the micro-average precision scores of these systems;conversely, the micro-average recall is much higher in these runs. As items wereplaced in multiple categories the chances of an item being correctly classifiedwere much greater, however the number of false positives also increased.As can be seen from the results, Run 5 performed particularly well in termsof precision, and relatively well (when compared to our other runs) in terms ofrecall. However, since this was on the metadata and not on the ASR transcripts,it cannot be directly compared to the others. The higher precision scores dosuggest that there may be merit in combining the different data sets available.One drawback that is immediately obvious with this system is that it is notpossible to guarantee that all items will be classified. If an item is not retrievedfor any of the queries, then it will not be placed in any of the category feeds. Asit happens, this was not the case for any of the runs with this particular dataset (probably due to the presence of highly generic labels such as “film” and“music”)Additionally, the number of terms added in the query expansion phase couldbe reduced. The maximum for this was 50, but elimination of duplicates meantthat the size of the set was generally much smaller. Nevertheless, it seems thattoo many terms were added to the queries, and this is supported by the differencein micro-average precision between Runs 1 and 3 and Runs 2 and 4. To overcomethis, we plan to implement a standard query expansion method where this canbe controlled.4 ConclusionsOn comparing the results of our runs with those of other participants (see TrackOverview [2] for full comparative analysis), Run 4 was shown to have the high-est f-score of all systems when averaged over the individual f-scores for each ofthe topic classes (macro-average). This was attained by deliberately promotingrecall over precision, in allowing videos to be classified under multiple topics.Furthermore, as mentioned in the Overview paper, the imbalance between pre-cision and recall may not result in a particularly useful system. From this pointof view, Run 1, which has the closest balance between precision and recall, maybe seen as most useful to a human evaluator.The results suggest that there is room for improvement in our system. Theprecision scores could be improved by finer-grained query expansion, whichwill be examined in future experiments. Additionally, the performance on theEnglish-language content could be improved by use of a stemming algorithm,such as Porter [3].AcknowledgementsThis work was supported by the EU IST programme of the 6th FP for RTDunder project MultiMATCH contract IST-033104.References1. Apache Software Foundation. Lucene: Java-based Indexing and Searching technol-ogy, http://lucene.apache.org/.2. M. Larson, E. Newman, and G. J. F. Jones. Overview of VideoCLEF2008: Auto-matic Generation of Topic-based Feeds for Dual Language Audio-Visual Content. InC. Peters, D. Giampiccolo, N. Ferro, V. Petras, J. Gonzalo, A. Pen˜as, T. Deselaers,T. Mandl, G. J. F. Jones, and M Kurimo, editors, Evaluating Systems for Multi-lingual and Multimodal Information Access – 9th Workshop of the Cross-LanguageEvaluation Forum, Lecture Notes in Computer Science, Aarhus, Denmark, Septem-ber 2008 (printed in 2009).3. M. Porter. An algorithm for suffix stripping. Program, July 1980.

An algorithm for suﬃx stripping. Program,

Apache Software Foundation. Lucene: Java-based Indexing and Searching technology,

Overview of VideoCLEF2008: Automatic Generation of Topic-based Feeds for Dual Language Audio-Visual Content.

DCU at VideoClef 2008

Eamonn Newman

Gareth J. F. Jones

Crossref

We describe a baseline system for the VideoCLEF Vid2RSS task. The system uses an unaltered off-the-shelf Information Retrieval system. ASR content is indexed using\ud
default stemming and stopping methods. The subject categories are populated by using the category label as a query on the collection, and assigning the retrieved items\ud
to that particular category. We describe the results of the system and provide some high-level analysis of its performance

Name not available

We describe a baseline system for the VideoCLEF Vid2RSS task. The system uses an unaltered off-the-shelf Information Retrieval system. ASR content is indexed using
default stemming and stopping methods. The subject categories are populated by using the category label as a query on the collection, and assigning the retrieved items
to that particular category. We describe the results of the system and provide some high-level analysis of its performance

Irish Universities

Abstract. We describe a baseline system for the VideoCLEF Vid2RSS task in which videos are to be classified into thematic categories based on their content. The system uses an off-the-shelf Information Retrieval system. Speech transcripts generated using automated speech recognition are indexed using default stemming and stopping methods. The categories are populated by using the category theme (or label) as a query on the collection, and assigning the retrieved items to that particular category. Run 4 of our system achieved the highest f-score in the task by maximising recall. We discuss this in terms of the primary aims of the task, i.e., automating video classification

Gareth J F Jones

CiteSeerX

http://doras.dcu.ie/16188/1/DCU_at_VideoClef_2008.pdf

DCU at VideoClef 2008

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Crossref

Name not available

Irish Universities

CiteSeerX