VideoCLEF is a new track for the CLEF 2008 campaign. This

track aims to develop and evaluate tasks in analyzing multilingual video content. A pilot of a Vid2RSS task involving assigning thematic class labels to video kicks off the VideoCLEF track in 2008. Task participants deliver classification results in the form of a series of feeds, one for each thematic class. The data for the task are dual language television documentaries. Dutch is the dominant language and English-language content (mostly interviews) is embedded. Participants are provided with speech recognition transcripts of the data in both Dutch and English, and also with metadata generated by archivists. In addition to the classification task, participants can choose to participate in a translation task (translating the

feed into a language of their choice) and a keyframe selection task (choosing a semantically appropriate keyframe for depiction of the videos in the feed)

Jones, Gareth J.F.

Larson, Martha

Newman, Eamonn

English

VideoCLEF is a new track for the CLEF 2008 campaign. This track aims to develop and evaluate tasks in analyzing multilingual video content. A pilot of a Vid2RSS task involving assigning thematic class labels to video kicks off the VideoCLEF track in 2008. Task participants deliver classification results in the form of a series of feeds, one for each thematic class. The data for the task are dual language television documentaries. Dutch is the dominant language and English-language content (mostly interviews) is embedded. Participants are provided with speech recognition transcripts of the data in both Dutch and English, and also with metadata generated by archivists. In addition to the classification task, participants can choose to participate in a translation task (translating the feed into a language of their choice) and a keyframe selection task (choosing a semantically appropriate keyframe for depiction of the videos in the feed)

Larson, M.

Newman, E.

Jones, G.J.F.

UvA-DARE

Classification of dual language audio-visual content: Introduction to the VideoCLEF 2008 pilot benchmark evaluation task

VideoCLEF is a new track for the CLEF 2008 campaign. This
track aims to develop and evaluate tasks in analyzing multilingual video content. A pilot of a Vid2RSS task involving assigning thematic class labels to video kicks off the VideoCLEF track in 2008. Task participants deliver classification results in the form of a series of feeds, one for each thematic class. The data for the task are dual language television documentaries. Dutch is the dominant language and English-language content (mostly interviews) is embedded. Participants are provided with speech recognition transcripts of the data in both Dutch and English, and also with metadata generated by archivists. In addition to the classification task, participants can choose to participate in a translation task (translating the
feed into a language of their choice) and a keyframe selection task (choosing a semantically appropriate keyframe for depiction of the videos in the feed)

Irish Universities

Classification of dual language audio-visual content:
Introduction to the VideoCLEF 2008 pilot benchmark
evaluation task

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Classification of dual language audio-visual content: Introduction to theVideoCLEF 2008 pilot benchmark evaluation taskLarson, M.; Newman, E.; Jones, G.J.F.Publication date2008Published inProceedings of the ACM SIGIR Workshop 'Searching Spontaneous Conversational Speech'Link to publicationCitation for published version (APA):Larson, M., Newman, E., & Jones, G. J. F. (2008). Classification of dual language audio-visual content: Introduction to the VideoCLEF 2008 pilot benchmark evaluation task. In J.Köhler, M. Larson, F. M. G. de Jong, W. Kraaij, & R. J. F. Ordelman (Eds.), Proceedings ofthe ACM SIGIR Workshop 'Searching Spontaneous Conversational Speech' (pp. 71-72).Centre for Telematics and Information Technology (CTIT).http://ilps.science.uva.nl/SSCS2008/Proceedings/sscs08_proceedings.pdfGeneral rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:31 Aug 2023Classification of Dual Language Audio-Visual Content:Introduction to the VideoCLEF 2008 Pilot BenchmarkEvaluation TaskMartha LarsonISLAUniversity of AmsterdamThe Netherlandsm.a.larson@uva.nlEamonn NewmanCentre for Digital VideoProcessingDublin City University, Irelandenewman@computing.dcu.ieGareth J. F. JonesCentre for Digital VideoProcessingDublin City University, Irelandgjones@computing.dcu.ieABSTRACTVideoCLEF is a new track for the CLEF 2008 campaign. Thistrack aims to develop and evaluate tasks in analyzing multilingualvideo content. A pilot of a Vid2RSS task involving assigning the-matic class labels to video kicks off the VideoCLEF track in 2008.Task participants deliver classification results in the form of a se-ries of feeds, one for each thematic class. The data for the task aredual language television documentaries. Dutch is the dominant lan-guage and English-language content (mostly interviews) is embed-ded. Participants are provided with speech recognition transcriptsof the data in both Dutch and English, and also with metadata gen-erated by archivists. In addition to the classification task, partici-pants can choose to participate in a translation task (translating thefeed into a language of their choice) and a keyframe selection task(choosing a semantically appropriate keyframe for depiction of thevideos in the feed).Categories and Subject DescriptorsH.3 [Information Storage and Retrieval]: H.3.1 Content Anal-ysis and Indexing; H.3.3 Information Search and Retrieval; H.3.4Systems and Software: Performance evaluationGeneral TermsMeasurement, Performance, Experimentation, LanguagesKeywordsMultimedia analysis, classification, speech recognition, benchmarktest, cross-language information retrieval1. INTRODUCTIONIn 2008, a VideoCLEF track has been introduced at the CrossLanguage Evaluation Forum (CLEF)1. The VideoCLEF track for2008 is being organized as a pilot of the Vid2RSS2 task, which in-volves a number of subtasks. The results are delivered as RSS-feeds3. The main task is classifying videos into thematic classes(e.g., Architecture, Chemistry, History). The auxiliary tasks aretranslating feeds (i.e., the textual information contained in a feed)into another language and semantic keyframe selection. The track1http://www.clef-campaign.org2http://ilps.science.uva.nl/Vid2RSS3Feeds in RSS, Real Simple Syndication, formatCopyright is held by the author/owner(s).SSCS’08 SIGIR 2008 workshop, July 24, Singapore.pursues the long term objective of developing the pipeline of pro-cessing necessary for fully automatic generation of topic-based feeds,specific to a particular information need and personalized to a par-ticular language preference. The video data for Vid2RSS 2008 areDutch television documentaries, which, while containing spokenDutch content as the matrix language, also contain a high propor-tion of spoken English (i.e., interview guests often speak in En-glish). A major focus of the Vid2RSS task is to optimize classifica-tion for videos whose spoken content is inherently multilingual.CLEF pursues the goal of supporting research and developmentin information access to multilingual content. It promotes the de-velopment of retrieval infrastructures and also test suites of data forbenchmarking that can be used and re-used by system developers.Previous and ongoing tasks have worked with multilingual newscollections, scientific data, question answering, web data, speechdata, geographic search, image search, and interactive retrieval andinformation processing [1]. The VideoCLEF task is designed to ex-tend the Cross-Language Speech Retrieval (CL-SR) run at CLEF inrecent years [3] to the broader challenge of search for video data.VideoCLEF attempts to complement the TRECVid benchmark[4] by placing its main emphasis on exploiting spoken content (viaspeech recognition transcripts) and metadata associated with videos.Although participants are free to use features derived from the vi-sual track of the video, it is not a required by the the task.RSS-format was chosen so that system output can be displayedusing a feed reader, which means that the runs performed by thepartners are immediately visualizable and can be assessed by users,for example the archive staff. In this way, dissemination and trans-fer of task achievements is promoted. Grouping data into thematiccategories is a task familiar to staff of large archives, namely to pre-pare a dossier of available resources on a certain topic for use byjournalists and editors who are creating new content for broadcast.Vid2RSS is motivated by the existence of video archives pre-dominantly in one language, but containing embedded content inanother. The extensive use of English-language interviews in Dutch-language programming means that Dutch media archives are a richsource of English language content for information seekers. Fur-ther, dual language programming offers a unique scientific oppor-tunity. The two languages occur side by side in the program, bothcontributing to the spoken content of the program. Unlike othermultilingual collections, the spoken content of the two languages istightly intertwined and one does not duplicate the other. Dual lan-guage video thus presents the challenge of how to exploit speechfeatures from both languages.The balance of this paper is dedicated to a more detailed descrip-tion of the Vid2RSS data collection and to explanation of the maintask of Vid2RSS, the classification task.712. VID2RSS 2008 TASK2.1 Task dataThe Vid2RSS 2008 task data comprise 50 episodes (30 hours)of dual language television programs that are predominantly docu-mentaries. The task data are broken down into a development set(10 episodes) and a test set (40 episodes). The video data havebeen supplied by Beeld & Geluid4 one of the largest audio/videoarchives in Europe. Beeld & Geluid is located in Hilversum, Nether-lands and is also referred to as Netherlands Institute of Sound andVision. Dutch is the matrix language and English the embeddedlanguage (i.e., the documentaries are in Dutch and the experts in-terviewed in the documentaries speak English). The videos con-tain unplanned interview speech and at times challenging back-ground conditions, such as music. Interview guests are not pro-fessional speakers and are often non-natives. Additional embeddedlanguages, e.g., German and French, occur occasionally.Automatic speech recognition (ASR) transcripts (first best hy-pothesis; MPEG-7 format) were kindly generated by the Universityof Twente using independent Dutch and English ASR-systems [2].Each episode was transcribed in its entirety by each recognizer;no language detection was applied. Keyframes with location timestamps extracted from the video data stream were generated byDublin City University [5] and are also made available to the par-ticipants, since keyframes are necessary to create the RSS-feeds,but keyframe selection is not a mandatory task. The archival meta-data record is also supplied with each video in the collection. Thisrecord contains title and description fields as well as informationabout series, creator, publisher, rights and broadcast dates. Thecontents of the subject fields have been deleted, since these are thethematic class labels Vid2RSS aims to automatically generate.2.2 Task outputThe output format is a set of RSS-feeds, one for each topic class.The RSS-feed format is trivial to generate and is used as an outputformat because it can be easily visualized in a feed reader such asNetvibes5. The RSS-feeds are created by concatenating feed items,cf. Figure 1, for the video episodes assigned to a given topic class.Figure 1: Example of feed item, the building block of a feedThe feed items contain title and description fields taken from thearchival metadata and are supplied with the task data.2.3 The Classification TaskOne step in the process of annotating a video for archival is toassign it one or more thematic class subject labels. Since archivestaff use class labels for annotation and retrieval, the class labelsautomatically generated by the Vid2RSS classification task haveconcrete potential to support video search in an archive setting, andarguably also beyond. Ten thematic categories have been chosenfor use in Vid2RSS 2008 (Archeology, Architecture, Chemistry,4http://www.beeldengeluid.nl5http://www.netvibes.comFigure 2: Visualization of example topic feed in NetvibesDance, Film, History, Music, Paintings, Scientific research and Vi-sual arts). These labels are a small subset of the class label inven-tory used by archive staff to annotate video for archival at Beeld &Geluid. We use archivist-assigned class labels contained in the sub-ject fields of the metadata record as a gold standard for evaluatingthe Vid2RSS classification results. Participants are supplied withthe topic labels and use them to collect their own training data.3. CONCLUSIONS AND OUTLOOKThe 2008 Vid2RSS task is intended as a pilot exercise to enableus to better understand the issues raised by the task of search inmultilingual video content. Experience accrued in 2008 will informthe development of new tasks for the following years. In the future,would like to scale up the task, e.g., with use of a larger video cor-pus and an extended list of topic class labels. Also under consider-ation are the inclusion of embedded languages beyond English andthe integration of additional sources of information such as subti-tles, speaker change boundaries, speaker prosody, audio events andlanguage identification. Further, introduction of new sub-tasks isenvisaged, e.g., selection of a representative series of keyframes,feed generation on the basis of ad hoc queries and summarizationaimed at automatic generation of description elements.4. ACKNOWLEDGEMENTSThis research was supported by the E.U. IST programme of the6th FP for RTD under project MultiMATCH contract IST-033104.5. REFERENCES[1] M. Braschler et al. From CLEF to TrebleCLEF: promoting Technol-ogy Transfer for Multilingual Information Retrieval. Second DELOSConference on Digital Libraries, 2007.[2] M. Huijbregts, R. Ordelman and F. de Jong Annotation of Hetero-geneous Multimedia Content Using Automatic Speech Recognition.Proceedings of SAMT, 2007.[3] P. Pecina, P. Hoffmannova, G. J. F. Jones, Y. Zhang and D. W. Oard,Overview of the CLEF 2007 Cross-Language Speech Retrieval Track,Proceedings of the CLEF 2007 Workshop, 2007.[4] A. F. Smeaton, P. Over and W. Kraaij, Evaluation Campaigns andTRECVid, In Proceedings of MIR 2006 - 8th ACM SIGMM Interna-tional Workshop on Multimedia Information Retrieval, 2006.[5] J. Calic, S. Sav, E. Izquierdo, S. Marlow, N. Murphy and N. O’ConnorTemporal Video Segmentation for Real-Time Key Frame ExtractionProceedings of ICASSP 2002.72

DCU Online Research Access Service

Classification of Dual Language Audio-Visual Content:Introduction to the VideoCLEF 2008 Pilot BenchmarkEvaluation TaskMartha LarsonISLAUniversity of AmsterdamThe Netherlandsm.a.larson@uva.nlEamonn NewmanCentre for Digital VideoProcessingDublin City University, Irelandenewman@computing.dcu.ieGareth J. F. JonesCentre for Digital VideoProcessingDublin City University, Irelandgjones@computing.dcu.ieABSTRACTVideoCLEF is a new track for the CLEF 2008 campaign. Thistrack aims to develop and evaluate tasks in analyzing multilingualvideo content. A pilot of a Vid2RSS task involving assigning the-matic class labels to video kicks off the VideoCLEF track in 2008.Task participants deliver classification results in the form of a se-ries of feeds, one for each thematic class. The data for the task aredual language television documentaries. Dutch is the dominant lan-guage and English-language content (mostly interviews) is embed-ded. Participants are provided with speech recognition transcriptsof the data in both Dutch and English, and also with metadata gen-erated by archivists. In addition to the classification task, partici-pants can choose to participate in a translation task (translating thefeed into a language of their choice) and a keyframe selection task(choosing a semantically appropriate keyframe for depiction of thevideos in the feed).Categories and Subject DescriptorsH.3 [Information Storage and Retrieval]: H.3.1 Content Anal-ysis and Indexing; H.3.3 Information Search and Retrieval; H.3.4Systems and Software: Performance evaluationGeneral TermsMeasurement, Performance, Experimentation, LanguagesKeywordsMultimedia analysis, classification, speech recognition, benchmarktest, cross-language information retrieval1. INTRODUCTIONIn 2008, a VideoCLEF track has been introduced at the CrossLanguage Evaluation Forum (CLEF)1. The VideoCLEF track for2008 is being organized as a pilot of the Vid2RSS2 task, which in-volves a number of subtasks. The results are delivered as RSS-feeds3. The main task is classifying videos into thematic classes(e.g., Architecture, Chemistry, History). The auxiliary tasks aretranslating feeds (i.e., the textual information contained in a feed)into another language and semantic keyframe selection. The track1http://www.clef-campaign.org2http://ilps.science.uva.nl/Vid2RSS3Feeds in RSS, Real Simple Syndication, formatCopyright is held by the author/owner(s).SSCS’08 SIGIR 2008 workshop, July 24, Singapore.pursues the long term objective of developing the pipeline of pro-cessing necessary for fully automatic generation of topic-based feeds,specific to a particular information need and personalized to a par-ticular language preference. The video data for Vid2RSS 2008 areDutch television documentaries, which, while containing spokenDutch content as the matrix language, also contain a high propor-tion of spoken English (i.e., interview guests often speak in En-glish). A major focus of the Vid2RSS task is to optimize classifica-tion for videos whose spoken content is inherently multilingual.CLEF pursues the goal of supporting research and developmentin information access to multilingual content. It promotes the de-velopment of retrieval infrastructures and also test suites of data forbenchmarking that can be used and re-used by system developers.Previous and ongoing tasks have worked with multilingual newscollections, scientific data, question answering, web data, speechdata, geographic search, image search, and interactive retrieval andinformation processing [1]. The VideoCLEF task is designed to ex-tend the Cross-Language Speech Retrieval (CL-SR) run at CLEF inrecent years [3] to the broader challenge of search for video data.VideoCLEF attempts to complement the TRECVid benchmark[4] by placing its main emphasis on exploiting spoken content (viaspeech recognition transcripts) and metadata associated with videos.Although participants are free to use features derived from the vi-sual track of the video, it is not a required by the the task.RSS-format was chosen so that system output can be displayedusing a feed reader, which means that the runs performed by thepartners are immediately visualizable and can be assessed by users,for example the archive staff. In this way, dissemination and trans-fer of task achievements is promoted. Grouping data into thematiccategories is a task familiar to staff of large archives, namely to pre-pare a dossier of available resources on a certain topic for use byjournalists and editors who are creating new content for broadcast.Vid2RSS is motivated by the existence of video archives pre-dominantly in one language, but containing embedded content inanother. The extensive use of English-language interviews in Dutch-language programming means that Dutch media archives are a richsource of English language content for information seekers. Fur-ther, dual language programming offers a unique scientific oppor-tunity. The two languages occur side by side in the program, bothcontributing to the spoken content of the program. Unlike othermultilingual collections, the spoken content of the two languages istightly intertwined and one does not duplicate the other. Dual lan-guage video thus presents the challenge of how to exploit speechfeatures from both languages.The balance of this paper is dedicated to a more detailed descrip-tion of the Vid2RSS data collection and to explanation of the maintask of Vid2RSS, the classification task.2. VID2RSS 2008 TASK2.1 Task dataThe Vid2RSS 2008 task data comprise 50 episodes (30 hours)of dual language television programs that are predominantly docu-mentaries. The task data are broken down into a development set(10 episodes) and a test set (40 episodes). The video data havebeen supplied by Beeld & Geluid4 one of the largest audio/videoarchives in Europe. Beeld &Geluid is located in Hilversum, Nether-lands and is also referred to as Netherlands Institute of Sound andVision. Dutch is the matrix language and English the embeddedlanguage (i.e., the documentaries are in Dutch and the experts in-terviewed in the documentaries speak English). The videos con-tain unplanned interview speech and at times challenging back-ground conditions, such as music. Interview guests are not pro-fessional speakers and are often non-natives. Additional embeddedlanguages, e.g., German and French, occur occasionally.Automatic speech recognition (ASR) transcripts (first best hy-pothesis; MPEG-7 format) were kindly generated by the Universityof Twente using independent Dutch and English ASR-systems [2].Each episode was transcribed in its entirety by each recognizer;no language detection was applied. Keyframes with location timestamps extracted from the video data stream were generated byDublin City University [5] and are also made available to the par-ticipants, since keyframes are necessary to create the RSS-feeds,but keyframe selection is not a mandatory task. The archival meta-data record is also supplied with each video in the collection. Thisrecord contains title and description fields as well as informationabout series, creator, publisher, rights and broadcast dates. Thecontents of the subject fields have been deleted, since these are thethematic class labels Vid2RSS aims to automatically generate.2.2 Task outputThe output format is a set of RSS-feeds, one for each topic class.The RSS-feed format is trivial to generate and is used as an outputformat because it can be easily visualized in a feed reader such asNetvibes5. The RSS-feeds are created by concatenating feed items,cf. Figure 1, for the video episodes assigned to a given topic class.Figure 1: Example of feed item, the building block of a feedThe feed items contain title and description fields taken from thearchival metadata and are supplied with the task data.2.3 The Classification TaskOne step in the process of annotating a video for archival is toassign it one or more thematic class subject labels. Since archivestaff use class labels for annotation and retrieval, the class labelsautomatically generated by the Vid2RSS classification task haveconcrete potential to support video search in an archive setting, andarguably also beyond. Ten thematic categories have been chosenfor use in Vid2RSS 2008 (Archeology, Architecture, Chemistry,4http://www.beeldengeluid.nl5http://www.netvibes.comFigure 2: Visualization of example topic feed in NetvibesDance, Film, History, Music, Paintings, Scientific research and Vi-sual arts). These labels are a small subset of the class label inven-tory used by archive staff to annotate video for archival at Beeld &Geluid. We use archivist-assigned class labels contained in the sub-ject fields of the metadata record as a gold standard for evaluatingthe Vid2RSS classification results. Participants are supplied withthe topic labels and use them to collect their own training data.3. CONCLUSIONS AND OUTLOOKThe 2008 Vid2RSS task is intended as a pilot exercise to enableus to better understand the issues raised by the task of search inmultilingual video content. Experience accrued in 2008 will informthe development of new tasks for the following years. In the future,would like to scale up the task, e.g., with use of a larger video cor-pus and an extended list of topic class labels. Also under consider-ation are the inclusion of embedded languages beyond English andthe integration of additional sources of information such as subti-tles, speaker change boundaries, speaker prosody, audio events andlanguage identification. Further, introduction of new sub-tasks isenvisaged, e.g., selection of a representative series of keyframes,feed generation on the basis of ad hoc queries and summarizationaimed at automatic generation of description elements.4. ACKNOWLEDGEMENTSThis research was supported by the E.U. IST programme of the6th FP for RTD under project MultiMATCH contract IST-033104.5. REFERENCES[1] M. Braschler et al. From CLEF to TrebleCLEF: promoting Technol-ogy Transfer for Multilingual Information Retrieval. Second DELOSConference on Digital Libraries, 2007.[2] M. Huijbregts, R. Ordelman and F. de Jong Annotation of Hetero-geneous Multimedia Content Using Automatic Speech Recognition.Proceedings of SAMT, 2007.[3] P. Pecina, P. Hoffmannova, G. J. F. Jones, Y. Zhang and D. W. Oard,Overview of the CLEF 2007 Cross-Language Speech Retrieval Track,Proceedings of the CLEF 2007 Workshop, 2007.[4] A. F. Smeaton, P. Over and W. Kraaij, Evaluation Campaigns andTRECVid, In Proceedings of MIR 2006 - 8th ACM SIGMM Interna-tional Workshop on Multimedia Information Retrieval, 2006.[5] J. Calic, S. Sav, E. Izquierdo, S. Marlow, N. Murphy and N. O’ConnorTemporal Video Segmentation for Real-Time Key Frame ExtractionProceedings of ICASSP 2002.

Annotation of Heterogeneous Multimedia Content Using Automatic Speech Recognition.

Cross-Language Speech Retrieval Track,

Evaluation Campaigns and TRECVid,

From CLEF to TrebleCLEF: promoting Technology Transfer for Multilingual Information Retrieval.

O’Connor Temporal Video Segmentation for Real-Time Key Frame Extraction

Classification of dual language audio-visual content:

Introduction to the VideoCLEF 2008 pilot benchmark

evaluation task

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Classification of dual language audio-visual content: Introduction to theVideoCLEF 2008 pilot benchmark evaluation taskLarson, M.; Newman, E.; Jones, G.J.F.Publication date2008Published inProceedings of the ACM SIGIR Workshop 'Searching Spontaneous Conversational Speech'Link to publicationCitation for published version (APA):Larson, M., Newman, E., & Jones, G. J. F. (2008). Classification of dual language audio-visual content: Introduction to the VideoCLEF 2008 pilot benchmark evaluation task. In J.Köhler, M. Larson, F. M. G. de Jong, W. Kraaij, & R. J. F. Ordelman (Eds.), Proceedings ofthe ACM SIGIR Workshop 'Searching Spontaneous Conversational Speech' (pp. 71-72).Centre for Telematics and Information Technology (CTIT).http://ilps.science.uva.nl/SSCS2008/Proceedings/sscs08_proceedings.pdfGeneral rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:25 Jul 2022Classification of Dual Language Audio-Visual Content:Introduction to the VideoCLEF 2008 Pilot BenchmarkEvaluation TaskMartha LarsonISLAUniversity of AmsterdamThe Netherlandsm.a.larson@uva.nlEamonn NewmanCentre for Digital VideoProcessingDublin City University, Irelandenewman@computing.dcu.ieGareth J. F. JonesCentre for Digital VideoProcessingDublin City University, Irelandgjones@computing.dcu.ieABSTRACTVideoCLEF is a new track for the CLEF 2008 campaign. Thistrack aims to develop and evaluate tasks in analyzing multilingualvideo content. A pilot of a Vid2RSS task involving assigning the-matic class labels to video kicks off the VideoCLEF track in 2008.Task participants deliver classification results in the form of a se-ries of feeds, one for each thematic class. The data for the task aredual language television documentaries. Dutch is the dominant lan-guage and English-language content (mostly interviews) is embed-ded. Participants are provided with speech recognition transcriptsof the data in both Dutch and English, and also with metadata gen-erated by archivists. In addition to the classification task, partici-pants can choose to participate in a translation task (translating thefeed into a language of their choice) and a keyframe selection task(choosing a semantically appropriate keyframe for depiction of thevideos in the feed).Categories and Subject DescriptorsH.3 [Information Storage and Retrieval]: H.3.1 Content Anal-ysis and Indexing; H.3.3 Information Search and Retrieval; H.3.4Systems and Software: Performance evaluationGeneral TermsMeasurement, Performance, Experimentation, LanguagesKeywordsMultimedia analysis, classification, speech recognition, benchmarktest, cross-language information retrieval1. INTRODUCTIONIn 2008, a VideoCLEF track has been introduced at the CrossLanguage Evaluation Forum (CLEF)1. The VideoCLEF track for2008 is being organized as a pilot of the Vid2RSS2 task, which in-volves a number of subtasks. The results are delivered as RSS-feeds3. The main task is classifying videos into thematic classes(e.g., Architecture, Chemistry, History). The auxiliary tasks aretranslating feeds (i.e., the textual information contained in a feed)into another language and semantic keyframe selection. The track1http://www.clef-campaign.org2http://ilps.science.uva.nl/Vid2RSS3Feeds in RSS, Real Simple Syndication, formatCopyright is held by the author/owner(s).SSCS’08 SIGIR 2008 workshop, July 24, Singapore.pursues the long term objective of developing the pipeline of pro-cessing necessary for fully automatic generation of topic-based feeds,specific to a particular information need and personalized to a par-ticular language preference. The video data for Vid2RSS 2008 areDutch television documentaries, which, while containing spokenDutch content as the matrix language, also contain a high propor-tion of spoken English (i.e., interview guests often speak in En-glish). A major focus of the Vid2RSS task is to optimize classifica-tion for videos whose spoken content is inherently multilingual.CLEF pursues the goal of supporting research and developmentin information access to multilingual content. It promotes the de-velopment of retrieval infrastructures and also test suites of data forbenchmarking that can be used and re-used by system developers.Previous and ongoing tasks have worked with multilingual newscollections, scientific data, question answering, web data, speechdata, geographic search, image search, and interactive retrieval andinformation processing [1]. The VideoCLEF task is designed to ex-tend the Cross-Language Speech Retrieval (CL-SR) run at CLEF inrecent years [3] to the broader challenge of search for video data.VideoCLEF attempts to complement the TRECVid benchmark[4] by placing its main emphasis on exploiting spoken content (viaspeech recognition transcripts) and metadata associated with videos.Although participants are free to use features derived from the vi-sual track of the video, it is not a required by the the task.RSS-format was chosen so that system output can be displayedusing a feed reader, which means that the runs performed by thepartners are immediately visualizable and can be assessed by users,for example the archive staff. In this way, dissemination and trans-fer of task achievements is promoted. Grouping data into thematiccategories is a task familiar to staff of large archives, namely to pre-pare a dossier of available resources on a certain topic for use byjournalists and editors who are creating new content for broadcast.Vid2RSS is motivated by the existence of video archives pre-dominantly in one language, but containing embedded content inanother. The extensive use of English-language interviews in Dutch-language programming means that Dutch media archives are a richsource of English language content for information seekers. Fur-ther, dual language programming offers a unique scientific oppor-tunity. The two languages occur side by side in the program, bothcontributing to the spoken content of the program. Unlike othermultilingual collections, the spoken content of the two languages istightly intertwined and one does not duplicate the other. Dual lan-guage video thus presents the challenge of how to exploit speechfeatures from both languages.The balance of this paper is dedicated to a more detailed descrip-tion of the Vid2RSS data collection and to explanation of the maintask of Vid2RSS, the classification task.712. VID2RSS 2008 TASK2.1 Task dataThe Vid2RSS 2008 task data comprise 50 episodes (30 hours)of dual language television programs that are predominantly docu-mentaries. The task data are broken down into a development set(10 episodes) and a test set (40 episodes). The video data havebeen supplied by Beeld & Geluid4 one of the largest audio/videoarchives in Europe. Beeld & Geluid is located in Hilversum, Nether-lands and is also referred to as Netherlands Institute of Sound andVision. Dutch is the matrix language and English the embeddedlanguage (i.e., the documentaries are in Dutch and the experts in-terviewed in the documentaries speak English). The videos con-tain unplanned interview speech and at times challenging back-ground conditions, such as music. Interview guests are not pro-fessional speakers and are often non-natives. Additional embeddedlanguages, e.g., German and French, occur occasionally.Automatic speech recognition (ASR) transcripts (first best hy-pothesis; MPEG-7 format) were kindly generated by the Universityof Twente using independent Dutch and English ASR-systems [2].Each episode was transcribed in its entirety by each recognizer;no language detection was applied. Keyframes with location timestamps extracted from the video data stream were generated byDublin City University [5] and are also made available to the par-ticipants, since keyframes are necessary to create the RSS-feeds,but keyframe selection is not a mandatory task. The archival meta-data record is also supplied with each video in the collection. Thisrecord contains title and description fields as well as informationabout series, creator, publisher, rights and broadcast dates. Thecontents of the subject fields have been deleted, since these are thethematic class labels Vid2RSS aims to automatically generate.2.2 Task outputThe output format is a set of RSS-feeds, one for each topic class.The RSS-feed format is trivial to generate and is used as an outputformat because it can be easily visualized in a feed reader such asNetvibes5. The RSS-feeds are created by concatenating feed items,cf. Figure 1, for the video episodes assigned to a given topic class.Figure 1: Example of feed item, the building block of a feedThe feed items contain title and description fields taken from thearchival metadata and are supplied with the task data.2.3 The Classification TaskOne step in the process of annotating a video for archival is toassign it one or more thematic class subject labels. Since archivestaff use class labels for annotation and retrieval, the class labelsautomatically generated by the Vid2RSS classification task haveconcrete potential to support video search in an archive setting, andarguably also beyond. Ten thematic categories have been chosenfor use in Vid2RSS 2008 (Archeology, Architecture, Chemistry,4http://www.beeldengeluid.nl5http://www.netvibes.comFigure 2: Visualization of example topic feed in NetvibesDance, Film, History, Music, Paintings, Scientific research and Vi-sual arts). These labels are a small subset of the class label inven-tory used by archive staff to annotate video for archival at Beeld &Geluid. We use archivist-assigned class labels contained in the sub-ject fields of the metadata record as a gold standard for evaluatingthe Vid2RSS classification results. Participants are supplied withthe topic labels and use them to collect their own training data.3. CONCLUSIONS AND OUTLOOKThe 2008 Vid2RSS task is intended as a pilot exercise to enableus to better understand the issues raised by the task of search inmultilingual video content. Experience accrued in 2008 will informthe development of new tasks for the following years. In the future,would like to scale up the task, e.g., with use of a larger video cor-pus and an extended list of topic class labels. Also under consider-ation are the inclusion of embedded languages beyond English andthe integration of additional sources of information such as subti-tles, speaker change boundaries, speaker prosody, audio events andlanguage identification. Further, introduction of new sub-tasks isenvisaged, e.g., selection of a representative series of keyframes,feed generation on the basis of ad hoc queries and summarizationaimed at automatic generation of description elements.4. ACKNOWLEDGEMENTSThis research was supported by the E.U. IST programme of the6th FP for RTD under project MultiMATCH contract IST-033104.5. REFERENCES[1] M. Braschler et al. From CLEF to TrebleCLEF: promoting Technol-ogy Transfer for Multilingual Information Retrieval. Second DELOSConference on Digital Libraries, 2007.[2] M. Huijbregts, R. Ordelman and F. de Jong Annotation of Hetero-geneous Multimedia Content Using Automatic Speech Recognition.Proceedings of SAMT, 2007.[3] P. Pecina, P. Hoffmannova, G. J. F. Jones, Y. Zhang and D. W. Oard,Overview of the CLEF 2007 Cross-Language Speech Retrieval Track,Proceedings of the CLEF 2007 Workshop, 2007.[4] A. F. Smeaton, P. Over and W. Kraaij, Evaluation Campaigns andTRECVid, In Proceedings of MIR 2006 - 8th ACM SIGMM Interna-tional Workshop on Multimedia Information Retrieval, 2006.[5] J. Calic, S. Sav, E. Izquierdo, S. Marlow, N. Murphy and N. O’ConnorTemporal Video Segmentation for Real-Time Key Frame ExtractionProceedings of ICASSP 2002.72

Search4Dev

UvA-DARE is a service provided by the library of the University of Amsterdam (http://dare.uva.nl)
UvA-DARE (Digital Academic Repository)
Classification of dual language audio-visual content: Introduction to the VideoCLEF 2008 pilot
benchmark evaluation task
Larson, M.; Newman, E.; Jones, G.J.F.
Published in:
Proceedings of the ACM SIGIR Workshop 'Searching Spontaneous Conversational Speech'
Link to publication
Citation for published version (APA):
Larson, M., Newman, E., & Jones, G. J. F. (2008). Classification of dual language audio-visual content:
Introduction to the VideoCLEF 2008 pilot benchmark evaluation task. In J. Köhler, M. Larson, F. M. G. de Jong,
W. Kraaij, & R. J. F. Ordelman (Eds.), Proceedings of the ACM SIGIR Workshop 'Searching Spontaneous
Conversational Speech' (pp. 71-72). Centre for Telematics and Information Technology (CTIT).
http://ilps.science.uva.nl/SSCS2008/Proceedings/sscs08_proceedings.pdf
General rights
It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s),
other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).
Disclaimer/Complaints regulations
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating
your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask
the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam,
The Netherlands. You will be contacted as soon as possible.
Download date: 04 Feb 2021
Classification of Dual Language Audio-Visual Content:
Introduction to the VideoCLEF 2008 Pilot Benchmark
Evaluation Task
Martha Larson
ISLA
University of Amsterdam
The Netherlands
m.a.larson@uva.nl
Eamonn Newman
Centre for Digital Video
Processing
Dublin City University, Ireland
enewman@computing.dcu.ie
Gareth J. F. Jones
Centre for Digital Video
Processing
Dublin City University, Ireland
gjones@computing.dcu.ie
ABSTRACT
VideoCLEF is a new track for the CLEF 2008 campaign. This
track aims to develop and evaluate tasks in analyzing multilingual
video content. A pilot of a Vid2RSS task involving assigning the-
matic class labels to video kicks off the VideoCLEF track in 2008.
Task participants deliver classification results in the form of a se-
ries of feeds, one for each thematic class. The data for the task are
dual language television documentaries. Dutch is the dominant lan-
guage and English-language content (mostly interviews) is embed-
ded. Participants are provided with speech recognition transcripts
of the data in both Dutch and English, and also with metadata gen-
erated by archivists. In addition to the classification task, partici-
pants can choose to participate in a translation task (translating the
feed into a language of their choice) and a keyframe selection task
(choosing a semantically appropriate keyframe for depiction of the
videos in the feed).
Categories and Subject Descriptors
H.3 [Information Storage and Retrieval]: H.3.1 Content Anal-
ysis and Indexing; H.3.3 Information Search and Retrieval; H.3.4
Systems and Software: Performance evaluation
General Terms
Measurement, Performance, Experimentation, Languages
Keywords
Multimedia analysis, classification, speech recognition, benchmark
test, cross-language information retrieval
1. INTRODUCTION
In 2008, a VideoCLEF track has been introduced at the Cross
Language Evaluation Forum (CLEF)1. The VideoCLEF track for
2008 is being organized as a pilot of the Vid2RSS2 task, which in-
volves a number of subtasks. The results are delivered as RSS-
feeds3. The main task is classifying videos into thematic classes
(e.g., Architecture, Chemistry, History). The auxiliary tasks are
translating feeds (i.e., the textual information contained in a feed)
into another language and semantic keyframe selection. The track
1http://www.clef-campaign.org
2http://ilps.science.uva.nl/Vid2RSS
3Feeds in RSS, Real Simple Syndication, format
Copyright is held by the author/owner(s).
SSCS’08 SIGIR 2008 workshop, July 24, Singapore
.
pursues the long term objective of developing the pipeline of pro-
cessing necessary for fully automatic generation of topic-based feeds,
specific to a particular information need and personalized to a par-
ticular language preference. The video data for Vid2RSS 2008 are
Dutch television documentaries, which, while containing spoken
Dutch content as the matrix language, also contain a high propor-
tion of spoken English (i.e., interview guests often speak in En-
glish). A major focus of the Vid2RSS task is to optimize classifica-
tion for videos whose spoken content is inherently multilingual.
CLEF pursues the goal of supporting research and development
in information access to multilingual content. It promotes the de-
velopment of retrieval infrastructures and also test suites of data for
benchmarking that can be used and re-used by system developers.
Previous and ongoing tasks have worked with multilingual news
collections, scientific data, question answering, web data, speech
data, geographic search, image search, and interactive retrieval and
information processing [1]. The VideoCLEF task is designed to ex-
tend the Cross-Language Speech Retrieval (CL-SR) run at CLEF in
recent years [3] to the broader challenge of search for video data.
VideoCLEF attempts to complement the TRECVid benchmark
[4] by placing its main emphasis on exploiting spoken content (via
speech recognition transcripts) and metadata associated with videos.
Although participants are free to use features derived from the vi-
sual track of the video, it is not a required by the the task.
RSS-format was chosen so that system output can be displayed
using a feed reader, which means that the runs performed by the
partners are immediately visualizable and can be assessed by users,
for example the archive staff. In this way, dissemination and trans-
fer of task achievements is promoted. Grouping data into thematic
categories is a task familiar to staff of large archives, namely to pre-
pare a dossier of available resources on a certain topic for use by
journalists and editors who are creating new content for broadcast.
Vid2RSS is motivated by the existence of video archives pre-
dominantly in one language, but containing embedded content in
another. The extensive use of English-language interviews in Dutch-
language programming means that Dutch media archives are a rich
source of English language content for information seekers. Fur-
ther, dual language programming offers a unique scientific oppor-
tunity. The two languages occur side by side in the program, both
contributing to the spoken content of the program. Unlike other
multilingual collections, the spoken content of the two languages is
tightly intertwined and one does not duplicate the other. Dual lan-
guage video thus presents the challenge of how to exploit speech
features from both languages.
The balance of this paper is dedicated to a more detailed descrip-
tion of the Vid2RSS data collection and to explanation of the main
task of Vid2RSS, the classification task.
71
2. VID2RSS 2008 TASK
2.1 Task data
The Vid2RSS 2008 task data comprise 50 episodes (30 hours)
of dual language television programs that are predominantly docu-
mentaries. The task data are broken down into a development set
(10 episodes) and a test set (40 episodes). The video data have
been supplied by Beeld & Geluid4 one of the largest audio/video
archives in Europe. Beeld & Geluid is located in Hilversum, Nether-
lands and is also referred to as Netherlands Institute of Sound and
Vision. Dutch is the matrix language and English the embedded
language (i.e., the documentaries are in Dutch and the experts in-
terviewed in the documentaries speak English). The videos con-
tain unplanned interview speech and at times challenging back-
ground conditions, such as music. Interview guests are not pro-
fessional speakers and are often non-natives. Additional embedded
languages, e.g., German and French, occur occasionally.
Automatic speech recognition (ASR) transcripts (first best hy-
pothesis; MPEG-7 format) were kindly generated by the University
of Twente using independent Dutch and English ASR-systems [2].
Each episode was transcribed in its entirety by each recognizer;
no language detection was applied. Keyframes with location time
stamps extracted from the video data stream were generated by
Dublin City University [5] and are also made available to the par-
ticipants, since keyframes are necessary to create the RSS-feeds,
but keyframe selection is not a mandatory task. The archival meta-
data record is also supplied with each video in the collection. This
record contains title and description fields as well as information
about series, creator, publisher, rights and broadcast dates. The
contents of the subject fields have been deleted, since these are the
thematic class labels Vid2RSS aims to automatically generate.
2.2 Task output
The output format is a set of RSS-feeds, one for each topic class.
The RSS-feed format is trivial to generate and is used as an output
format because it can be easily visualized in a feed reader such as
Netvibes5. The RSS-feeds are created by concatenating feed items,
cf. Figure 1, for the video episodes assigned to a given topic class.
Figure 1: Example of feed item, the building block of a feed
The feed items contain title and description fields taken from the
archival metadata and are supplied with the task data.
2.3 The Classification Task
One step in the process of annotating a video for archival is to
assign it one or more thematic class subject labels. Since archive
staff use class labels for annotation and retrieval, the class labels
automatically generated by the Vid2RSS classification task have
concrete potential to support video search in an archive setting, and
arguably also beyond. Ten thematic categories have been chosen
for use in Vid2RSS 2008 (Archeology, Architecture, Chemistry,
4http://www.beeldengeluid.nl
5http://www.netvibes.com
Figure 2: Visualization of example topic feed in Netvibes
Dance, Film, History, Music, Paintings, Scientific research and Vi-
sual arts). These labels are a small subset of the class label inven-
tory used by archive staff to annotate video for archival at Beeld &
Geluid. We use archivist-assigned class labels contained in the sub-
ject fields of the metadata record as a gold standard for evaluating
the Vid2RSS classification results. Participants are supplied with
the topic labels and use them to collect their own training data.
3. CONCLUSIONS AND OUTLOOK
The 2008 Vid2RSS task is intended as a pilot exercise to enable
us to better understand the issues raised by the task of search in
multilingual video content. Experience accrued in 2008 will inform
the development of new tasks for the following years. In the future,
would like to scale up the task, e.g., with use of a larger video cor-
pus and an extended list of topic class labels. Also under consider-
ation are the inclusion of embedded languages beyond English and
the integration of additional sources of information such as subti-
tles, speaker change boundaries, speaker prosody, audio events and
language identification. Further, introduction of new sub-tasks is
envisaged, e.g., selection of a representative series of keyframes,
feed generation on the basis of ad hoc queries and summarization
aimed at automatic generation of description elements.
4. ACKNOWLEDGEMENTS
This research was supported by the E.U. IST programme of the
6th FP for RTD under project MultiMATCH contract IST-033104.
5. REFERENCES
[1] M. Braschler et al. From CLEF to TrebleCLEF: promoting Technol-
ogy Transfer for Multilingual Information Retrieval. Second DELOS
Conference on Digital Libraries, 2007.
[2] M. Huijbregts, R. Ordelman and F. de Jong Annotation of Hetero-
geneous Multimedia Content Using Automatic Speech Recognition.
Proceedings of SAMT, 2007.
[3] P. Pecina, P. Hoffmannova, G. J. F. Jones, Y. Zhang and D. W. Oard,
Overview of the CLEF 2007 Cross-Language Speech Retrieval Track,
Proceedings of the CLEF 2007 Workshop, 2007.
[4] A. F. Smeaton, P. Over and W. Kraaij, Evaluation Campaigns and
TRECVid, In Proceedings of MIR 2006 - 8th ACM SIGMM Interna-
tional Workshop on Multimedia Information Retrieval, 2006.
[5] J. Calic, S. Sav, E. Izquierdo, S. Marlow, N. Murphy and N. O’Connor
Temporal Video Segmentation for Real-Time Key Frame Extraction
Proceedings of ICASSP 2002.
72


https://pure.uva.nl/ws/files/1093230/62220_296646.pdf

Classification of dual language audio-visual content: Introduction to the VideoCLEF 2008 pilot benchmark evaluation task

Abstract

Similar works

Full text

Available Versions

UvA-DARE

Irish Universities

International Migration, Integration and Social Cohesion online publications

DCU Online Research Access Service

International Migration, Integration and Social Cohesion online publications

Search4Dev