The MediaEval 2011 Rich Speech Retrieval Tasks and Genre Tagging Tasks are two new tasks oered in MediaEval 2011 that are designed to explore the development of techniques for semi-professional user generated content (SPUG). They both use the same data set: the MediaEval 2010 Wild Wild Web Tagging Task (ME10WWW). The ME10WWW data set contains Creative Commons licensed video collected from blip.tv in 2009. It was created by the PetaMedia Network of Excellence (http://www.petamedia.eu) in order to test retrieval algorithms for video content as it occurs `in the wild' on the Internet and, in particular, for user contributed multimedia that is embedded within a social network. In this overview paper, we repeat the essential characteristics of the data set, describe the tasks and specify how they are evaluated

Eskevich, Maria

Jones, Gareth J.F.

Kofler, Christoph

Larson, Martha

Ordelman, Roeland

Schmiedeke, Sebastian

English

Name not available

Overview of MediaEval 2011 Rich Speech Retrieval Taskand Genre Tagging TaskMartha Larson∗m.a.larson@tudelft.nlMaria Eskevich†meskevich@computing.dcu.ieRoeland Ordelman‡roeland.ordelman@utwente.nlChristoph Kofler∗c.kofler@tudelft.nlSebastian Schmiedeke§schmiedeke@nue.tu-berlin.deGareth J. F. Jones†Gareth.Jones@computing.dcu.ieCategories and Subject DescriptorsH.3.1 [Information Storage and Retrieval]: ContentAnalysis and Indexing—Indexing methodsGeneral TermsMeasurement, Performance1. INTRODUCTIONThe MediaEval 2011 Rich Speech Retrieval Tasks andGenre Tagging Tasks are two new tasks offered in MediaEval2011 that are designed to explore the development of tech-niques for semi-professional user generated content (SPUG).They both use the same data set: the MediaEval 2010 WildWild Web Tagging Task (ME10WWW). The ME10WWWdata set contains Creative Commons licensed video collectedfrom blip.tv in 2009. It was created by the PetaMedia Net-work of Excellence (http://www.petamedia.eu) in order totest retrieval algorithms for video content as it occurs ‘in thewild’ on the Internet and, in particular, for user contributedmultimedia that is embedded within a social network. Thedata set was initially described in [3]. In this overview pa-per, we repeat the essential characteristics of the data set,describe the tasks and specify how they are evaluated.2. THE ME10WWW BLIP.TV DATA SETThe ME10WWW data set consists of video episodes fromblip.tv and a social network comprised of Twitter users tweet-ing about them. Videos were collected for shows for whichthe link to one of their episodes had been tweeted. Topsy(http://topsy.com) was used to collect blip.tv links fromtweets. Their licenses were checked to confirm that theywere Creative Commons and then the videos were down-loaded from blip.tv. Topsy was then searched again to gatherall users mentioning any one of the videos. We crawled thetweets and social network of these users for two steps (i.e.,collected their interlocutors and their interlocutors’s inter-locutors). The data set contains 1974 episodes (247 devel-opment and 1727 test) comprising a total of ca. 350 hours of∗Delft University of Technology, Netherlands†Dublin City University, Ireland‡Netherlands Institute for Sound & Vision and Universityof Twente, Netherlands§Technische Universita¨t Berlin, GermanyCopyright is held by the author/owner(s).MediaEval 2011 Workshop, September 1-2, 2011, Pisa, Italydata. The development set is small with respect to the testset and is not intended for training, but rather for param-eter tuning. The episodes were chosen from 460 differentshows—shows with less than four episodes were not consid-ered for inclusion in the data set. Each video is associatedwith metadata record including uploader assigned informa-tion (title, description, license, tags, uploaded ID/series ID).The data set is accompanied by automatic speech recog-nition (ASR) transcripts [2], which were generously pro-vided by LIMSI (http://www.limsi.fr/) and Vocapia Re-search (http://www.vocapia.com/) to MediaEval. In orderto be included in the ME10WWW set, a video needed tohave been transcribed by the ASR-system with an averageword-level confidence score of > 0.7. The set is predom-inantly English with approximate 6 hours of non-Englishcontent divided over French, Spanish and Dutch. Specifi-cally for 2011, LIMSI/Vocapia also provided a second set of“confusion networks”, meaning that for a given time-point(time code), the transcripts may contain more than one hy-potheses of the ASR-system.The data set is also accompanied by the output of a shotdetection system developed at the Technische Universita¨tBerlin [1]. Both shot boundaries and extracted keyframes(one per shot) are included.3. RICH SPEECH RETRIEVAL TASKThe Rich Speech Retrieval task is a known-item retrievaltask and requires participants to return a ranked list of re-sults in response to a query. The features used can be de-rived from speech, audio, visual content or metadata. Thetask can be considered to be a ‘new generation’ spoken con-tent retrieval task in two respects. First, instead of requiringthe identification of spoken documents or speech segments,the task requires the return of jump-in points, time pointsin the video at which users must start watching to viewmaterial relevant to the query. Second, the queries used ex-press the user information need along three dimensions: thetopical content, the speech act and the visual content.The speech act dimension is, to our knowledge, investi-gated for the first time with this data set. When speakersspeak, they are, on the one hand, pronouncing words, buton the other hand they are also actually ‘doing’ something.Treating spoken content in terms of ‘illocutionary speechacts’ (http://en.wikipedia.org/wiki/Speech acts) emphasizeswhat speakers are accomplishing by speaking. The five speechacts used are ‘apology’, ‘definition’, ‘opinion’, ‘promise’ and‘warning’. These categories are similar to the ‘speech-actlike’ units that have ben used in dialogue act modeling [5].We are motivated to investigate this dimension because weconjecture that a connection exists between the reason forwhich a speaker makes an utterance and the reason for whicha user would later search for the utterance. We assume thatthis connection could be used to improve spoken contentretrieval systems.The data set includes 30 queries associated with the de-velopment set and 50 queries associated with the test set.The form of queries includes a long form (<title>), a shortform (<short_title>) and a label indicating the speech acttype (act). An example of an apology is the following. Longform: ‘How does Peter Busch, staff member of the MorningSwim Show, save face after the faux pas he made during hisinterview with Terry Denison?’. Short form: ‘Peter Buschpresident chairman Denison morning Swim Show’. Althoughsome queries in the data set make reference to visual char-acteristics (i.e., the speaker’s clothing), most resemble thisexample and do not clearly need a visual contribution. Par-ticipants are required to make one submission for all testtopics that uses only the provided ASR transcript (2010),and a second one using any combination of techniques whichthey believe will give the best overall performance. In therequired run the participants must use the full queries.The query set was created by having human annotatorslocate portions of the videos that they would be interestedin sharing online and then formulating a comment on whatthe video portion was about (long form) and a query thatwould allow them to re-find jump-in points correspondingto those portions (short form). Online-sharing was cho-sen as the scenario, since it seemed to be the most natural,widely-understood reason for which users would be attempt-ing to re-find previously seen video segments (i.e., knownitems). Access to a sufficient number of human annotatorswas secured by using a crowdsourcing platform, Amazon’sMechanical Turk (http://www.mturk.com). The task wascarefully designed, the context of online video-sharing wasclearly described and the language kept simple, i.e., “Whenyou come across something interesting you might want toshare on Facebook, Twitter or your favorite social network.”Examples were given of quotes falling into each of the speechact categories. We note that ‘opinions’ were much more fre-quently located in the corpus than other categories such as‘warning’ or ‘promise’. The pairs of query+jump-in-pointreturned by the crowdsourcing workers were subjected toa standard quality control procedure (which eliminated ca.40% of the results) and then by a screening for suitabilityand completeness (which eliminated a further ca. 40%).The official evaluation metric of the task is mGAP, cf. [4],which generalizes the relevance of hypothesized jump-in pointsin relation to ground truth points by imposing a symmetricstep-wise linearly decaying penalty function within a windowof tolerance (10s, 30s, 60s windows are used). Since RSR is aknown-item task, the metric is effectively a ‘mGRR’ (meanGeneralize Reciprocal Rank).4. GENRE TAGGING TASKGenre information in the form of genre tags can pro-vide valuable support for users searching and browsing theInternet for video. However, much video—and especiallySPUG video—is not accurately or adequately tagged. Thistask attempts to automatically generate genre labels suchas they are used to organize videos on video platforms suchas blip.tv. Genre tagging is related to the genre classifica-tion task set out by Google as an ACM Multimedia GrandChallenge task in 2009 and 2010.The Genre Tagging task requires participants to automat-ically assign genre tags to videos using features derived fromspeech, audio, visual content or associated textual or socialinformation (Twitter social network). The tag set users arerequired to predict contains 26 genre tags.1 Each video isassociated with only one genre. Genre tags were collectedfor each episode using the API provided by blip.tv. A genretag is represented by the field categoryName in the JSONoutput provided by the API. Whitespace in the genre tagswas replaced by the underscore character (‘ ’), ampersands(‘&’) by the word ‘and’.Participants submit up to five runs in total representingfive different approaches to the task (i.e., experimental con-ditions). They are required to complete one run using ASRtranscripts only and one run including metadata. Within themetadata, we assume that user-assigned tags might alreadycontain explicit genre information. Since we are interested inalgorithms that work under the (likely) event that no tagsare available, one run is also required with metadata, butno tags. Upon request, groups with a particular focus wereexcused from specific required runs. The official evaluationmetric is MAP (Mean Average Precision).5. ACKNOWLEDGMENTSThank you to Isabelle Ferrane´ (IRIT) who served as or-ganizer of the Genre Tagging Task on the behalf of Quaero(http://www.quaero.org) and gave the task overview at theMediaEval 2011 workshop. Thank you to Thijs Verschoor(Twente) for his contribution to the RSR Task. This Medi-aEval effort received funding from EC FP7 NoE PetaMediaand Science Foundation Ireland RFP 2008.6. REFERENCES[1] P. Kelm, S. Schmiedeke, and T. Sikora. Feature-basedvideo key frame extraction for low quality videosequences. In WIAMIS ’09, pages 25–28, 2009.[2] L. Lamel and J.-L. Gauvain. Speech processing foraudio indexing. In Advances in Natural LanguageProcessing, LNCS 5221, pages 4–15. Springer, 2008.[3] M. Larson, M. Soleymani, P. Serdyukov, S. Rudinac,C. Wartena, V. Murdock, G. Friedland, R. Ordelman,and G. J. F. Jones. Automatic tagging and geotaggingin video collections and communities. In ACM ICMR’11, pages 51:1–51:8, 2011.[4] B. Liu and D. W. Oard. One-sided measures forevaluating ranked retrieval effectiveness withspontaneous conversational speech. In SIGIR ’06, pages673–674, 2006.[5] A. Stolcke, N. Coccaro, R. Bates, P. Taylor, C. VanEss-Dykema, K. Ries, E. Shriberg, D. Jurafsky,R. Martin, and M. Meteer. Dialogue act modeling forautomatic tagging and recognition of conversationalspeech. Comput. Linguist., 26:339–373, September 2000.1art, autos and vehicles, business, citizen journalism, com-edy, conferences and other events, default category,documentary, educational food and drink, gam-ing, health, literature, movies and television, mu-sic and entertainment, personal or auto-biographical,politics, religion, school and education, sports, technol-ogy, the environment, the mainstream media, travel,videoblogging, web development and sites

Overview of MediaEval 2011 rich speech retrieval task\ud
and genre tagging task

DCU Online Research Access Service

Automatic tagging and geotagging in video collections and communities.

Dialogue act modeling for automatic tagging and recognition of conversational speech.

Feature-based video key frame extraction for low quality video sequences.

One-sided measures for evaluating ranked retrieval eectiveness with spontaneous conversational speech.

Speech processing for audio indexing.

Overview of MediaEval 2011 rich speech retrieval task

and genre tagging task

The MediaEval 2011 Rich Speech Retrieval Tasks and Genre Tagging Tasks are two new tasks oered in MediaEval 2011 that are designed to explore the development of techniques for semi-professional user generated content (SPUG). They both use the same data set: the MediaEval 2010 Wild Wild Web Tagging Task (ME10WWW). The ME10WWW data set contains Creative Commons licensed video collected from blip.tv in 2009. It was created by the PetaMedia Network of Excellence (http://www.petamedia.eu) in order to test retrieval algorithms for video content as it occurs `in the wild\u27 on the Internet and, in particular, for user contributed multimedia that is embedded within a social network. In this overview paper, we repeat the essential characteristics of the data set, describe the tasks and specify how they are evaluated

Irish Universities

Overview of MediaEval 2011 rich speech retrieval task
and genre tagging task

http://doras.dcu.ie/16891/1/Larson_RSR_and_Genre_overview.pdf

Overview of MediaEval 2011 rich speech retrieval task and genre tagging task

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service

Irish Universities