The ACM SIGIR Workshop on Searching Spontaneous Conversational Speech was held as part of the 2007 ACM SIGIR Conference in Amsterdam.\ud
The workshop program was a mix of elements, including a keynote speech, paper presentations and panel discussions. This brief report describes the organization of this workshop and summarizes the discussions

Jong, Franciska de

Oard, Douglas W.

Ordelman, Roeland

Raaijmakers, Stephan

English

The ACM SIGIR Workshop on Searching Spontaneous Conversational Speech was held as part of the 2007 ACM SIGIR Conference in Amsterdam. The workshop program was a mix of elements, including a keynote speech, paper presentations and panel discussions. This brief report describes the organization of this workshop and summarizes the discussions

de Jong, Franciska M.G.

Ordelman, Roeland J.F.

NARCIS 

Searching Spontaneous Conversational Speech

University of Twente Research Information

WORKSHOP REPORTSearching Spontaneous Conversational SpeechFranciska de JongUniversity of Twente, The Netherlandsfdejong@ewi.utwente.nlDouglas W. OardUniversity of Maryland, USAoard@umd.eduRoeland OrdelmanUniversity of Twente, The Netherlandsordelman@ewi.utwente.nlStephan RaaijmakersTNO ICT, The Netherlandsstephan.raaijmakers@tno.nlJuly 27, 2007AbstractThe SIGIR Workshop on Searching Spontaneous Conversational Speech was held as partof the 2007 ACM SIGIR Conference in Amsterdam. The workshop program was a mix ofelements, including a keynote speech, paper presentations and panel discussions. This briefreport describes the organization of this workshop and summarizes the discussions.1 BackgroundNearly a decade ago, we learned from the Text Retrieval Conference’s Spoken DocumentRetrieval track that searching speech was a “solved problem.” Three factors were key to thissuccess:• Broadcast news has a ”story” structure that resembles written documents.• The redundancy present in human language means that search effectiveness held upwell over a reasonable range of transcription accuracy.• Sufficiently accurate Large-Vocabulary Continuous Speech Recognition (LVCSR) sys-tems had been built for the planned speech of news announcers.The long-term trend in speech recognition research has been toward transcription ofprogressively more challenging sources. Over the last few years, LVCSR for spontaneousconversational speech has improved to a degree where transcription accuracy comparableto what was previously found to be effective for broadcast news can now be achieved for adiverse range of sources. This has inspired a renaissance in research on search and browsingtechnology for spoken word collections in communities focused on:• Archived cultural heritage materials (e.g., interviews and parliamentary debates).• Discussion venues (e.g., business meetings and classroom instruction).• Broadcast conversations (e.g., in-studio talk shows and call-in programs).ACM SIGIR Forum 104 Vol. 41  No. 2  December 2007Test collections are being developed in individual projects around the world, includingAMI/AMIDA and CHIL (recorded meeting projects funded by the EU under the 6th Frame-work Program), IM2 (a Swiss recorded meeting project), MALACH (a NSF-funded project inthe USA working with oral history), CHoral (a cultural heritage project in the Dutch NWO-funded programme CATCH), and GALE (a DARPA-funded project in the USA workingwith broadcast conversations). Some comparative evaluation activities for speech searchtechnology are ongoing, including the Spoken Term Detection (STD) evaluation run by theNational Institute for Standards and Technology (NIST) in the USA and the Cross-LanguageEvaluation Forum’s Cross-Language Speech Retrieval track in Europe.Each of the research communities involved in the initiatives mentioned above has es-tablished venues for agenda setting and for comparison of research results. For recordedmeetings, this has included the MLMI workshops, and the NIST Rich Transcription eval-uation, and the CLEAR evaluation sponsored by NIST and CHIL. Research on culturalheritage materials has recently been reported at workshops at the 2007 conference of theAssociation for Computational Linguistics in Prague and at the 2007 User Modeling con-ference in Corfu, Greece. For broadcast conversations, the DARPA GALE program (whichincludes research teams in North America, Europe and Asia) has to date been a principalresearch venue. Cross-cutting workshops have been held before at SIGIR (in 2001) and atthe Human Language Technologies conference (in 2004), and a EU/NSF working group onspoken word archives recently identified several research issues related to the accessibilityof recorded speech [3]. The time therefore seemed right to look more broadly across theseresearch communities for potential synergies that can help to shape the information retrievalresearch agenda.2 Before the WorkshopIn the call for participation, contributions on a range of cross-cutting issues were solicited,including segmentation, content characterization, classification, exploiting multimodality,search effectiveness, interaction design, evaluation, and broader issues (e.g., applications,intellectual property, privacy). We invited fifteen experts from industry and academia toserve on the workshop’s program committee. On the basis of their recommendations, sevenpapers that together spanned the identified topics were accepted.On July 16, Technology Review published an interview with Peter Norvig (head of GoogleResearch) in which he remarked on the key role of speech retrieval technology for provid-ing access to large collections of multimedia materials [4]. Eleven days later, we met inAmsterdam to take up that challenge.3 During the WorkshopThirty researchers with a broad range of experience and expertise participated in the work-shop. The program included a mix of elements designed to maximize interaction amongparticipants from diverse backgrounds.ACM SIGIR Forum 105 Vol. 41  No. 2  December 2007Authors TitleCuendet et al. An Analysis of Sentence Segmentation Features for Broadcast News,Broadcast Conversations, and MeetingsFiscus et al. Results of the 2006 Spoken Term Detection EvaluationJones et al. Examining the Contributions of Automatic Speech Transcriptions andMetadata Sources for Searching Spontaneous Conversational SpeechKim et al. Advances in SpeechFind: CRSS-UTD Spoken Document Retrieval SystemLarson et al. Supporting Radio Archive Workflows with Vocabulary IndependentSpoken Keyword SearchOlsson Improved Measures for Predicting the Usefulness of Recognition Latticesin Ranked Utterance Retrievalvan der Werff et al. Evaluating ASR Output for Information RetrievalTable 1: Papers presented at the workshop.3.1 KeynoteMark Maybury, Executive Director of MITRE’s Information Technology Division (USA), ledoff the workshop with a keynote address. He began by summarizing the challenges posedby searching spontaneous conversational speech. Two MITRE efforts were then presentedto illustrate some of those challenges: Audio Hot-Spotting and Cross-Language AutomaticSpeech Recognition. Some promising opportunities for future research were outlined as well.The keynote session was followed by a discussant, Gareth Jones (Dublin City University,Ireland).3.2 Presentations and PanelsTable 1 briefly summarizes the seven research papers that were presented; full titles, authorlists and abstracts are available on the workshop’s Web page1, and the full text of each pa-per is available in the workshop proceedings [2]. In addition to the paper presentations, oneinvited presentation (by Doug Oard, entitled Who needs this? ) was included to stimulate dis-cussion of interactions between user needs and technical capabilities. Two panels discussionwe interleaved with the more formal presentations. The first, on “What new technologies dowe need?” included Pavel Ircing, Marijn Huijbregts, Martha Larson, and Jonathan Mamouas panelists, with Stephan Raaijmakers as moderator. The second, on “Research directions”included Ken Church, Jon Fiscus, Franciska de Jong and Mark Maybury as panelists, withDoug Oard as moderator.1http://hmi.ewi.utwente.nl/sscs/ACM SIGIR Forum 106 Vol. 41  No. 2  December 20073.3 Discussion ThemesSessions were structured to maximize opportunities for discussion, and a wide range of bothhigh-level and detailed issues were addressed. The summary below is an effort to drawtogether some of the broader themes that emerged.• Leveraging Existing Capabilities. Word error rates (WER) for planned speech(e.g., by news announcers) in studio conditions are nowadays around 10%, whereasfor conversational speech, error rates are still often as high as 30 or 40%. Variationsacross recordings are, however, often far greater than variations across words: it istherefore often more reasonable from an IR perspective to ask what fraction of thecontent can be processed well enough to support specific tasks. Supervised machinelearning techniques for topic segmentation, for example, place a greater premium onconsistency than on raw accuracy, and “bag of words” retrieval techniques are robustin the presence of occasional errors. Extractive summarization, by contrast, requiresthat consecutive words be correctly recognized (so higher error rates may yield shorterand less informative snippets), and more sophisticated analysis (e.g., the entity taggingused in question answering systems) may be even more sensitive to recognition errors.As one of our panelists observed many years ago (in a machine translation context [1]),we already have some “good applications for crummy speech recognition.” Those op-portunities deserve our attention, even as speech researchers work to further improvetheir techniques.• Getting Beyond the Laboratory. As is often the case early in the technology lifecycle, leading-edge speech technology has relied on carefully controlled benchmark eval-uations to stimulate and evaluate progress. One consequence of this is that robustnessto training-test mismatch is well understood as an important issue, but it remains anunder-researched problem. Scalability is recognized as another important challenge,but present speech processing techniques are in general quite resource-intensive. Infor-mation retrieval research, by contrast, often emphasizes both robustness and scalability.There is therefore significant potential for synergy, with speech research bringing us newcapabilities that we can productively use, and our experience bringing new applicationcontexts that can help to drive speech research in important directions.• Operational Employment. Questions about what technologies we can build are animportant first step, but our long experience with users of our technology allow us tobring another important set of questions to the table. Indexing workflows often containspecialized resources (e.g., topic inventories for use with text classification systems),and the “digital library” researchers with whom we work often pay particular attentionto how those resources will be created. Selecting and preparing domain-specific trainingdata for speech recognition would be one example of a similar task in the context ofspeech processing. Can we foster the development of a new generation of tools thatleverage the participation of domain experts in such tasks? The collections peoplework with in the real world are often quite diverse; can we provide ways for managersof such collections to use some of their materials (e.g., e-text) to improve access toothers (e.g., by allowing large scale adaptation of language models in the field ratherthan in the laboratory)? And do we have anything to say to the people who areinitially creating spoken word materials; for example, are there simple techniques (e.g.,speaker enrollment for talk show hosts) that might dramatically improve access inACM SIGIR Forum 107 Vol. 41  No. 2  December 2007some applications if the search technology could be designed to optimally leverage theresulting improvements?Ultimately, information retrieval research brings two things to the table: real collections,and real users. The recent progress on processing spontaneous conversational speech servesa complementary role, bringing us new types of collections, and hence new types of researchquestions. Together, it seems that we’re a good match!4 AcknowledgmentsThe organizers are grateful to the IST project AMIDA (http://www.amidaproject.org/)and CTIT (http://www.ctit.utwente.nl/research/sro/nice) for supporting the travelof the keynote speaker and some student participants. The work of Douglas Oard wassupported in part by NSF award IIS-0122466. The work of Franciska de Jong and Roe-land Ordelman was partly supported by the Dutch bsik-programme MultimediaN (http://www.multimedian.nl) and the EU projects MESH (http://www.mesh-ip.eu/ ), and Me-diaCampaign (http://www.media-campaign.eu/ ).References[1] K.W Church and E.H. Hovy. Good applications for crummy machine translation. Ma-chine Translation, 8(4):239–258, 1993.[2] F.M.G. de Jong, D.W. Oard, R. Ordelman, and S. Raaijmakers (eds.), editors. Proceed-ings of ACM SIGIR Workshop on Searching Spontaneous Conversational Speech. 2007.ISBN=978-90-365-2542-8.[3] J. Goldmann and S. Renals et al. Accessing the spoken word. International Journal onDigital Libraries, 5(4):287–298, 2005. ISSN=1432-5012.[4] K. Greene. The future of search: The head of google research talks about his group’sprojects. Technology Review, 2007. http://www.technologyreview.com/Biztech/19050/.ACM SIGIR Forum 108 Vol. 41  No. 2  December 2007

Accessing the spoken word.

Good applications for crummy machine translation.

The future of search: The head of google research talks about his group’s projects. Technology Review,

https://ris.utwente.nl/ws/files/6729865/2007d_sigirforum_dejong.pdf

Searching Spontaneous Conversational Speech

Abstract

Similar works

Full text

Available Versions

NARCIS

University of Twente Research Information