transLectures (Transcription and Translation of Video Lectures)
is an EU STREP project in which advanced automatic speech
recognition and machine translation techniques are being tested on large
video lecture repositories. The project began in November 2011 and will
run for three years. This paper will outline the project¿s main motivation
and objectives, and give a brief description of the two main repositories
being considered: VideoLectures.NET and poliMedia. The first results
obtained by the UPV group for the poliMedia repository will also be
provided.The research leading to these results has received funding
from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 287755. Funding was also provided by the Spanish Government (iTrans2 project, TIN2009-14511; FPI scholarship BES-2010-033005;
FPU scholarship AP2010-4349)Silvestre Cerdà, JA.; Del Agua Teba, MA.; Garcés Díaz-Munío, GV.; Gascó Mora, G.; Giménez Pastor, A.; Martínez-Villaronga, AA.; Pérez González De Martos, AM.... (2012). TransLectures. IberSPEECH 2012. 345-351. http://hdl.handle.net/10251/3729034535

Andrés Ferrer, Jesús

Civera Saiz, Jorge

Del Agua Teba, Miguel Angel

Garcés Díaz-Munío, Gonzalo Vicente

Gascó Mora, Guillem

Giménez Pastor, Adrián

Juan Císcar, Alfonso

Martínez-Villaronga, Adrià Agustí

Pérez González de Martos, Alejandro Manuel

Sanchis Navarro, José Alberto

Serrano Martínez-Santos, Nicolás

Silvestre Cerdà, Joan Albert

Spencer, Rachel Nadine

Sánchez-Cortina, Isaías

Valor Miró, Juan Daniel

RiuNet

transLecturesJ. A. Silvestre-Cerda`, M. A. del Agua, G. Garce´s, G. Gasco´, A. Gime´nez,A. Mart´ınez, A. Pe´rez, I. Sa´nchez, N. Serrano, R. Spencer, J. D. Valor,J. Andre´s-Ferrer, J. Civera, A. Sanchis, and A. JuanUniversitat Polite`cnica de Vale`ncia (UPV), Camı´ de Vera s/n, 46022 Vale`ncia{jsilvestre,mdelagua,ggarces,ggasco,agimenez,aperez,isanchez,nserranornadine,jvalor,jandres,jcivera,josanna,ajuan}@dsic.upv.eshttp://translectures.euAbstract. transLectures (Transcription and Translation of Video Lec-tures) is an EU STREP project in which advanced automatic speechrecognition and machine translation techniques are being tested on largevideo lecture repositories. The project began in November 2011 and willrun for three years. This paper will outline the project’s main motivationand objectives, and give a brief description of the two main repositoriesbeing considered: VideoLectures.NET and poliMedia. The first resultsobtained by the UPV group for the poliMedia repository will also beprovided.Keywords: language technologies, machine translation, automatic speechrecognition, massive adaptation, intelligent interaction, education, videolectures, multilingualism, accessibility, opencast matterhorn1 IntroductiontransLectures is the acronym adopted for the EU (FP7-ICT-2011-7) STREPproject entitled “Transcription and Translation of Video Lectures”. It beganon 1 November 2011 and will run until 31 October 2014. The transLecturesconsortium includes video lecture providers (users), experts in automatic speechrecognition (ASR) and machine translation (MT), and professional transcriptionand translation providers:UPV Universitat Polite`cnica de Vale`ncia, Vale`ncia, Spain.XEROX Xerox S.A.S., Grenoble, France.JSI Jozef Stefan Institute, Ljubljana, Slovenia.RWTH Rheinisch-Westfaelische Technische Hochschule, Aachen, Germany.EML European Media Laboratory GmbH, Heidelberg, Germany.DDS Deluxe Digital Studios Ltd, London, UK.The aim of this paper is to outline the motivation behind the transLecturesproject (sec. 2) and the main objectives established for it. Brief descriptions willbe provided of the poliMedia and VideoLectures.NET repositories (sec. 3), aswell as of the first results obtained by UPV for poliMedia (sec. 4).2 Motivation and project objectivesOnline multimedia repositories are growing rapidly and becoming evermore con-solidated as key knowledge assets. This is particularly true in the area of educa-tion, where large repositories of video lectures are being established on the backof increasingly available and standardized infrastructures. A well-known exam-ple of this is VideoLectures.NET, a free and open access web portal that hasalready published more than 10,000 educational videos. VideoLectures.NET is amajor player in the diffusion of the open-source Matterhorn platform currentlybeing adopted by many education institutions and organizations within what isknown as the Opencast community1.As with many other repositories, most of the video lectures available on Vide-oLectures.NET are neither transcribed nor translated, despite the clear need tomake their content accessible to speakers of different languages and people withdisabilities. Transcription and translation would also enable the incorporationof search and analysis functions, such as lecture classification, summarisationor plagiarism detection. However, a cost-effective solution that can do so to areasonable degree of accuracy has yet to be developed.The aim of the transLectures project is to do just this: to develop innovative,cost-effective solutions for producing accurate transcriptions and translationsof the lectures published on VideoLectures.NET, with their compatibility andusability across other Matterhorn-related repositories informing all design de-cisions. Our starting hypothesis is that the gap that current automatic speechrecognition (ASR) and machine translation (MT) technologies must bridge inorder to achieve acceptable results for the kind of audiovisual collections beingconsidered in the project is relatively small, and can be closed by pursuing thefollowing three scientific and technological objectives:1. Improvement of transcription and translation quality by massive adaptation.ASR has yet to reveal its full potential in the generation of acceptable tran-scriptions for large collections of audiovisual material. However, that poten-tial is within reach and relatively little further research into ASR technologyis required; rather we must learn to better exploit the wealth of knowledgewe have at hand. More precisely, through this project we hope to demon-strate that transcriptions of a reasonable quality can be obtained through themassive adaptation of general-purpose models on the basis of lecture-specificvariables, such as speaker, topic and, more importantly, time-aligned slides.Only once we have achieved acceptable transcriptions can we hope to addressthe adaptation of translation models with any degree of success.2. Improvement of transcription and translation quality by intelligent interac-tion.Massive adaptation can and will deliver substantial contributions to the im-provement of overall quality, but it is our belief that sufficiently accuratetranslations are unlikely to be achieved through fully-automated approachesalone: in order to reach the desired levels of accuracy, we must consider user1 See http://www.opencastproject.org/project/matterhorninteraction. Current user models for the transcription and translation of au-diovisual material tend to be batch-oriented. Under this model, an initialtranscription/translation is generated by the system oﬄine and then sentto the user to be post-edited manually without system participation. In ourview, these models only give satisfactory results when highly collaborativeusers are working on near-perfect system output. Otherwise, a more intelli-gent interaction model is required that saves on user supervision and allowsthe system to learn from user supervision actions dynamically. In transLec-tures, our aim is to develop innovative, truly-interactive models in which thesystem learns from and reacts to each user supervision action as and whenreceived. We are also exploring various modes of interaction, to reflect themultiple roles users can have.3. Integration into Matterhorn to enable real-life evaluation.In contrast to many past research efforts in which system prototypes areevaluated in the lab alone and are largely inapplicable to real-life settings,we will be developing tools and models for use with Matterhorn. We willtherefore be able to evaluate their usefulness using real-life data in a real-lifecontext.We are confident that the innovative solutions we develop in these three ar-eas will be deployed rapidly across many educational repositories in Europe andworldwide, allowing these portals to overcome language barriers and reach wideraudiences while supporting linguistic diversity. In transLectures, we will be test-ing our ideas on VideoLectures.NET and on a smaller repository of Spanish videolectures, poliMedia, which is also part of the Matterhorn community. For tran-scription, we are considering English and Slovenian through VideoLectures.NETand Spanish through poliMedia, with the former accounting for more than 90%of all lectures. Meanwhile, for translation we are considering the language pairs{Spanish, Slovenian} into English, and English into {French, German, Slovenian,Spanish}.3 VideoLectures.NET and poliMediaThe best proof of the effectiveness of the transLectures approach would be thecost-effective production of accurate transcriptions and translations within thecontext of one or more video lecture repositories. As discussed above, we aretesting our ideas on two case studies: VideoLectures.NET and poliMedia.VideoLectures.NET was founded in 2001 as an internally-funded project andis now run by the dedicated Centre for Knowledge Transfer in Information Tech-nologies at the Josef Stefan Institute (JSI), Ljubljana, Slovenia. It is a free andopen access repository of video lectures mostly filmed by people from JSI2 at ma-jor conferences, summer schools, workshops and promotional events from manyscientific fields. Indeed, VideoLectures.NET is being used as an educational plat-form for several EU-funded research projects3; and by various open education2 See promo video at http://videolectures.net/april2010 videolectures promo3 See http://videolectures.net/eu supportedresource organizations such as the OpenCourseWare Consortium, MIT Open-CourseWare and Open Yale Courses; as well as other scientific institutions likeCERN. In this way, VideoLectures.NET brings together high-quality educationalcontent that has been recorded to similarly high quality standards. All lectures,accompanying documents, information and links are systematically selected andclassified through the editorial process, and user comments are also taken intoaccount. Video editing is carried out in-house and is completely uncensored, inthe sense that lectures are never edited in a way that would allow content orviewer manipulation. Most lectures are also accompanied by time-aligned pre-sentation slides4.poliMedia is a recent, innovative service for the creation and distribution ofmultimedia educational content at the UPV. It is designed primarily to allowUPV professors to record their courses in video blocks lasting up to 10 minutes.As in VideoLectures.NET, the videos are accompanied by time-aligned slides.However, in contrast to VideoLectures.NET, the video recordings are filmed atspecialized studios under controlled conditions to ensure maximum recordingquality and homogeneity. Professors are filmed against a constant-colour back-ground in order to be able to post-produce presentations in which the professorappears, properly scaled, against a backdrop of the corresponding slides. Pleasevisit http://polimedia.blogs.upv.es/?lang=en for more details and examples5.transLectures is focusing on English and Slovenian video lectures from Vide-oLectures.NET and on Spanish video lectures taken from poliMedia. For an ideaas to the complexity of the task, some basic statistics have been provided inTable 1.English Slovenian Spanish TotalAuthors 6900 1347 734 8981Lectures 9720 1103 5056 15879Avg. lecture duration (min) 45 45 9 34Transcribed lectures 111 0 7 118Lectures with slides 7013 734Table 1. Basic statistics on the video lectures considered in transLectures.4 First UPV results on poliMediaThe UPV obtained its first results for poliMedia working from a set of 115 hoursof video lectures manually transcribed using the Transcriber tool [1]. From thisset, a standard partition was defined with three speaker-independent sets: train-ing, development and test. This will allow ongoing scientific evaluation through-4 e.g. http://videolectures.net/translingeu2010 uszkoreit rossi welcome5 See http://polimedia.upv.es/visor/?id=39f62a9a-4cf5-bd4e-92f3-cb34e4792a85 for abrief presentation in Spanish, with subtitles available in English.out the project. Table 2 shows the basic statistics for this standard partition.Training Development TestVideos 559 26 23Speakers 71 5 5Hours 99h 3.8h 3.4hSentences 37K 1.3K 1.1KVocabulary 28K 4.7K 4.3KRunning words 931K 35K 31KOOV words - 4.6% 5.6%Perplexity - 222 235Table 2. Statistics for the standard partition for scientific evaluations on poliMedia.The baseline UPV ASR system is based on the RWTH ASR system [2, 4]for acoustic modelling and the SRILM toolkit [6] for language modelling. TheRWTH ASR system includes state-of-the-art speech recognition technology foracoustic model training and recognition. It also includes speaker adaptation,speaker adaptive training, feature extraction for audio files, unsupervised train-ing, a finite state automata library and an efficient tree search recognizer. For itspart, the SRILM toolkit is a well-known language modelling toolkit used acrossdifferent natural language applications.The audio data was extracted and preprocessed from the videos, in order tothen extract the mel-frequency cepstral coefficients (MFCCs) [5]. Then, mono-phoneme and triphoneme acoustic models were trained by adjusting differentparameters such as the number of states, Gaussian components, leaves of thetriphoneme CART, etc. in the development set. The lexicon model was obtainedby applying phonetic transliteration to all of the training vocabulary words. Ann-gram language model was trained on the transcribed text after filtering outfunctional symbols such as punctuation marks, silence annotations, etc. Mean-while, external resources were used to enrich the in-domain language model.Specifically, we considered the linear combination of our in-domain languagemodel with a large, external out-of-domain language model computed from theGoogle n-gram corpus [3]. A single parameter λ governs the linear combination ofthe poliMedia language model and the Google n-gram model, which is optimizedin terms of perplexity on the development set.The entire system, including the acoustic, lexicon and language models, wastrained on the poliMedia training set. The ASR system parameters were opti-mized in terms of word error rate (WER) on the development set. A significantimprovement of more than 5 WER points was observed when moving from mono-phoneme to triphoneme acoustic models. Triphoneme models were inferred usingthe conventional CART model using 800 leaves. In addition, other parametersobtained to train the best acoustic model included 29 components per Gaussianmixture, 4 iterations per mixture, and 5 states per phoneme. The in-domainlanguage model was an interpolated trigram model with Kneser-Ney discount.Higher and lower order n-gram models were also assessed, but no better perfor-mance was observed.From the triphoneme ASR system, several refinements to the language modelwere evaluated. The in-domain language model trained on the poliMedia corpuswas interpolated with the out-of-domain Google n-gram corpus [3]. These twolanguage models were interpolated in order to minimize perplexity in the devel-opment set, using an approximate λ value of 0.65 for the in-domain languagemodel and of 0.35 for the out-of-domain language model. Two interpolationswere performed using different vocabulary sets, the first containing only vocab-ulary matching poliMedia (“LM Interpolation”) and a second made up of thepoliMedia vocabulary plus the 50,000 most frequent words in the Google n-gramcorpus (“Google 50K”). The final experimental results in terms of WER in thetest set are shown in Table 3.System WER OOVMonophoneme Model 44.6 5.6%Triphoneme Model 39.4 5.6%+LM Interpolation 34.6 5.6%+Google 50K Vocab 33.7 3.5%Table 3. Test-set WER for several ASR system refinements.As shown in Table 3, there is a significant improvement by 5.7 WER pointsover the triphoneme system when the language model interpolated with the“Google 50K Vocab” vocabulary set is applied. As expected, the decrease inWER is directly correlated with the number of out-of-vocabulary words (OOVs)in the test set, since the Google n-gram corpus provides a better vocabularycoverage. A similar trend is observed when comparing perplexity figures forthe triphoneme system with those observed for the “LM Interpolation” system.Specifically, perplexity drops significantly from 235 to 176 simply by interpo-lating our in-domain language model with the Google n-gram language modelcontaining poliMedia vocabulary alone.5 Concluding remarksIn this paper, we have outlined the transLectures project’s main motivation andobjectives, and given a brief description of the two video lecture repositoriesbeing considered: VideoLectures.NET and poliMedia. We have also reported thefirst ASR results obtained by UPV for poliMedia.Our current work is aimed at the application of massive adaptation tech-niques on the basis of lecture-specific knowledge and, in particular, time-alignedslides. We are also running similar experiments on the larger VideoLectures.NETrepository.Acknowledgments. The research leading to these results has received fundingfrom the European Union Seventh Framework Programme (FP7/2007-2013) un-der grant agreement no. 287755. Funding was also provided by the Spanish Gov-ernment (iTrans2 project, TIN2009-14511; FPI scholarship BES-2010-033005;FPU scholarship AP2010-4349).References1. Barras, C., Geoffrois, E., Wu, Z., Liberman, M.: Transcriber: development and useof a tool for assisting speech corpora production. Speech Communication specialissue on Speech Annotation and Corpus Tools 33(1–2) (2000)2. Lf, J., Gollan, C., Hahn, S., Heigold, G., Hoffmeister, B., Plahl, C., Rybach, D.,Schlter, R., Ney, H.: The rwth 2007 tc-star evaluation system for european englishand spanish. In: Proc. of Interspeech. pp. 2145–2148 (2007)3. Michel, J.B., et al.: Quantitative analysis of culture using millions of digitized books.Science 331(6014), 176–182.4. Rybach, D., Gollan, C., Heigold, G., Hoffmeister, B., Lf, J., Schlter, R., Ney, H.:The rwth aachen university open source speech recognition system. In: Proc. ofInterspeech. pp. 2111–2114 (2009)5. Sahidullah, M., Saha, G.: Design, analysis and experimental evaluation of blockbased transformation in MFCC computation for speaker recognition 54(4), 543–565(2012)6. Stolcke, A.: SRILM - An Extensible Language Modeling Toolkit. In: Proc. of ICSLP(2002)

TransLectures

https://riunet.upv.es/bitstream/handle/10251/37290/1209IberSpeech.pdf?sequence=1&isAllowed=y

TransLectures

Abstract

Similar works

Full text

Available Versions

RiuNet