We present a pair of prototype tools that enable users to mark up video with annotations and later explore related materials using Semantic Web and Linked Data approaches. The �first tool helps academics preparing Open University course materials to mark up videos with information about the subject matter and audio-visual content. The second tool enables users, such as students or academics, to find video and other materials relevant to their study

Lambert, David

Yu, Hong Qing

English

We present a pair of prototype tools that enable users to mark up video with annotations and later explore related materials using Semantic Web and Linked Data approaches. The first tool helps academics preparing Open University course materials to mark up videos with information about the subject matter and audio-visual content. The second tool enables users, such as students or academics, to find video and other materials relevant to their study

Lambert, Dave

Southampton (e-Prints Soton)

Linked Data Based Video Annotation andBrowsing for Distance LearningDave Lambert and Hong Qing YuKnowledge Media InstituteThe Open UniversityMilton Keynes, United KingdomAbstract. We present a pair of prototype tools that enable users tomark up video with annotations and later explore related materialsusing Semantic Web and Linked Data approaches. The rst tool helpsacademics preparing Open University course materials to mark up videoswith information about the subject matter and audio-visual content. Thesecond tool enables users, such as students or academics, to nd videoand other materials relevant to their study.1 IntroductionThe Open University (OU) is a leading distance learning institute, and the largestprovider of tertiary education in the United Kingdom. It makes innovative use ofmultimedia resources in its teaching materials, and is famous for its televisionprogrammes, both those designed for its students, and those targeted at thegeneral population. The OU course programme transcripts alone occupy 92 metresof shelving in the university's library, and the associated video material representsa substantial investment and resource, and the university is considering how tobetter exploit it for:{ academics preparing new courses, who can survey a larger amount of materialmore quickly and eectively{ students who wish to follow a topic of interest beyond the prescribed materials{ serendipitous use by anyoneWe have built two tools to investigate how using Semantic Web and Linked OpenData (LOD) principles [1] could aid search and reuse. We cover the background andmotivation behind the Open University's development of such tools (Section 2),then introduce the annotation tool `Annomation' and ontology (Section 3), thebrowser software `SugarTube' (Section 4), and the Linked Data sets and servicesthey use (Section 5). We compare our work with related systems and approaches(Section 6) and conclude (Section 7).2 BackgroundThe Open University is one of the world's largest providers of tertiary education: ithas over 200,000 students currently enrolled on courses at bachelors, masters, andPhD level. Since its founding in 1969, the OU has pioneered innovative teachingmethods. Most famously, it has produced a wide range of television programmes,often in collaboration with the British Broadcasting Corporation (BBC), both forits own students and the general population. Today, the OU continues to use newmultimedia technologies, having moved from television through CD, DVD andnow online delivery mechanisms.1 The OU has also developed online teachingfacilities like course forums, multi-participant audio environments for languagelearning, and is now moving to make greater use of the Semantic Web.Although supported by local, face-to-face tutorial sessions, Open Universitycourses are primarily taught at a distance and from the beginning they haveused multimedia content to engage and educate students. Sta preparing anOU course will work with BBC archivists to nd appropriate video material toillustrate certain points of the course. This is labour intensive: a text databaseof available footage (principally from the OU and BBC archives) is searched toidentify dozens of hours of video which may be of interest. The course teamwill then view this video, manually noting sections of interest. When the nalset of video segments are identied for the course (typically an hour or two ofvideo will result), the information on unused material is lost. The OU and theBBC are currently making large investments in Semantic Web technologies, andthe cataloguing of video material is an ideal candidate. We therefore developedprototype systems that enable OU course sta to mark up video in a SemanticWeb or Linked Data style, and to subsequently navigate it and augment it withadditional materials.3 AnnotationOur rst tool is Annomation,2 which handles the input of annotations from users.Annomation provides a simple Web browser interface (see Figure 1) split intofour main areas:{ a video player (top-left){ a list of videos, and existing annotations for the current video (top-right){ a set of controls for the video player, and input widgets to enter theannotations (across the centre){ a set of panels to aid in nding suitable linked data annotations (at bottom)In operation, the user rst selects which of the videos to annotate. The videosof interest at the moment are transcoded from the original material by a BBCarchivist at the request of the course team, and we serve them directly fromthe Annomation server, but the architecture supports using video served fromanywhere on the Web. Having selected a video, the user will be presented with atimeline of any annotations previously made by herself and others: the user canskip to particular instants or durations noted in the annotations. The most basic1 http://open.edu/itunes/2 http://annomation.open.ac.uk/annomationway to create an annotation is simply to pause the video at the appropriate point,enter a duration if appropriate, and add a Semantic Web/Linked Data URI. Thisis sent to the server, and the annotation recorded. Annotations can be optionallymarked as applying to something visible in the video, audible in the soundtrack,or to the conceptual subject matter of the video. Since nding appropriate URIsis non-trivial, the fourth part of the interface is dedicated to helping the user ndthem. The details of the data sources are discussed in Section 5, but include theDewey Decimal and Library of Congress classications, geographic searches usinginteractive maps, suggestions based on natural language processing and entityextraction, and internal Open University classications. The user can also perusethe URIs that have been previously used on other videos or by other users.Fig.1. The Annomation interfaceWe implemented the system (Figure 2) in the Clojure language, using Sesameas an RDF quad store, and RDF2Go as an abstraction over the store. Annomationprovides programmatic APIs in the form of a SPARQL end point for querying, andRESTful interfaces for adding and removing annotations, and exploring existingones. It is our intent that Annomation's RDF become part of the Linked OpenData cloud. In the client we use Javascript with the Yahoo YUI library and theJQuery plugin RDFQuery, and FlowPlayer for video playback. We use OpenIdfor user identication and authentication.Fig.2. Annomation architecture3.1 Data representationAt the heart of our approach is the Linked Data that is created and stored inAnnomation, and searched by SugarTube. Users' annotations are recorded in anRDF quad store, with each user having their own context (a named graph) tofacilitate privacy and provenance. The data sets related to user accounts andvideos are also stored in the same store, in a management context. Most entitiesdescribed in the Annomation store, including videos, users, and annotations,are assigned generated, globally unique URLs so they can be named as partof the Linked Open Data cloud. We reused several existing RDF Schema [2]vocabularies, and introduced a few new terms specically for Annomation. Theexisting vocabularies we reused are:{ Friend-of-a-Friend (FOAF) [3], for identifying users and their accounts.{ The Timeline ontology [4], for identifying temporal instants and durationson the video timeline.{ Dublin Core, in its RDFS form,3 for meta-data such as a video's title, andthe author and creation time of each annotation.To tie together the data described using these vocabularies, we designed our ownsmall ontology:@prefix : <http://annomation.open.ac.uk/annomation/ns/annomation#> .@prefix tl: <http://purl.org/NET/c4dm/timeline.owl#> .@prefix owltime: <http://www.w3.org/TR/owl-time/#> .:Video a rdfs:Class .:Annotation a rdfs:Class .:Mood a rdfs:Class .:topic a :Mood ;rdfs:label "Topic of conversation"@en .:inVideo a :Mood ;rdfs:label "In video stream"@en .:inAudio a :Mood ;rdfs:label "In audio stream"@en .:annotates a rdfs:Property ;rdfs:domain :Annotation ; rdfs:range owltime:TemporalEntity .:reference a rdfs:Property ;rdfs:domain :Annotation ; rdfs:range rdfs:Resource .:mood a rdfs:Property ;rdfs:domain :Annotation ; rdfs:range :Mood .We allocate a unique URI to every video, user, time, and annotation. In this way,we can identify a video URI as a Video, reference that URI from a TemporalEntityin the timeline ontology, and then create an Annotation to tie together the timewith its associated video, the `mood' of the annotation and the annotation itself.As an illustration, the annotation about the tank seen in Figure 1) is::fa031dd6 -89cb-49ec-b9d9 -43e9f6ad7b65 a anno:Annotation ;anno:annotates :dd5e5b59 -7f83-4b30-b1a4 -15f16781d9a2 ;anno:reference <http://dbpedia.org/resource/Tank> ;anno:mood anno:topic , anno:inVideo , anno:inAudio ;dc:creator :ae8d5a6d -f3d3 -4b68-be3d -47160a26d497 ;dc:date "2010-09-16T10:20:15Z"^^xsd:dateTime .:dd5e5b59 -7f83-4b30-b1a4 -15f16781d9a2 a tl:Interval ;tl:onTimeline :c2b91ed5 -ba09 -48ff-97f6-1fe0e0bb9c12 ;tl:at "PT01M00.000S"^^xsd:duration ;tl:duration "PT10.000S"^^xsd:duration .The dc:creator is the Annomation account of the user that made the annotation,and this in stored as FOAF and linked to the OpenId URI.4 BrowsingTo search the annotated videos, and explore related materials through LinkedData approaches, we developed the SugarTube browser (Semantics Used to Get3 http://www.dublincore.org/schemas/rdfs/Annotated video Recording). The SugarTube interface and system architecture isillustrated in Figure 3. The SugarTube architecture has two layers: the applicationlayer, responsible for presentation and interaction, and the semantic data layerwhich nds and integrates data that has been published in the LOD cloudby dierent organisations, including annotation data from Annomation. TheSemantic Web data is obtained through RESTful services and SPARQL endpoints.The application layer allows the user to query and navigate the video searchresults using the semantic data (e.g. related knowledge, concepts, videos, andlearning materials).Fig.3. SugarTube browser (top) and architecture (bottom)SugarTube is designed to help the learner in navigating learning resources usingLinked Data, so its rst task is to connect the learner with appropriate conceptsin the Semantic Web. It provides various search functions including those basedon general keywords, or specic names of events, places, or people. For example,if learner types in a place name \Berlin", SugarTube will invoke RESTful servicesprovided by GeoNames and DBpedia4 to get semantically relevant concepts anddata about the city of Berlin such as the facts that it is the capital city of thecountry of Germany, has latitude of 52.52437 and longitude of 13.41053, and thatmore RDF data can be found at the URL http://dbpedia.org/page/Berlin.4 These and other data sources will be discussed in Section 5.In addition to specic searching by keywords, the learner can use more othersearching methods: through geographical maps, entity recognition of naturallanguage text supplied directly or from a URI, or by time. SugarTube implementsthe time search using the WorldHistory service, allowing the user to nd relatedhistorical events and browse them in a graphical timeline format.Having found relevant conceptual URIs in the Linked Data cloud, SugarTubecan then allow the user to navigate the links. Of course, SugarTube can queryAnnomation for videos that are tagged using the URIs of interest, but it can alsond related video material through YouTube, BBC and OpenLearn services. Theuser can be prompted with related concepts, Web documents, and other resourcesby SugarTube's analysis of neighbouring point LOD cloud, including RESTfulRDF services. For example, from \Berlin", the Sindice Web service can providefurther Semantic Web links connected to the related terms \West Berlin", \EastBerlin" and \Berlin wall".5 Data sourcesBoth Annomation and SugarTube make use of several services and data sourceson the Web to help the user annotate and nd video. Presently, Semantic Web(RDF) sources include:{ Dewey Decimal and Library of Congress classications: The top level DeweyDecimal Classication (covering the rst three digits of a Dewey number)has been published in RDF form by the Online Computer Library Center,5and the resulting taxonomy is presented to the annotater as a browsabletree. The Library of Congress has published its entire classication systemin RDF,6 but this is much too large to present directly to the user. Instead,Annomation provides an interface to the Library of Congress keyword searchservice, which returns suitable RDF for the user to choose from.{ GeoNames7 [5]: The GeoNames API is used to identify named locationsaccording using keyword search, or perform reverse lookup to nd namedlocations in a vicinity. The results are used to provide position and categoryinformation to allow visual display on maps and URIs to use for annotation.{ OU Linked Data8: Currently under development, this repository will extract,interlink, and publish in the LOD cloud material previously available invarious disconnected institutional repositories of the Open University.{ Sindice9 [6]: A semantic search engine, Sindice crawls and collates theSemantic Web (including microformats), and based on this provides servicesincluding keyword searching for linked data, and accessing cached fragmentsof the Semantic Web.5 http://dewey.info/6 http://id.loc.gov/authorities/7 http://www.geonames.org/8 http://data.open.ac.uk/9 http://sindice.com/{ Zemanta10: A service aimed at helping bloggers augment their posts withlinks and images, Zemanta analyses natural language text to identify variousconcepts and named entities, returning URIs to Linked Data hosted by,amongst others, DBpedia and Freebase.In reusing the datasets and services listed above, we make use of another coupleof ontologies: the Simple Knowledge Organization System (SKOS) [7], and theWGS84-based RDF vocabulary for geographical data [8]. We also use severalnon-semantic services including:{ BBC11: machine readable programme information.{ Maps: interactive Ajax mapping services from Google12 and Yahoo.13{ OpenLearn14: Free access to a subset of the Open University's course materi-als, along with metadata.{ YouTube data services15: Used in SugarTube to nd videos by keyword.6 Related workThere is signicant prior work on video annotation tools, diering in twodimensions from the work presented here: they are either fat-client softwarerather than Web browser based, or do not create Linked Data annotations. Animportant early system was Vannotea [9] which relied on a dedicated clientapplication, but enabled collaborative annotation, but the annotations were notin a Semantic Web style. M-OntoMat-Annotizer [10] did use Semantic Webannotations, linking them to annotations embedded using MPEG-7, but againwas a fat client.Videos are traditionally searched by syntactic matching mechanism (e.g. [11]),but recently, with more videos being annotated or tagged using in a LinkedData manner, researchers have begun looking at more Semantic Web orientedfashion. The two major approaches are the semantic indexing process and naturallanguage analysis process. The indexing process assumes the video annotationsare made from a xed vocabulary that does not often change (e.g. [12]). Althoughthis process can be ecient, the xed ontology can introduce a gap betweenuser's knowledge and indexed annotations. In an educational environment, videoswill often be annotated by dierent teachers or students who will each wishto apply dierent annotations to the same video in the context of dierentcourses, groups, and key points. The natural language analysis process focusesmore on adding semantics tags to the user's search input (e.g. [13]). However,most of these approaches require some machine-learning mechanism to assistto dynamically add tags, usually restricting its application to smaller, closeddomains of discourse.10 http://www.zemanta.com/11 http://backstage.bbc.co.uk/data/BbcWebApi12 http://code.google.com/apis/maps/index.html13 http://developer.yahoo.com/maps/14 http://openlearn.open.ac.uk/15 http://code.google.com/apis/youtube/overview.html7 ConclusionThe Open University has a need to better catalogue and nd material from itsarchives to reuse in teaching. More broadly, as the Internet makes it easier forstudents to nd and use video as part of their education, it is important thatwe can provide tools to help them do so. We presented two tools that allowedacademics to annotate educational video resources in a Linked Data approach,and to subsequently nd appropriate video materials by navigating the Web ofLinked Data.The university will be trialing the Annomation tool in the production ofa new history course, with professional historians providing the annotations.Future work will integrate other data sources such as video metadata includingprovenance and rights management, programme transcripts where available,address active collaboration in annotation, and add interfaces for dierent classesof users.AcknowledgementsThis research was funded by the Open University's Open Broadcast Unit, andEU FP7 projects SOA4All and NoTube.References1. Berners-Lee, T.: Linked Data (2006) Online at http://www.w3.org/DesignIssues/LinkedData.html.2. Brickley, D., Guha, R.: RDF Vocabulary Description Language 1.0: RDF Schema.W3C recommendation, World Wide Web Consortium (W3C) (February 2004)3. Brickley, D., Miller, L.: FOAF vocabulary specication 0.91 (2007) Online athttp://xmlns.com/foaf/spec/.4. Raimond, Y., Abdallah, S.: The Timeline Ontology (2007) Online at http://purl.org/NET/c4dm/timeline.owl#.5. Vatant, B., Wick, M.: Geonames Ontology (2006) Online at http://www.geonames.org/ontology/.6. Tummarello, G., Delbru, R., Oren, E.: Weaving the open linked data. In: Proceedingsof the 6th International Semantic Web Conference. Volume 4825 of Lecture Notesin Computer Science., Springer (2007) 552{5657. Miles, A., Bechhofer, S.: SKOS Reference. W3C recommendation (August 2009)Online at http://www.w3.org/TR/skos-reference.8. Brickley, D.: Basic Geo (WGS84 lat/long) Vocabulary (2004) Online at http://www.w3.org/2003/01/geo/.9. Hunter, J., Schroeter, R., Koopman, B., Henderson, M.: Using the Semantic Gridto Build Bridges Between Museums and Indigenous Communities. In: Proceedingsof the GGF11|Semantic Grid Applications Workshop. (June 2004)10. Bloehdorn, S., Petridis, K., Saatho, C., Simou, N., Tzouaras, V., Avrithis, Y.,Handschuh, S., Kompatsiaris, Y., Staab, S., Strintzis, M.: Semantic Annotation ofImages and Videos for Multimedia Analysis. In: Proceedings of the 2nd EuropeanSemantic Web Conference. Volume 3532 of Lecture Notes in Computer Science.,Springer (2005)11. Wei, S., Zhao, Y., Zhu, Z., Liu, N.: Multimodal fusion for video search reranking.IEEE Transactions on Knowledge and Data Engineering 22 (2010) 1191{119912. Hakeem, A., Lee, M.W., Javed, O., Haering, N.: Semantic video search using naturallanguage queries. In: MM '09: Proceedings of the seventeen ACM internationalconference on Multimedia, New York, NY, USA, ACM (2009) 605{60813. Sebastine, S.C., Thuraisingham, B.M., Prabhakaran, B.: Semantic web for contentbased video retrieval. In: Proceedings of the 3rd IEEE International Conferenceon Semantic Computing (ICSC 2009), 14-16 September 2009, Berkeley, CA, USA,IEEE Computer Society (2009) 103{108

Basic Geo (WGS84 lat/long) Vocabulary

FOAF vocabulary speci 0.91

Geonames Ontology

Linked Data

RDF Vocabulary Description Language 1.0: RDF Schema. W3C recommendation, World Wide Web Consortium

Semantic Annotation of Images and Videos for Multimedia Analysis. In:

Semantic video search using natural language queries. In:

Semantic web for content based video retrieval. In:

SKOS Reference. W3C recommendation

The Timeline Ontology

Using the Semantic Grid to Build Bridges Between Museums and Indigenous Communities. In:

Weaving the open linked data. In:

Linked Data Based Video Annotation and Browsing for Distance Learning

Open Research Online

Linked Data based video annotation and browsing for distance learning

Abstract. We present a pair of prototype tools that enable users to mark up video with annotations and later explore related materials using Semantic Web and Linked Data approaches. The first tool helps academics preparing Open University course materials to mark up videos with information about the subject matter and audio-visual content. The second tool enables users, such as students or academics, to find video and other materials relevant to their study. 

Dave Lambert

Hong Qing Yu

CiteSeerX

http://oro.open.ac.uk/28901/1/semhe10_lambert-yu.pdf

Linked Data based video annotation and browsing for distance learning

Abstract

Similar works

Full text

Available Versions

Southampton (e-Prints Soton)

Open Research Online

CiteSeerX