MAC-REALM: A video content feature extraction and modelling framework

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.A consequence of the ‘data deluge’ is the exponential increase in digital video footage, while the ability to find relevant video clips diminishes. Traditional text based search engines are no longer optimal for searching, as they cannot provide a granular search of the content inside video footage. To be able to search the video in a content based manner, the content features of the video need to be extracted and modelled into a content model, which can then act as a searchable proxy for the video content. This thesis focuses on the extraction of syntactic and semantic content features and content modelling, using machine driven processes, with either little or no user interaction. Our abstract framework design extracts syntactic and semantic content features and compiles them into an integrated content model. The framework integrates a four plane strategy that consists of a pre-processing plane that removes redundant data and filters the media to improve the feature extraction properties of the media; a syntactic feature extraction plane that extracts low level syntactic feature and mid-level syntactic features that have semantic attributes; a semantic relationship analysis and linkage plane, where the spatial and temporal relationships of all the content features are defined, and finally a content modelling stage where the syntactic and semantic content features are integrated into a content model. Each of the four planes can be split into three layers namely, the content layer, where the content to be processed is stored; the application layer, where the content is converted into content descriptions, and the MPEG-7 layer, where content descriptions are serialised. Using MPEG-7 standards to produce the content model will provide wide-ranging interoperability, while facilitating granular multi-content type searches. The framework is aiming to ‘bridge’ the semantic gap, by integrating the syntactic and semantic content features from extraction through to modelling. The design of the framework has been implemented into a prototype called MAC-REALM, which has been tested and evaluated for its effectiveness to extract and model content features. Conclusions are drawn about the research output as a whole and whether they have met the objectives. Finally, future work is presented on how concept detection and crowd sourcing can be used with MAC-REALM

Security strategies in genomic files

There are new mechanisms to sequence and process the genomic code, discovering thus diagnostic tools and treatments. The file for a sequenced genome can reach hundreds of gigabytes. Thus, for further studies, we need new means to compress the information and a standardized representation to simplify the development of new tools. The ISO standardization group MPEG has used its expertise in compressing multimedia content to compress genomic information and develop its ´MPEG-G standard’. Given the sensitivity of the data, security is a major identified requirement. This thesis proposes novel technologies that assure the security of both the sequenced data and its metadata. We define a container-based file format to group data, metadata, and security information at the syntactical level. It includes new features like grouping multiple results in a same file to simplify the transport of whole studies. We use the granularity of the encoder’s output to enhance security. The information is represented in units, each dedicated to a specific region of the genome, which allows to provide encryption and signature features on a region base. We analyze the trade-off between security and an even more fine-grained approach and prove that apparently secure settings can be insecure: if the file creator may encrypt only specific elements of a unit, cross-checking unencrypted information permits to infer encrypted content. Most of the proposals for MPEG-G coming from other research groups and companies focused on data compression and representation. However, the need was recognized to find a solution for metadata encoding. Our proposal was included in the standard: an XML-based solution, separated in a core specification and extensions. It permits to adapt the metadata schema to the different genomic repositories' frameworks, without importing requirements from one framework to another. To simplify the handling of the resulting metadata, we define profiles, i.e. lists of extensions that must be present in a given framework. We use XML signature and XML encryption for metadata security. The MPEG requirements also concern access rules. Our privacy solutions limit the range of persons with access and we propose access rules represented with XACML to convey under which circumstances a user is granted access to a specific action among the ones specified in MPEG-G's API, e.g. filtering data by attributes. We also specify algorithms to combine multiple rules by defining default behaviors and exceptions. The standard’s security mechanisms protect the information only during transport and access. Once the data is obtained, the user could publish it. In order to identify leakers, we propose an algorithm that generates unique, virtually undetectable variations. Our solution is novel as the marking can be undone (and the utility of the data preserved) if the corresponding secret key is revealed. We also show how to combine multiple secret keys to avoid collusion. The API retained for MPEG-G considers search criteria not present in the indexing tables, which highlights shortcomings. Based on the proposed MPEG-G API we have developed a solution. It is based on a collaboration framework where the different users' needs and the patient's privacy settings result in a purpose-built file format that optimizes query times and provides privacy and authenticity on the patient-defined genomic regions. The encrypted output units are created and indexed to optimize query times and avoid rarely used indexing fields. Our approach resolves the shortcomings of MPEG-G's indexing strategy. We have submitted our technologies to the MPEG standardization committee. Many have been included in the final standard, via merging with other proposals (e.g. file format), discussion (e.g. security mechanisms), or direct acceptance (e.g. privacy rules).Hi han nous mètodes per la seqüenciació i el processament del codi genòmic, permetent descobrir eines de diagnòstic i tractaments en l’àmbit mèdic. El resultat de la seqüenciació d’un genoma es representa en un fitxer, que pot ocupar centenars de gigabytes. Degut a això, hi ha una necessitat d’una representació estandarditzada on la informació és comprimida. Dins de la ISO, el grup MPEG ha fet servir la seva experiència en compressió de dades multimèdia per comprimir dades genòmiques i desenvolupar l'estàndard MPEG-G, sent la seguretat un dels requeriments principals. L'objectiu de la tesi és garantir aquesta seguretat (encriptant, firmant i definint regles d¿ accés) tan per les dades seqüenciades com per les seves metadades. El primer pas és definir com transportar les dades, metadades i paràmetres de seguretat. Especifiquem un format de fitxer basat en contenidors per tal d'agrupar aquets elements a nivell sintàctic. La nostra solució proposa noves funcionalitats com agrupar múltiples resultats en un mateix fitxer. Pel que fa la seguretat de dades, la nostra proposta utilitza les propietats de la sortida del codificador. Aquesta sortida és estructurada en unitats, cadascuna dedicada a una regió concreta del genoma, permetent una encriptació i firma de dades específica a la unitat. Analitzem el compromís entre seguretat i un enfocament de gra més fi demostrant que configuracions aparentment vàlides poden no ser-ho: si es permet encriptar sols certes sub-unitats d'informació, creuant els continguts no encriptats, podem inferir el contingut encriptat. Quant a metadades, proposem una solució basada en XML separada en una especificació bàsica i en extensions. Podem adaptar l'esquema de metadades als diferents marcs de repositoris genòmics, sense imposar requeriments d’un marc a un altre. Per simplificar l'ús, plantegem la definició de perfils, és a dir, una llista de les extensions que han de ser present per un marc concret. Fem servir firmes XML i encriptació XML per implementar la seguretat de les metadades. Les nostres solucions per la privacitat limiten qui té accés a les dades, però no en limita l’ús. Proposem regles d’accés representades amb XACML per indicar en quines circumstàncies un usuari té dret d'executar una de les accions especificades a l'API de MPEG-G (per exemple, filtrar les dades per atributs). Presentem algoritmes per combinar regles, per tal de poder definir casos per defecte i excepcions. Els mecanismes de seguretat de MPEG-G protegeixen la informació durant el transport i l'accés. Una vegada l’usuari ha accedit a les dades, les podria publicar. Per tal d'identificar qui és l'origen del filtratge de dades, proposem un algoritme que genera modificacions úniques i virtualment no detectables. La nostra solució és pionera, ja que els canvis es poden desfer si el secret corresponent és publicat. Per tant, la utilitat de les dades és mantinguda. Demostrem que combinant varis secrets, podem evitar col·lusions. L'API seleccionada per MPEG-G, considera criteris de cerca que no són presents en les taules d’indexació. Basant-nos en aquesta API, hem desenvolupat una solució. És basada en un marc de col·laboració, on la combinació de les necessitats dels diferents usuaris i els requeriments de privacitat del pacient, es combinen en una representació ad-hoc que optimitza temps d’accessos tot i garantint la privacitat i autenticitat de les dades. La majoria de les nostres propostes s’han inclòs a la versió final de l'estàndard, fusionant-les amb altres proposes (com amb el format del fitxer), demostrant la seva superioritat (com amb els mecanismes de seguretat), i fins i tot sent acceptades directament (com amb les regles de privacitat)

Highly efficient low-level feature extraction for video representation and retrieval.

PhDWitnessing the omnipresence of digital video media, the research community has raised the question of its meaningful use and management. Stored in immense multimedia databases, digital videos need to be retrieved and structured in an intelligent way, relying on the content and the rich semantics involved. Current Content Based Video Indexing and Retrieval systems face the problem of the semantic gap between the simplicity of the available visual features and the richness of user semantics. This work focuses on the issues of efficiency and scalability in video indexing and retrieval to facilitate a video representation model capable of semantic annotation. A highly efficient algorithm for temporal analysis and key-frame extraction is developed. It is based on the prediction information extracted directly from the compressed domain features and the robust scalable analysis in the temporal domain. Furthermore, a hierarchical quantisation of the colour features in the descriptor space is presented. Derived from the extracted set of low-level features, a video representation model that enables semantic annotation and contextual genre classification is designed. Results demonstrate the efficiency and robustness of the temporal analysis algorithm that runs in real time maintaining the high precision and recall of the detection task. Adaptive key-frame extraction and summarisation achieve a good overview of the visual content, while the colour quantisation algorithm efficiently creates hierarchical set of descriptors. Finally, the video representation model, supported by the genre classification algorithm, achieves excellent results in an automatic annotation system by linking the video clips with a limited lexicon of related keywords

Security strategies in genomic files

There are new mechanisms to sequence and process the genomic code, discovering thus diagnostic tools and treatments. The file for a sequenced genome can reach hundreds of gigabytes. Thus, for further studies, we need new means to compress the information and a standardized representation to simplify the development of new tools. The ISO standardization group MPEG has used its expertise in compressing multimedia content to compress genomic information and develop its ´MPEG-G standard’. Given the sensitivity of the data, security is a major identified requirement. This thesis proposes novel technologies that assure the security of both the sequenced data and its metadata. We define a container-based file format to group data, metadata, and security information at the syntactical level. It includes new features like grouping multiple results in a same file to simplify the transport of whole studies. We use the granularity of the encoder’s output to enhance security. The information is represented in units, each dedicated to a specific region of the genome, which allows to provide encryption and signature features on a region base. We analyze the trade-off between security and an even more fine-grained approach and prove that apparently secure settings can be insecure: if the file creator may encrypt only specific elements of a unit, cross-checking unencrypted information permits to infer encrypted content. Most of the proposals for MPEG-G coming from other research groups and companies focused on data compression and representation. However, the need was recognized to find a solution for metadata encoding. Our proposal was included in the standard: an XML-based solution, separated in a core specification and extensions. It permits to adapt the metadata schema to the different genomic repositories' frameworks, without importing requirements from one framework to another. To simplify the handling of the resulting metadata, we define profiles, i.e. lists of extensions that must be present in a given framework. We use XML signature and XML encryption for metadata security. The MPEG requirements also concern access rules. Our privacy solutions limit the range of persons with access and we propose access rules represented with XACML to convey under which circumstances a user is granted access to a specific action among the ones specified in MPEG-G's API, e.g. filtering data by attributes. We also specify algorithms to combine multiple rules by defining default behaviors and exceptions. The standard’s security mechanisms protect the information only during transport and access. Once the data is obtained, the user could publish it. In order to identify leakers, we propose an algorithm that generates unique, virtually undetectable variations. Our solution is novel as the marking can be undone (and the utility of the data preserved) if the corresponding secret key is revealed. We also show how to combine multiple secret keys to avoid collusion. The API retained for MPEG-G considers search criteria not present in the indexing tables, which highlights shortcomings. Based on the proposed MPEG-G API we have developed a solution. It is based on a collaboration framework where the different users' needs and the patient's privacy settings result in a purpose-built file format that optimizes query times and provides privacy and authenticity on the patient-defined genomic regions. The encrypted output units are created and indexed to optimize query times and avoid rarely used indexing fields. Our approach resolves the shortcomings of MPEG-G's indexing strategy. We have submitted our technologies to the MPEG standardization committee. Many have been included in the final standard, via merging with other proposals (e.g. file format), discussion (e.g. security mechanisms), or direct acceptance (e.g. privacy rules).Hi han nous mètodes per la seqüenciació i el processament del codi genòmic, permetent descobrir eines de diagnòstic i tractaments en l’àmbit mèdic. El resultat de la seqüenciació d’un genoma es representa en un fitxer, que pot ocupar centenars de gigabytes. Degut a això, hi ha una necessitat d’una representació estandarditzada on la informació és comprimida. Dins de la ISO, el grup MPEG ha fet servir la seva experiència en compressió de dades multimèdia per comprimir dades genòmiques i desenvolupar l'estàndard MPEG-G, sent la seguretat un dels requeriments principals. L'objectiu de la tesi és garantir aquesta seguretat (encriptant, firmant i definint regles d¿ accés) tan per les dades seqüenciades com per les seves metadades. El primer pas és definir com transportar les dades, metadades i paràmetres de seguretat. Especifiquem un format de fitxer basat en contenidors per tal d'agrupar aquets elements a nivell sintàctic. La nostra solució proposa noves funcionalitats com agrupar múltiples resultats en un mateix fitxer. Pel que fa la seguretat de dades, la nostra proposta utilitza les propietats de la sortida del codificador. Aquesta sortida és estructurada en unitats, cadascuna dedicada a una regió concreta del genoma, permetent una encriptació i firma de dades específica a la unitat. Analitzem el compromís entre seguretat i un enfocament de gra més fi demostrant que configuracions aparentment vàlides poden no ser-ho: si es permet encriptar sols certes sub-unitats d'informació, creuant els continguts no encriptats, podem inferir el contingut encriptat. Quant a metadades, proposem una solució basada en XML separada en una especificació bàsica i en extensions. Podem adaptar l'esquema de metadades als diferents marcs de repositoris genòmics, sense imposar requeriments d’un marc a un altre. Per simplificar l'ús, plantegem la definició de perfils, és a dir, una llista de les extensions que han de ser present per un marc concret. Fem servir firmes XML i encriptació XML per implementar la seguretat de les metadades. Les nostres solucions per la privacitat limiten qui té accés a les dades, però no en limita l’ús. Proposem regles d’accés representades amb XACML per indicar en quines circumstàncies un usuari té dret d'executar una de les accions especificades a l'API de MPEG-G (per exemple, filtrar les dades per atributs). Presentem algoritmes per combinar regles, per tal de poder definir casos per defecte i excepcions. Els mecanismes de seguretat de MPEG-G protegeixen la informació durant el transport i l'accés. Una vegada l’usuari ha accedit a les dades, les podria publicar. Per tal d'identificar qui és l'origen del filtratge de dades, proposem un algoritme que genera modificacions úniques i virtualment no detectables. La nostra solució és pionera, ja que els canvis es poden desfer si el secret corresponent és publicat. Per tant, la utilitat de les dades és mantinguda. Demostrem que combinant varis secrets, podem evitar col·lusions. L'API seleccionada per MPEG-G, considera criteris de cerca que no són presents en les taules d’indexació. Basant-nos en aquesta API, hem desenvolupat una solució. És basada en un marc de col·laboració, on la combinació de les necessitats dels diferents usuaris i els requeriments de privacitat del pacient, es combinen en una representació ad-hoc que optimitza temps d’accessos tot i garantint la privacitat i autenticitat de les dades. La majoria de les nostres propostes s’han inclòs a la versió final de l'estàndard, fusionant-les amb altres proposes (com amb el format del fitxer), demostrant la seva superioritat (com amb els mecanismes de seguretat), i fins i tot sent acceptades directament (com amb les regles de privacitat)

Security strategies in genomic files

There are new mechanisms to sequence and process the genomic code, discovering thus diagnostic tools and treatments. The file for a sequenced genome can reach hundreds of gigabytes. Thus, for further studies, we need new means to compress the information and a standardized representation to simplify the development of new tools. The ISO standardization group MPEG has used its expertise in compressing multimedia content to compress genomic information and develop its ´MPEG-G standard’. Given the sensitivity of the data, security is a major identified requirement. This thesis proposes novel technologies that assure the security of both the sequenced data and its metadata. We define a container-based file format to group data, metadata, and security information at the syntactical level. It includes new features like grouping multiple results in a same file to simplify the transport of whole studies. We use the granularity of the encoder’s output to enhance security. The information is represented in units, each dedicated to a specific region of the genome, which allows to provide encryption and signature features on a region base. We analyze the trade-off between security and an even more fine-grained approach and prove that apparently secure settings can be insecure: if the file creator may encrypt only specific elements of a unit, cross-checking unencrypted information permits to infer encrypted content. Most of the proposals for MPEG-G coming from other research groups and companies focused on data compression and representation. However, the need was recognized to find a solution for metadata encoding. Our proposal was included in the standard: an XML-based solution, separated in a core specification and extensions. It permits to adapt the metadata schema to the different genomic repositories' frameworks, without importing requirements from one framework to another. To simplify the handling of the resulting metadata, we define profiles, i.e. lists of extensions that must be present in a given framework. We use XML signature and XML encryption for metadata security. The MPEG requirements also concern access rules. Our privacy solutions limit the range of persons with access and we propose access rules represented with XACML to convey under which circumstances a user is granted access to a specific action among the ones specified in MPEG-G's API, e.g. filtering data by attributes. We also specify algorithms to combine multiple rules by defining default behaviors and exceptions. The standard’s security mechanisms protect the information only during transport and access. Once the data is obtained, the user could publish it. In order to identify leakers, we propose an algorithm that generates unique, virtually undetectable variations. Our solution is novel as the marking can be undone (and the utility of the data preserved) if the corresponding secret key is revealed. We also show how to combine multiple secret keys to avoid collusion. The API retained for MPEG-G considers search criteria not present in the indexing tables, which highlights shortcomings. Based on the proposed MPEG-G API we have developed a solution. It is based on a collaboration framework where the different users' needs and the patient's privacy settings result in a purpose-built file format that optimizes query times and provides privacy and authenticity on the patient-defined genomic regions. The encrypted output units are created and indexed to optimize query times and avoid rarely used indexing fields. Our approach resolves the shortcomings of MPEG-G's indexing strategy. We have submitted our technologies to the MPEG standardization committee. Many have been included in the final standard, via merging with other proposals (e.g. file format), discussion (e.g. security mechanisms), or direct acceptance (e.g. privacy rules).Hi han nous mètodes per la seqüenciació i el processament del codi genòmic, permetent descobrir eines de diagnòstic i tractaments en l’àmbit mèdic. El resultat de la seqüenciació d’un genoma es representa en un fitxer, que pot ocupar centenars de gigabytes. Degut a això, hi ha una necessitat d’una representació estandarditzada on la informació és comprimida. Dins de la ISO, el grup MPEG ha fet servir la seva experiència en compressió de dades multimèdia per comprimir dades genòmiques i desenvolupar l'estàndard MPEG-G, sent la seguretat un dels requeriments principals. L'objectiu de la tesi és garantir aquesta seguretat (encriptant, firmant i definint regles d¿ accés) tan per les dades seqüenciades com per les seves metadades. El primer pas és definir com transportar les dades, metadades i paràmetres de seguretat. Especifiquem un format de fitxer basat en contenidors per tal d'agrupar aquets elements a nivell sintàctic. La nostra solució proposa noves funcionalitats com agrupar múltiples resultats en un mateix fitxer. Pel que fa la seguretat de dades, la nostra proposta utilitza les propietats de la sortida del codificador. Aquesta sortida és estructurada en unitats, cadascuna dedicada a una regió concreta del genoma, permetent una encriptació i firma de dades específica a la unitat. Analitzem el compromís entre seguretat i un enfocament de gra més fi demostrant que configuracions aparentment vàlides poden no ser-ho: si es permet encriptar sols certes sub-unitats d'informació, creuant els continguts no encriptats, podem inferir el contingut encriptat. Quant a metadades, proposem una solució basada en XML separada en una especificació bàsica i en extensions. Podem adaptar l'esquema de metadades als diferents marcs de repositoris genòmics, sense imposar requeriments d’un marc a un altre. Per simplificar l'ús, plantegem la definició de perfils, és a dir, una llista de les extensions que han de ser present per un marc concret. Fem servir firmes XML i encriptació XML per implementar la seguretat de les metadades. Les nostres solucions per la privacitat limiten qui té accés a les dades, però no en limita l’ús. Proposem regles d’accés representades amb XACML per indicar en quines circumstàncies un usuari té dret d'executar una de les accions especificades a l'API de MPEG-G (per exemple, filtrar les dades per atributs). Presentem algoritmes per combinar regles, per tal de poder definir casos per defecte i excepcions. Els mecanismes de seguretat de MPEG-G protegeixen la informació durant el transport i l'accés. Una vegada l’usuari ha accedit a les dades, les podria publicar. Per tal d'identificar qui és l'origen del filtratge de dades, proposem un algoritme que genera modificacions úniques i virtualment no detectables. La nostra solució és pionera, ja que els canvis es poden desfer si el secret corresponent és publicat. Per tant, la utilitat de les dades és mantinguda. Demostrem que combinant varis secrets, podem evitar col·lusions. L'API seleccionada per MPEG-G, considera criteris de cerca que no són presents en les taules d’indexació. Basant-nos en aquesta API, hem desenvolupat una solució. És basada en un marc de col·laboració, on la combinació de les necessitats dels diferents usuaris i els requeriments de privacitat del pacient, es combinen en una representació ad-hoc que optimitza temps d’accessos tot i garantint la privacitat i autenticitat de les dades. La majoria de les nostres propostes s’han inclòs a la versió final de l'estàndard, fusionant-les amb altres proposes (com amb el format del fitxer), demostrant la seva superioritat (com amb els mecanismes de seguretat), i fins i tot sent acceptades directament (com amb les regles de privacitat).Postprint (published version

A light-weight concept ontology for annotating digital music.

In the recent time, the digital music items on the internet have been evolving to an enormous information space where we try to find/locate the piece of information of our choice by means of search engine. The current trend of searching for music by means of music consumers' keywords/tags is unable to provide satisfactory search results; and search and retrieval of music may be potentially improved if music metadata is created from semantic information provided by association of end-users' tags with acoustic metadata which is easy to extract automatically from digital music items. Based on this observation, our research objective was to investigate how music producers may be able to annotate music against MPEG-7 description (with its acoustic metadata) to deliver meaningful search results. In addressing this question, we investigated the potential of multimedia ontologies to serve as backbone for annotating music items and prospective application scenarios of semantic technologies in the digital music industry. We achieved with our main contribution under this thesis is the first prototype of mpeg-7Music annotation ontology that establishes a mapping of end-users tags with MPEG-7 acoustic metadata as well as extends upper level multimedia ontologies with end-user tags. Additionally, we have developed a semi-automatic annotation tool to demonstrate the potential of the mpeg-7Music ontology to serve as light weight concept ontology for annotating digital music by music producers. The proposed ontology has been encoded in dominant semantic web ontology standard OWL1.0 and provides a standard interoperable representation of the generated semantic metadata. Our innovations in designing the semantic annotation tool were focussed on supporting the music annotation vocabulary (i.e. the mpeg-7Music) in an attempt to turn the music metadata information space to a knowledgebase

An automatic audio classification system for radio newscast

Current web search engines generally do not enable searches into audio files. Informative metadata would allow searches into audio files, but producing such metadata is a tedious manual task. Tools for automatic production of metadata are therefore needed. This project describes the work done on the development of an automatic audio classification system which can be used for this metadata extraction. In order to design this system I used adapting it to our case of study, the matlab code of the MPEG-7 Experimental Model [15]

Report of CE on Semantic DS

ISO/IEC JTC1/SC29/WG11, MPEG00/M6355, 53rd meeting, Jul. 2000, Beijing, PR