Benchmarking Semantic Web Technology

This paper summarises the research problem that the PhD thesis I am currently writing addresses. It presents an overview of the thesis, its goals taking into account the deciencies of the State of the Art, the approach followed, and the work performed for the thesis since its beginning in 2004

Benchmarking Semantic Web technology

Semantic Web technologies need to interchange ontologies for further use. Due to the heterogeneity in the knowledge representation formalisms of the different existing technologies, interoperability is a problem in the SemanticWeb and the limits of the interoperability of current technologies are yet unknown. A massive improvement of the interoperability of current SemanticWeb technologies, or of any other characteristic of these technologies, requires continuous evaluations that should be defined and conducted in consensus, using generic, reusable, freely-available, and affordable tools and methods. This thesis presents the following contributions to the field of benchmarking within Semantic Web technologies: It proposes a benchmarking methodology for Semantic Web technologies. It defines the UPM Framework for Benchmarking Interoperability, an evaluation infrastructure that includes all the resources (experiment definitions, benchmark suites and tools) needed for benchmarking the interoperability of Semantic Web technologies using RDF(S) and OWL as interchange languages. It describes two interoperability benchmarking activities carried out over Semantic Web technologies and provides detailed interoperability results of the tools that participated in them; the RDF(S) Interoperability Benchmarking that contemplates interoperability using RDF(S) as the interchange language, and the OWL Interoperability Benchmarking that contemplates interoperability using OWL as the interchange language. Las diferentes tecnologías de la Web Semántica necesitan intercambiar ontologías para su posterior utilización. La Web Semántica, por otro lado, tiene que hacer frente al problema de la interoperabilidad, que está causado, en gran medida, por la heterogeneidad de los formalismos de representación de conocimiento de las distintas tecnologías existentes, siendo la interoperabilidad en la Web Semántica un problema cuyos límites hoy se desconocen. Una mejora masiva de la interoperabilidad de las tecnologías actuales de la Web Semántica, o de cualquier otra característica de dichas tecnologías, requiere evaluaciones continuas, que sean definidas y realizadas en consenso, utilizando herramientas y métodos que sean genéricos, reutilizables, públicos y económicos. Esta tesis presenta las siguientes contribuciones al campo de benchmarking en las tecnologías de la Web Semántica: Propone una metodología de benchmarking para las tecnologías de la Web Semántica. Define el UPM Framework for Benchmarking Interoperability, una infraestructura de evaluación que incluye todos los recursos (definiciones de experimentos, conjuntos de pruebas y herramientas) necesarios para hacer benchmarking de la interoperabilidad de las tecnologías de la Web Semántica utilizando RDF(S) y OWL como lenguajes de intercambio. Describe dos actividades de benchmarking de las tecnologías de la Web Semántica y ofrece los resultados detallados de las herramientas que participaron en las mismas; el RDF(S) Interoperability Benchmarking, que contempla la interoperabilidad utilizando RDF(S) como lenguaje de intercambio, y el OWL Interoperability Benchmarking, que también contempla la interoperabilidad pero utiliza OWL como lenguaje de intercambio

Interoperability results for Semantic Web technologies using OWL as the interchange language

Using Semantic Web technologies in complex scenarios requires that such technologies correctly interoperate by interchanging ontologies using the RDF(S) and OWL languages. This interoperability is not straightforward because of the high heterogeneity in Semantic Web technologies and, while the number of such technologies grows, affordable mechanisms for evaluating Semantic Web technology interoperability are needed to comprehend the current and future interoperability of Semantic Web technologies. This paper presents the OWL Interoperability Benchmarking, an international benchmarking activity that involved the evaluation of the interoperability of different Semantic Web technologies using OWL as the interchange language. It describes the evaluation resources used in this benchmarking activity, the OWL Lite Import Benchmark Suite and the IBSE tool, and presents how to use them for evaluating the OWL interoperability of Semantic Web technologies. Moreover, the paper offers an overview of the OWL interoperability results of the eight tools participating in the benchmarking: one ontology-based annotation tool (GATE), three ontology frameworks (Jena, KAON2, and SWI-Prolog), and four ontology development tools (Protégé Frames, Protégé OWL, SemTalk, and WebODE)

Benchmarking in the Semantic Web

The Semantic Web technology needs to be thoroughly evaluated for providing objective results and obtaining massive improvement in its quality; thus, the transfer of this technology from research to industry will speed up. This chapter presents software benchmarking, a process that aims to improve the Semantic Web technology and to find the best practices. The chapter also describes a specific software benchmarking methodology and shows how this methodology has been used to benchmark the interoperability of ontology development tools, employing RDF(S) as the interchange language

Towards the improvement of the Semantic Web technology

The Semantic Web technology needs to be thoroughly evalu- ated for providing objective results and to attain a massive improvement in their quality in order to be consolidated in the industrial and in the academic world. This paper presents software benchmarking as a process to carry out over the SemanticWeb technology in order to improve it and to search for best practices. It also describes a software benchmarking methodology and provides recommendations for performing evaluations in benchmarking activities

Infraestructura tecnológica de servicios semánticos para la Web Semántica

This project aims at creating a network of distributed interoperable semantic services for building more complex ones. These services will be available in semantic Web service libraries, so that they can be invoked by other systems (e.g., semantic portals, software agents, etc.). Thus, to accomplish this objective, the project proposes: a) To create specific technology for developing and composing Semantic Web Services. b) To migrate the WebODE ontology development workbench to this new distributed interoperable semantic service architecture. c) To develop new semantic services (ontology learning, ontology mappings, incremental ontology evaluation, and ontology evolution). d) To develop technological support that eases semantic portal interoperability, using Web services and Semantic Web Services. The project results will be open source, so as to improve their technological transfer. The quality of these results is ensured by a benchmarking process. Keywords: Ontologies and Semantic We

IBSE: An OWL Interoperability Evaluation Infrastructure

The technology that supports the Semantic Web presents a great diversity and, whereas all the tools use different types of ontologies, not all of them share a common knowledge representation model, which may pose problems when they try to interoperate. The Knowledge Web European Network of Excellence is organizing a benchmarking of interoperability of ontology tools using OWL as interchange language with the goal of assessing and improving tool interoperability. This paper presents the development of IBSE, an evaluation infrastructure that allows executing automatically the benchmarking experiments and provides an easy way of analysing the results. Thus,including new tools into the evaluation infrastructure will be simple and straightforward

A pragmatic approach to semantic repositories benchmarking

The aim of this paper is to benchmark various semantic repositories in order to evaluate their deployment in a commercial image retrieval and browsing application. We adopt a two-phase approach for evaluating the target semantic repositories: analytical parameters such as query language and reasoning support are used to select the pool of the target repositories, and practical parameters such as load and query response times are used to select the best match to application requirements. In addition to utilising a widely accepted benchmark for OWL repositories (UOBM), we also use a real-life dataset from the target application, which provides us with the opportunity of consolidating our findings. A distinctive advantage of this benchmarking study is that the essential requirements for the target system such as the semantic expressivity and data scalability are clearly defined, which allows us to claim contribution to the benchmarking methodology for this class of applications

Knowledge web: realising the semantic web... all the way to knowledge-enhanced multimedia documents

The semantic web and semantic web services are major efforts in order to spread and to integrate knowledge technology to the whole web. The Knowledge Web network of excellence aims at supporting their developments at the best and largest European level and supporting industry in adopting them. It especially investigates the solution of scalability, heterogeneity and dynamics obstacles to the full development of the semantic web. We explain how Knowledge Web results should benefit knowledge-enhanced multimedia applications

CHORUS Deliverable 4.3: Report from CHORUS workshops on national initiatives and metadata

Minutes of the following Workshops: • National Initiatives on Multimedia Content Description and Retrieval, Geneva, October 10th, 2007. • Metadata in Audio-Visual/Multimedia production and archiving, Munich, IRT, 21st – 22nd November 2007 Workshop in Geneva 10/10/2007 This highly successful workshop was organised in cooperation with the European Commission. The event brought together the technical, administrative and financial representatives of the various national initiatives, which have been established recently in some European countries to support research and technical development in the area of audio-visual content processing, indexing and searching for the next generation Internet using semantic technologies, and which may lead to an internet-based knowledge infrastructure. The objective of this workshop was to provide a platform for mutual information and exchange between these initiatives, the European Commission and the participants. Top speakers were present from each of the national initiatives. There was time for discussions with the audience and amongst the European National Initiatives. The challenges, communalities, difficulties, targeted/expected impact, success criteria, etc. were tackled. This workshop addressed how these national initiatives could work together and benefit from each other. Workshop in Munich 11/21-22/2007 Numerous EU and national research projects are working on the automatic or semi-automatic generation of descriptive and functional metadata derived from analysing audio-visual content. The owners of AV archives and production facilities are eagerly awaiting such methods which would help them to better exploit their assets.Hand in hand with the digitization of analogue archives and the archiving of digital AV material, metadatashould be generated on an as high semantic level as possible, preferably fully automatically. All users of metadata rely on a certain metadata model. All AV/multimedia search engines, developed or under current development, would have to respect some compatibility or compliance with the metadata models in use. The purpose of this workshop is to draw attention to the specific problem of metadata models in the context of (semi)-automatic multimedia search