Relaxed Precision and Recall for Ontology Matching

ehrig2005aInternational audienceIn order to evaluate the performance of ontology matching algorithms it is necessary to confront them with test ontologies and to compare the results. The most prominent criteria are precision and recall originating from information retrieval. However, it can happen that an alignment be very close to the expected result and another quite remote from it, and they both share the same precision and recall. This is due to the inability of precision and recall to measure the closeness of the results. To overcome this problem, we present a framework for generalizing precision and recall. This framework is instantiated by three different measures and we show in a motivating example that the proposed measures are prone to solve the problem of rigidity of classical precision and recall

Descubrimiento automático de mappings en un caso de uso real con altas exigencias de certeza

Los sistemas de integración de información resuelven las diferencias entre las fuentes, en la mayoría de los casos, mediante la creación de mappings, puentes semánticos entre los elementos de las fuentes. Hasta ahora se han propuesto comparadores para generar un conjunto de mappings para cada par de elementos de las fuentes a integrar, y se han realizado estudios experimentales con ellos. El valor añadido del presente trabajo frente a los trabajos experimentales anteriores es que se ha llevado a cabo en un caso real embebido en una aplicación real (en el dominio geográfico) con altas exigencias de certeza

Generalizing precision and recall for evaluating ontology matching

ehrig2005bInternational audienceWe observe that the precision and recall measures are not able to discriminate between very bad and slightly out of target alignments. We propose to generalise these measures by determining the distance between the obtained alignment and the expected one. This generalisation is done so that precision and recall results are at least preserved. In addition, the measures keep some tolerance to errors, i.e., accounting for some correspondences that are close to the target instead of out of target

Debugging Ontology Mappings: A Static Approach

Ontology mapping is the bottleneck in solving interoperation between Semantic Web applications using heterogeneous ontologies. Many mapping methods have been proposed in recent years, but in practice, it is still difficult to obtain satisfactory mapping results having high precision and recall. Different from existing methods, which focus on finding efficient and effective solutions for the ontology mapping problem, we place emphasis on analyzing the mapping result to detect/diagnose the mapping defects. In this paper, a novel technique called debugging ontology mappings is presented. During debugging, some types of mapping errors, such as redundant and inconsistent mappings, can be detected. Some warnings, including imprecise mappings or abnormal mappings, are also locked by analyzing the features of mapping result. More importantly, some errors and warnings can be repaired automatically or can be presented to users with revising suggestions. The experimental results reveal that the ontology debugging technique is promising, and it can improve the quality of mapping result

Peut-on évaluer les outils d'acquisition de connaissances à partir de textes ?

National audienceMalgré les années de recul et d'expériences accumulées, il est difficile de se faire une idée claire de l'état d'avancement des recherches en acquisition de connaissances à partir de textes. Le manque de protocoles d'évaluation ne facilite pas la comparaison des résultats. Nous développons, dans cet article, la question de l'évaluation des outils d'acquisition de terminologies et d'ontologies en soulignant les princi- pales difficultés et en décrivant nos premières propositions dans ce domaine

Results of the ontology alignment evaluation initiative 2019

The Ontology Alignment Evaluation Initiative (OAEI) aims at comparing ontology matching systems on precisely defined test cases. These test cases can be based on ontologies of different levels of complexity (from simple thesauri to expressive OWL ontologies) and use different evaluation modalities (e.g., blind evaluation, open evaluation, or consensus). The OAEI 2019 campaign offered 11 tracks with 29 test cases, and was attended by 20 participants. This paper is an overall presentation of that campaign

A probabilistic evaluation procedure for process model matching techniques

Process model matching refers to the automatic identification of corresponding activities between two process models. It represents the basis for many advanced process model analysis techniques such as the identification of similar process parts or process model search. A central problem is how to evaluate the performance of process model matching techniques. Current evaluation methods require a binary gold standard that clearly defines which correspondences are correct. The problem is that often not even humans can agree on a set of correct correspondences. Hence, evaluating the performance of matching techniques based on a binary gold standard does not take the true complexity of the matching problem into account and does not fairly assess the capabilities of a matching technique. In this paper, we propose a novel evaluation procedure for process model matching techniques. In particular, we build on the assessments of multiple annotators to define the notion of a non-binary gold standard. In this way, we avoid the problem of agreeing on a single set of correct correspondences. Based on this non-binary gold standard, we introduce probabilistic versions of precision, recall, and F-measure as well as a distance-based performance measure. We use a dataset from the Process Model Matching Contest 2015 and a total of 16 matching systems to assess and compare the insights that can be obtained by using our evaluation procedure. We find that our probabilistic evaluation procedure allows us to gain more detailed insights into the performance of matching systems than a traditional evaluation based on a binary gold standard

Semantic precision and recall for ontology alignment evaluation

euzenat2007aInternational audienceIn order to evaluate ontology matching algorithms it is necessary to confront them with test ontologies and to compare the results with some reference. The most prominent comparison criteria are precision and recall originating from information retrieval. Precision and recall are thought of as some degree of correction and completeness of results. However, when the objects to compare are semantically defined, like ontologies and alignments, it can happen that a fully correct alignment has low precision. This is due to the restricted set-theoretic foundation of these measures. Drawing on previous syntactic generalizations of precision and recall, semantically justified measures that satisfy maximal precision and maximal recall for correct and complete alignments is proposed. These new measures are compatible with classical precision and recall and can be computed