Towards Best Practices for Crowdsourcing Ontology Alignment Benchmarks

Ontology alignment systems establish the semantic links between ontologies that enable knowledge from various sources and domains to be used by automated applications in many different ways. Unfortunately, these systems are not perfect. Currently, the results of even the best-performing automated alignment systems need to be manually verified in order to be fully trusted. Ontology alignment researchers have turned to crowdsourcing platforms such as Amazon\u27s Mechanical Turk to accomplish this. However, there has been little systematic analysis of the accuracy of crowdsourcing for alignment verification and the establishment of best practices. In this work, we analyze the impact of the presentation of the context of potential matches and the way in which the question is presented to workers on the accuracy of crowdsourcing for alignment verification. Our overall recommendations are that users interested in high precision are likely to achieve the best results by presenting the definitions of the entity labels and allowing workers to respond with true/false to the question of whether or not an equivalence relationship exists. Conversely, if the alignment researcher is interested in high recall, they are better off presenting workers with a graphical depiction of the entity relationships and a set of options about the type of relation that exists, if any

Ontologies as a Set to Describe Legal Information

The article discusses the features of legal knowledge ontology creation. It is determined that ontology is the most appropriate way to describe legal knowledge. The particular qualities of legal information and the features of the language of a right were investigated. A review of legal knowledge ontologies that are used in various branches of law was made. The properties of legal information and the requirements for regulatory documentation in Ukraine were described. The formalization of the structure of the ontology database was presented, taking into account the required attributes of the concepts. The methodology of the work with the knowledge base was proposed to use the independent work of many users. The legal knowledge ontology at the law university was filled by all users of the software package, but experts checked the quality of this content. Crowdsourcing was considered as the main technique of the ontology filling process. Several branches of the ontology of legal knowledge were filled. The results of the experimental operation of this ontology by university students were analyzed

Towards Best Practices for Crowdsourcing Ontology Alignment Benchmarks

Ontology alignment systems establish the semantic links between ontologies that enable knowledge from various sources and domains to be used by automated applications in many different ways. Unfortunately, these systems are not perfect. Currently, the results of even the best-performing automated alignment systems need to be manually verified in order to be fully trusted. Ontology alignment researchers have turned to crowdsourcing platforms such as Amazon\u27s Mechanical Turk to accomplish this. However, there has been little systematic analysis of the accuracy of crowdsourcing for alignment verification and the establishment of best practices. In this work, we analyze the impact of the presentation of the context of potential matches and the way in which the question is presented to workers on the accuracy of crowdsourcing for alignment verification. Our overall recommendations are that users interested in high precision are likely to achieve the best results by presenting the definitions of the entity labels and allowing workers to respond with true/false to the question of whether or not an equivalence relationship exists. Conversely, if the alignment researcher is interested in high recall, they are better off presenting workers with a graphical depiction of the entity relationships and a set of options about the type of relation that exists, if any