6 research outputs found
Towards explainable evaluation of language models on the semantic similarity of visual concepts
Recent breakthroughs in NLP research, such as the advent of Transformer
models have indisputably contributed to major advancements in several tasks.
However, few works research robustness and explainability issues of their
evaluation strategies. In this work, we examine the behavior of high-performing
pre-trained language models, focusing on the task of semantic similarity for
visual vocabularies. First, we address the need for explainable evaluation
metrics, necessary for understanding the conceptual quality of retrieved
instances. Our proposed metrics provide valuable insights in local and global
level, showcasing the inabilities of widely used approaches. Secondly,
adversarial interventions on salient query semantics expose vulnerabilities of
opaque metrics and highlight patterns in learned linguistic representations
Employing Crowdsourcing for Enriching a Music Knowledge Base in Higher Education
This paper describes the methodology followed and the lessons learned from
employing crowdsourcing techniques as part of a homework assignment involving
higher education students of computer science. Making use of a platform that
supports crowdsourcing in the cultural heritage domain students were solicited
to enrich the metadata associated with a selection of music tracks. The results
of the campaign were further analyzed and exploited by students through the use
of semantic web technologies. In total, 98 students participated in the
campaign, contributing more than 6400 annotations concerning 854 tracks. The
process also led to the creation of an openly available annotated dataset,
which can be useful for machine learning models for music tagging. The
campaign's results and the comments gathered through an online survey enable us
to draw some useful insights about the benefits and challenges of integrating
crowdsourcing into computer science curricula and how this can enhance
students' engagement in the learning process.Comment: To be published in The 4th International Conference on Artificial
Intelligence in Education Technology (AIET 2023), Berlin, Germany, 31 June-2
July 2023. For The GitHub code for the created music dataset, see
https://github.com/vaslyb/MusicCro
CrowdHeritage: Crowdsourcing for Improving the Quality of Cultural Heritage Metadata
The lack of granular and rich descriptive metadata highly affects the discoverability and usability of cultural heritage collections aggregated and served through digital platforms, such as Europeana, thus compromising the user experience. In this context, metadata enrichment services through automated analysis and feature extraction along with crowdsourcing annotation services can offer a great opportunity for improving the metadata quality of digital cultural content in a scalable way, while at the same time engaging different user communities and raising awareness about cultural heritage assets. To address this need, we propose the CrowdHeritage open end-to-end enrichment and crowdsourcing ecosystem, which supports an end-to-end workflow for the improvement of cultural heritage metadata by employing crowdsourcing and by combining machine and human intelligence to serve the particular requirements of the cultural heritage domain. The proposed solution repurposes, extends, and combines in an innovative way general-purpose state-of-the-art AI tools, semantic technologies, and aggregation mechanisms with a novel crowdsourcing platform, so as to support seamless enrichment workflows for improving the quality of CH metadata in a scalable, cost-effective, and amusing way
CrowdHeritage: Crowdsourcing for Improving the Quality of Cultural Heritage Metadata
The lack of granular and rich descriptive metadata highly affects the discoverability and usability of cultural heritage collections aggregated and served through digital platforms, such as Europeana, thus compromising the user experience. In this context, metadata enrichment services through automated analysis and feature extraction along with crowdsourcing annotation services can offer a great opportunity for improving the metadata quality of digital cultural content in a scalable way, while at the same time engaging different user communities and raising awareness about cultural heritage assets. To address this need, we propose the CrowdHeritage open end-to-end enrichment and crowdsourcing ecosystem, which supports an end-to-end workflow for the improvement of cultural heritage metadata by employing crowdsourcing and by combining machine and human intelligence to serve the particular requirements of the cultural heritage domain. The proposed solution repurposes, extends, and combines in an innovative way general-purpose state-of-the-art AI tools, semantic technologies, and aggregation mechanisms with a novel crowdsourcing platform, so as to support seamless enrichment workflows for improving the quality of CH metadata in a scalable, cost-effective, and amusing way