An introduction to crowdsourcing for language and multimedia technology research

Language and multimedia technology research often relies on large manually constructed datasets for training or evaluation of algorithms and systems. Constructing these datasets is often expensive with significant challenges in terms of recruitment of personnel to carry out the work. Crowdsourcing methods using scalable pools of workers available on-demand offers a flexible means of rapid low-cost construction of many of these datasets to support existing research requirements and potentially promote new research initiatives that would otherwise not be possible

Symbiosis between the TRECVid benchmark and video libraries at the Netherlands Institute for Sound and Vision

Audiovisual archives are investing in large-scale digitisation efforts of their analogue holdings and, in parallel, ingesting an ever-increasing amount of born- digital files in their digital storage facilities. Digitisation opens up new access paradigms and boosted re-use of audiovisual content. Query-log analyses show the shortcomings of manual annotation, therefore archives are complementing these annotations by developing novel search engines that automatically extract information from both audio and the visual tracks. Over the past few years, the TRECVid benchmark has developed a novel relationship with the Netherlands Institute of Sound and Vision (NISV) which goes beyond the NISV just providing data and use cases to TRECVid. Prototype and demonstrator systems developed as part of TRECVid are set to become a key driver in improving the quality of search engines at the NISV and will ultimately help other audiovisual archives to offer more efficient and more fine-grained access to their collections. This paper reports the experiences of NISV in leveraging the activities of the TRECVid benchmark

Popular music as cultural heritage: scoping out the field of practice

This paper sets out to deepen our understanding of the relationship between popular music and cultural heritage and to delineate the practices of popular music as cultural heritage. The paper illustrates how the term has been mobilised by a variety of actors, from the public to the private sector, to highlight the value of particular popular music manifestations and justify or encourage their preservation and diffusion for posterity. We focus on Austria, England, France and the Netherlands – countries with diverse popular music histories and with varying national and international reach. Popular music heritage is present in national and local public sector heritage institutions and practices in a number of ways. These range from the preservation and exhibition of the material culture of heritage in museums and archives, to a variety of ‘bottom-up’ initiatives, delineating a rich landscape of emblematic places, valued for their attachment to particular musicians or music scenes. The paper points to an underlying tension between the adoption and replication of conventional heritage practices to the preservation and remembrance of the popular music and its celebration as an express

Places of popular music heritage: the local framing of a global cultural form in Dutch museums and archives

Through the prism of popular music, this article examines how the preservation and display of this global cultural form positions itself at the nexus of the local and the global, and in so doing mediates attachment to place. Springing from the increasing cultural legitimacy of popular music and the growing participation of fans and local communities in heritage practices, Dutch private and public heritage initiatives are analyzed to explore how the local histories and lived experiences of popular music reverberate in the framework of wider global cultural developments. The results of this study indicate that museums and archives give places meaning through three interrelated processes. They present local sociocultural histories, foster a sense of belonging and cultural pride, and seek to document the artistic legacy of places. Furthermore, it is found that despite the strong transnational dimension of popular music, the studied heritage practices strongly resonate with local and national cultural identities, as narratives of popular music and heritage are mediated by locally situated cultural gatekeepers. These findings are based on in-depth interviews conducted with archivists, collectors and curators from the Netherlands

Run spot run: capturing and tagging footage of a race by crowds of spectators

There has been a massive growth in the number of people who film and upload amateur footage of events to services such as Facebook and Youtube, or even stream live to services such as LiveStream. We present an exploratory study that investigates the potential of these spectators in creating footage en masse; in this case, during a live trial at a local marathon. We deployed a prototype app, RunSpotRun, as a technology probe to see what kinds of footage spectators would produce. We present an analysis of this footage in terms of its coverage, quality, and contents, and also discuss the implications for a) spectators enjoying the race, and b) extracting the stories of individual runners throughout the race. We conclude with a discussion of the challenges that remain for deploying such technology at a larger scale

Augmenting the performance of image similarity search through crowdsourcing

Crowdsourcing is defined as “outsourcing a task that is traditionally performed by an employee to a large group of people in the form of an open call” (Howe 2006). Many platforms designed to perform several types of crowdsourcing and studies have shown that results produced by crowds in crowdsourcing platforms are generally accurate and reliable. Crowdsourcing can provide a fast and efficient way to use the power of human computation to solve problems that are difficult for machines to perform. From several different microtasking crowdsourcing platforms available, we decided to perform our study using Amazon Mechanical Turk. In the context of our research we studied the effect of user interface design and its corresponding cognitive load on the performance of crowd-produced results. Our results highlighted the importance of a well-designed user interface on crowdsourcing performance. Using crowdsourcing platforms such as Amazon Mechanical Turk, we can utilize humans to solve problems that are difficult for computers, such as image similarity search. However, in tasks like image similarity search, it is more efficient to design a hybrid human–machine system. In the context of our research, we studied the effect of involving the crowd on the performance of an image similarity search system and proposed a hybrid human–machine image similarity search system. Our proposed system uses machine power to perform heavy computations and to search for similar images within the image dataset and uses crowdsourcing to refine results. We designed our content-based image retrieval (CBIR) system using SIFT, SURF, SURF128 and ORB feature detector/descriptors and compared the performance of the system using each feature detector/descriptor. Our experiment confirmed that crowdsourcing can dramatically improve the CBIR system performance

Playing with difficult objects: game designs for crowdsourcing museum metadata

This project explores the potential for casual browser-based games to help improve the quality of museum catalogue records. The project goal was to design and build casual yet compelling games that would have a positive impact on a practical level, helping improve the mass of 'difficult' – technical, near-duplicate, poorly catalogued or scantily digitised – records that make up the majority of many history museum collections. The project was successful in designing games that created improved metadata for 'difficult' objects from two science and history museum collections: Dora, a tagging game, and Donald, an experimental 'trivia' game that explored emergent game-play around longer forms of content that required some form of research or personal reference