Using a new video rating tool to crowd-source analysis of behavioural reaction to stimuli

Quantifying the intensity of animals’ reaction to stimuli is notoriously difficult as classic unidimensional measures of responses such as latency or duration of looking can fail to capture the overall strength of behavioural responses. More holistic rating can be useful but have the inherent risks of subjective bias and lack of repeatability. Here, we explored whether crowdsourcing could be used to efficiently and reliably overcome these potential flaws. A total of 396 participants watched online videos of dogs reacting to auditory stimuli and provided 23,248 ratings of the strength of the dogs’ responses from zero (default) to 100 using an online survey form. We found that raters achieved very high inter-rater reliability across multiple datasets (although their responses were affected by their sex, age, and attitude towards animals) and that as few as 10 raters could be used to achieve a reliable result. A linear mixed model applied to PCA components of behaviours discovered that the dogs’ facial expressions and head orientation influenced the strength of behaviour ratings the most. Further linear mixed models showed that that strength of behaviour ratings was moderately correlated to the duration of dogs’ reactions but not to dogs’ reaction latency (from the stimulus onset). This suggests that observers’ ratings captured consistent dimensions of animals’ responses that are not fully represented by more classic unidimensional metrics. Finally, we report that overall participants strongly enjoyed the experience. Thus, we suggest that using crowdsourcing can offer a useful, repeatable tool to assess behavioural intensity in experimental or observational studies where unidimensional coding may miss nuance, or where coding multiple dimensions may be too time-consuming

Using Citizen Scientists To Inform Machine Learning Algorithms To Automate The Detection Of Species In Ecological Imagery

Modern data collection techniques used by ecologists has created a deluge of data that is becoming increasingly difficult to store, filter, and analyze in an efficient and timely manner. In just two summers, over 65,000 unmanned aerial system (UAS) images were collected, comprising the several terabytes (TB) of data that was reviewed by citizen scientists to generate inputs for machine learning algorithms. Uncontrolled conditions and the small size of target species relative to the background further increase the difficulty of manually cataloging the images. To assist with locating and identifying snow geese in the UAS images, a citizen science web portal was created as part of Wildlife@Home. It is demonstrated that aggregate citizen scientist observations are similar in quality to observations made by trained experts and can be used to train convolutional neural networks (CNN) to automate the detection of species in the imagery. Using a dataset comprising of the aggregate observations produces consistently better results than datasets consisting of observations from a single altitude, indicating that more numerous but slightly variable observations is preferable to more consistent but less numerous observations. The framework developed requires system administrators to manually run scripts to populate the database with new images; however, this framework can be extended to allow researchers to create their own projects, upload new images, and download data for CNN training

Comparison of manual, machine learning, and hybrid methods for video annotation to extract parental care data

Measuring parental care behaviour in the wild is central to the study of animal ecology and evolution, but it is often labour- and time-intensive. Efficient open-source tools have recently emerged that allow animal behaviour to be quantified from videos using machine learning and computer vision techniques, but there is limited appraisal of how these tools perform compared to traditional methods. To gain insight into how different methods perform in extracting data from videos taken in the field, we compared estimates of the parental provisioning rate of wild house sparrows Passer domesticus from video recordings. We compared four methods: manual annotation by experts, crowd-sourcing, automatic detection based on the open-source software DeepMeerkat, and a hybrid annotation method. We found that the data collected by the automatic method correlated with expert annotation (r = 0.62) and further show that these data are biologically meaningful as they predict brood survival. However, the automatic method produced largely biased estimates due to the detection of non-visitation events, while the crowd-sourcing and hybrid annotation produced estimates that are equivalent to expert annotation. The hybrid annotation method takes approximately 20% of annotation time compared to manual annotation, making it a more cost-effective way to collect data from videos. We provide a successful case study of how different approaches can be adopted and evaluated with a pre-existing dataset, to make informed decisions on the best way to process video datasets. If pre-existing frameworks produce biased estimates, we encourage researchers to adopt a hybrid approach of first using machine learning frameworks to preprocess videos, and then to do manual annotation to save annotation time. As open-source machine learning tools are becoming more accessible, we encourage biologists to make use of these tools to cut annotation time but still get equally accurate results without the need to develop novel algorithms from scratch

Using Computer Vision And Volunteer Computing To Analyze Avian Nesting Patterns And Reduce Scientist Workload

This paper examines the use of feature detection and background subtraction algorithms to classify and detect events of interest within uncontrolled outdoor avian nesting video from the Wildlife@Home project. We tested feature detection using Speeded Up Robust Features (SURF) and a Support Vector Machine (SVM) along with four background subtraction algorithms — Mixture of Guassians (MOG), Running Gaussian Average (AccAvg), ViBe, and Pixel-Based Adaptive Segmentation (PBAS) — as methods to automatically detect and classify events from surveillance cameras. AccAvg and modified PBAS are shown to provide robust results and compensate for issues caused by cryptic coloration of the monitored species. Both methods utilize the Berkeley Open Infrastructure for Network Computing (BOINC) in order to provide the resources to be able to analyze the 68,000+ hours of video in the Wildlife@Home project in a reasonable amount of time. The feature detection technique failed to handle the many challenges found in the low quality uncontrolled outdoor video. The background subtraction work with AccAvg and the modified version of PBAS is shown to provide more accurate detection of events

Worker-Job Recommendation for Mixed Crowdsourcing Systems: Algorithms, Models, Metrics and Service-Oriented Architecture

Crowdsourcing is used as model to distribute work over the Internet via an open call to anonymous human workers, who opt to take up work offerings sometimes for some small compensation. Increasingly, crowdsourcing systems are integrated into workflows to provide human computation capabilities. These workflows consist of machine-based workers that work harmoniously on different phases of a task with their human counterparts. This body of work addresses workflows where machines and human workers have the capacity to fulfill the requirements for same tasks. To maximize performance through the delegation of work to the most competent worker, this work outlines a collaborative filtering based approach with a bottom up evaluation based on workers' performance history and their inferred skillsets. Within the model, there are several algorithms, formulae and evaluative metrics. The work also introduces the notion of an Open Push-Pull model; a paradigm that maximizes on the services and strengths of the open call model, while seeking to address its weaknesses such as platform lock-in that affects access to jobs and availability of the worker pool. The work outlines the model in terms of a service-oriented architecture (SOA). It provides a supporting conceptual model for the architecture and an operational model that facilitates both human and machine workers. It also defines evaluative metrics for understanding the true capabilities of the worker pool. Techniques presented in this work can be used to expand the potential worker pool to compete for tasks through the incorporation of machine-oriented workers via virtualization and other electronic services, and human workers via existing crowds. Results in this work articulate the flexibility of our approach to support both human and machine workers within a competitive model while supporting tasks spanning multiple domains and problem spaces. It addresses the inefficiencies of current top-down approaches in worker-job recommendation through use of a bottom-up approach which adapts to dynamic and rapidly changing data. The work contrasts the shortcomings of top-down approaches' dependency on professed profiles which can be under-represented, over-represented or falsified in other ways with evaluative metrics that can be used for the individual and collective assessment of workers within a labor pool.Ph.D., Computer Science -- Drexel University, 201

Aplicando Crowdsourcing na Sincronização de Vídeos Gerados por Usuários

Crowdsourcing é uma estratégia para resolução de problemas baseada na coleta de resultados parciais a partir das contribuições de indivíduos, agregando-as em um resultado unificado. Com base nesta estratégia, esta tese mostra como a crowd é capaz de sincronizar um conjunto de vídeos produzidos por usuários quaisquer, correlacionados a um mesmo evento social. Cada usuário filma o evento com seu ponto de vista e de acordo com suas limitações (ângulo do evento, oclusões na filmagem, qualidade da câmera utilizada, etc.). Nesse cenário, não é possível garantir que todos os conteúdos gerados possuam características homogêneas (instante de início e duração de captura, resolução, qualidade, etc.), dificultando o uso de um processo puramente automático de sincronização. Além disso, os vídeos gerados por usuário são disponibilizados de forma distribuída entre diversos servidores de conteúdo independentes. A hipótese desta tese é que a capacidade de adaptação da inteligência humana pode ser usada para processar um grupo de vídeos produzidos de forma descoordenada e distribuída, e relacionados a um mesmo evento social, gerando a sua sincronização. Para comprovar esta hipótese, as seguintes etapas foram executadas: (i) o desenvolvimento de um método de sincronização para múltiplos vídeos provenientes de fontes independentes; (ii) a execução de um mapeamento sistemático acerca do uso de crowdsourcing para processamento de vídeos; (iii) o desenvolvimento de técnicas para o uso da crowd na sincronização de vídeos; (iv) o desenvolvimento de um modelo funcional para desenvolvimento de aplicações de sincronização utilizando crowdsourcing, que pode ser estendido para aplicações de vídeos em geral; e (v) a realização de experimentos que permitem mostrar a capacidade da crowd para realizar a sincronização. Os resultados encontrados após estas etapas mostram que a crowd é capaz de participar do processo de sincronização e que diversos fatores podem influenciar na precisão dos resultados obtidos