Classifying Flies Based on Reconstructed Audio Signals

Advancements in sensor technology and processing power have made it possible to create recording equipment that can reconstruct the audio signal of insects passing through a directed infrared beam. The widespread deployment of such devices would allow for a range of applications previously not practical. A sensor net of detectors could be used to help model population dynamics, assess the efficiency of interventions and serve as an early warning system. At the core of any such system is a classification problem: given a segment of audio collected as something passes through a sensor, can we classify it? We examine the case of detecting the presence of fly species, with a particular focus on mosquitoes. This gives rise to a range of problems such as: can we discriminate between species of fly? Can we detect different species of mosquito? Can we detect the sex of the insect? Automated classification would significantly improve the effectiveness and efficiency of vector monitoring using these sensor nets. We assess a range of time series classification (TSC) algorithms on data from two projects working in this area. We assess our prior belief that spectral features are most effective, and we remark on all approaches with respect to whether they can be considered ``real-time''

Automating the Surveillance of Mosquito Vectors from Trapped Specimens Using Computer Vision Techniques

Among all animals, mosquitoes are responsible for the most deaths worldwide. Interestingly, not all types of mosquitoes spread diseases, but rather, a select few alone are competent enough to do so. In the case of any disease outbreak, an important first step is surveillance of vectors (i.e., those mosquitoes capable of spreading diseases). To do this today, public health workers lay several mosquito traps in the area of interest. Hundreds of mosquitoes will get trapped. Naturally, among these hundreds, taxonomists have to identify only the vectors to gauge their density. This process today is manual, requires complex expertise/ training, and is based on visual inspection of each trapped specimen under a microscope. It is long, stressful and self-limiting. This paper presents an innovative solution to this problem. Our technique assumes the presence of an embedded camera (similar to those in smart-phones) that can take pictures of trapped mosquitoes. Our techniques proposed here will then process these images to automatically classify the genus and species type. Our CNN model based on Inception-ResNet V2 and Transfer Learning yielded an overall accuracy of 80% in classifying mosquitoes when trained on 25,867 images of 250 trapped mosquito vector specimens captured via many smart-phone cameras. In particular, the accuracy of our model in classifying Aedes aegypti and Anopheles stephensi mosquitoes (both of which are deadly vectors) is amongst the highest. We present important lessons learned and practical impact of our techniques towards the end of the paper

Classifying dangerous species of mosquito using machine learning

This thesis begins by presenting the performance of modern Time Series Classification (TSC) approaches, including HIVE-COTEv2 & InceptionTime, on 4 new insect wingbeat datasets. The experiments throughout this thesis endeavour to explore whether it is possible to classify flying insects into their respective species and into group based on their sex. Furthermore, it is hypothesised that a hierarchical approach to classifying flying insects is possible via filtering “easy” cases using cheap to obtain features, reducing the number of times processing intensive approaches are utilised. Experiments are undertaken on 3 representations of the data: Harmonic Spectral Product (HSP), the raw data and spectral data. HSP is a method of extracting the fundamental frequency of a signal. It represents a logical benchmark for comparison and, is easy and quick to extract. In one dataset, InsectSounds, species are separated into sex. Evaluation of the results achieved with the HSP representation showed that despite a relatively poor overall accuracy this feature produces a low type II error with respect to female mosquitoes. It is shown that classes of mosquitoes species that are female were more likely to be miss-classified as other female mosquito classes and, where fly classes are miss-classified as mosquito classes, they are typically classified as male mosquitoes. Previous work had shown that transformation into the frequency domain has a positive effect on performance. Audio data is typically recorded at a high sample rate, which results in high spectral resolution. As a result, approaches from the literature have used truncation of high and low frequency data to reduce runtime. It is hypothesised that inclusion of low frequency data will aid classification. This is because low frequency data is likely caused by the body of the mosquito and morphological differences, such as size, are strongly correlated to sex. The results show that the performance of all approaches was improved by the use of spectral data. The results also showed that spectral data that included low frequency information resulted in a higher overall accuracy than transformations that discarded it. Formative experiments showed that HIVE-COTEv1 was the most accurate approach at classifying flying insects. HIVE-COTEv1 is a heterogeneous approach that consists of 4 modules, Random Interval Spectral Ensemble (RISE), Bag Of SFA Symbols (BOSS), Shapelet Transform Classifier (STC) and Time Series Forest (TSF). The predictive power of these modules are combined via Cross-validation Accuracy Weighted Probabilistic Ensemble (CAWPE). The RISE approach was chosen as the spectral component as it was “best in class” at the inception of HIVE-COTEv1. It is suggested that a significant improvement to the usability and accuracy of RISE, would translate as an improvement in the performance of HIVE-COTEv1. The introduction of contracting provided a method through witch the training time of RISE could be effectively controlled, improving its usability. A review of the interval selection procedure led to improvements that had a significant positive effect on accuracy. A review of spectral transforms and the method of combining them led to a further improvement to accuracy, and an architecture in which multiple transformations are applied. In order for smart traps to be effective they are required to work for extended periods in rural locations. Implementations of hierarchical approaches show that two expert features, HSP and time of flight (TOF) are effective in reducing test time and therefore the amount of processing required. This is achieved via first classifying the test case using simple approaches, such as BayesNet, and only if the confidence in the prediction does not meet a parameterised threshold using a more powerful approach. In an evaluation of several methods of combination, the most efficient of these is shown to increase classification accuracy by 0.6%, increase the TPR of female mosquitoes by 48/10,000, decrease the FNR of female mosquitoes by 83/15,000 and reduce test time by 1.5 hours over 25,000 instances, when compared to the single best approach InceptionTime. Furthermore, a cumulative approach to combining the expert features with the InceptionTime approach resulted in a 4.14% increase in accuracy, an increase in the TPR of female mosquitoes of 139/10,000 and a decrease in the FNR of female mosquitoes of 45/15,000

Analysis of physiological signals using machine learning methods

Technological advances in data collection enable scientists to suggest novel approaches, such as Machine Learning algorithms, to process and make sense of this information. However, during this process of collection, data loss and damage can occur for reasons such as faulty device sensors or miscommunication. In the context of time-series data such as multi-channel bio-signals, there is a possibility of losing a whole channel. In such cases, existing research suggests imputing the missing parts when the majority of data is available. One way of understanding and classifying complex signals is by using deep neural networks. The hyper-parameters of such models have been optimised using the process of back propagation. Over time, improvements have been suggested to enhance this algorithm. However, an essential drawback of the back propagation can be the sensitivity to noisy data. This thesis proposes two novel approaches to address the missing data challenge and back propagation drawbacks: First, suggesting a gradient-free model in order to discover the optimal hyper-parameters of a deep neural network. The complexity of deep networks and high-dimensional optimisation parameters presents challenges to find a suitable network structure and hyper-parameter configuration. This thesis proposes the use of a minimalist swarm optimiser, Dispersive Flies Optimisation(DFO), to enable the selected model to achieve better results in comparison with the traditional back propagation algorithm in certain conditions such as limited number of training samples. The DFO algorithm offers a robust search process for finding and determining the hyper-parameter configurations. Second, imputing whole missing bio-signals within a multi-channel sample. This approach comprises two experiments, namely the two-signal and five-signal imputation models. The first experiment attempts to implement and evaluate the performance of a model mapping bio-signals from A toB and vice versa. Conceptually, this is an extension to transfer learning using CycleGenerative Adversarial Networks (CycleGANs). The second experiment attempts to suggest a mechanism imputing missing signals in instances where multiple data channels are available for each sample. The capability to map to a target signal through multiple source domains achieves a more accurate estimate for the target domain. The results of the experiments performed indicate that in certain circumstances, such as having a limited number of samples, finding the optimal hyper-parameters of a neural network using gradient-free algorithms outperforms traditional gradient-based algorithms, leading to more accurate classification results. In addition, Generative Adversarial Networks could be used to impute the missing data channels in multi-channel bio-signals, and the generated data used for further analysis and classification tasks

A review of domain adaptation without target labels

Domain adaptation has become a prominent problem setting in machine learning and related fields. This review asks the question: how can a classifier learn from a source domain and generalize to a target domain? We present a categorization of approaches, divided into, what we refer to as, sample-based, feature-based and inference-based methods. Sample-based methods focus on weighting individual observations during training based on their importance to the target domain. Feature-based methods revolve around on mapping, projecting and representing features such that a source classifier performs well on the target domain and inference-based methods incorporate adaptation into the parameter estimation procedure, for instance through constraints on the optimization procedure. Additionally, we review a number of conditions that allow for formulating bounds on the cross-domain generalization error. Our categorization highlights recurring ideas and raises questions important to further research.Comment: 20 pages, 5 figure

Social Influences on Songbird Behavior: From Song Learning to Motion Coordination

Social animals learn during development how to integrate successfully into their group. How do social interactions combine to maintain group cohesion? We first review how social environments can influence the development of vocal learners, such as songbirds and humans (Chapter 1). To bypass the complexity of natural social interactions and gain experimental control, we developed Virtual Social Environments, surrounding the bird with videos of manipulated playbacks. This way we were able to design sensory and social scenarios and test how social zebra finches adjust their behavior (Chapters 2 & 3). A serious challenge is that the color output of a video monitor does not match the color vision of zebra finches. To minimize chromatic distortion, we eliminated all of the colors from the videos, except in the beak and cheeks where we superimposed colors that match the sensitivity of zebra finch photoreceptors (Chapter 2). Birds strongly preferred to watch these manipulated ‘bird appropriate’ videos. We also designed Virtual Social Environments for assessing how observing movement patterns might affect behavior in real-time (Chapter 3). We found that presenting birds with manipulated movement patterns of virtual males promptly affects the mobility of birds watching the videos: birds move more when virtual males increase their movements, and they decrease their movements and ‘cuddle’ next to virtual males that stop moving. These results suggest that individuals adjust their activity levels to the statistical patterns of observed conspecific movements, which can explain zebra finch group cohesion. Finally, we studied the song development process in the absence of social input to determine how intrinsic biases and external stimuli shape song from undifferentiated syllables into well-defined categorical signals of adult song (Chapter 4). Do juveniles learn the statistics of early sub-song to guide vocal development? We trained juvenile zebra finches with playbacks of their own, highly variable, developing song and showed that these self-tutored birds developed distinct syllable types (categories) as fast as birds that were trained with a categorical, adult song template. Therefore, the statistical structure of early input seems to have no bearing on the development of phonetic categories. Overall, our results uncover social forces that influence individual behaviors, from motion coordination to vocal development, which have implications for how group structures and vocal culture are maintained

Female Genitalia Concealment Promotes Intimate Male Courtship in a Water Strider

Violent coercive mating initiation is typical for animals with sexual conflict over mating. In these species, the coevolutionary arms-race between female defenses against coercive mating and male counter-adaptations for increased mating success leads to coevolutionary chases of male and female traits that influence the mating. It has been controversial whether one of the sexes can evolve traits that allow them to “win” this arms race. Here, we use morphological analysis (traditional and scanning electron micrographs), laboratory experiments and comparative methods to show how females of a species characterized by typical coercive mating initiation appear to “win” a particular stage of the sexual conflict by evolving morphology to hide their genitalia from direct, forceful access by males. In an apparent response to the female morphological adaptation, males of this species added to their typically violent coercive mounting of the female new post-mounting, pre-copulatory courtship signals produced by tapping the water's surface with the mid-legs. These courtship signals are intimate in the sense that they are aimed at the female, on whom the male is already mounted. Females respond to the signals by exposing their hidden genitalia for copulatory intromission. Our results indicate that the apparent victory of coevolutionary arms race by one sex in terms of morphology may trigger evolution of a behavioral phenotype in the opposite sex

Quantitative analysis of harmonic convergence in mosquito auditory interactions

This article analyses the hearing and behaviour of mosquitoes in the context of inter-individual acoustic interactions. The acoustic interactions of tethered live pairs of Aedes aegypti mosquitoes, from same and opposite sex mosquitoes of the species, are recorded on independent and unique audio channels, together with the response of tethered individual mosquitoes to playbacks of pre-recorded flight tones of lone or paired individuals. A time-dependent representation of each mosquito's non-stationary wing beat frequency signature is constructed, based on Hilbert spectral analysis. A range of algorithmic tools is developed to automatically analyse these data, and used to perform a robust quantitative identification of the 'harmonic convergence' phenomenon. The results suggest that harmonic convergence is an active phenomenon, which does not occur by chance. It occurs for live pairs, as well as for lone individuals responding to playback recordings, whether from the same or opposite sex. Male-female behaviour is dominated by frequency convergence at a wider range of harmonic combinations than previously reported, and requires participation from both partners in the duet. New evidence is found to show that male-male interactions are more varied than strict frequency avoidance. Rather, they can be divided into two groups: convergent pairs, typified by tightly bound wing beat frequencies, and divergent pairs, that remain widely spaced in the frequency domain. Overall, the results reveal that mosquito acoustic interaction is a delicate and intricate time-dependent active process that involves both individuals, takes place at many different frequencies, and which merits further enquiry

Adaptations to changes in the acoustic scene of the echolocating bat

Our natural environment is noisy and in order to navigate it successfully, we must filter out the important components so that we may guide our next steps. In analyzing our acoustic scene, one of the most common challenges is to segregate speech communication sounds from background noise; this process is not unique to humans. Echolocating bats emit high frequency biosonar signals and listen to echoes returning off objects in their environment. The sound wave they receive is a merging of echoes reflecting off target prey and other scattered objects, conspecific calls and echoes, and any naturally-occurring environmental noises. The bat is faced with the challenge of segregating this complex sound wave into the components of interest to adapt its flight and echolocation behavior in response to fast and dynamic environmental changes. In this thesis, we employ two approaches to investigate the mechanisms that may aid the bat in analyzing its acoustic scene. First, we test the bat’s adaptations to changes of controlled echo-acoustic flow patterns, similar to those it may encounter when flying along forest edges and among clutter. Our findings show that big brown bats adapt their flight paths in response to the intervals between echoes, and suggest that there is a limit to how close objects can be spaced, before the bat does not represent them as distinct any longer. Further, we consider how bats that use different echolocation signals may navigate similar environments, and provide evidence of species-specific flight and echolocation adaptations. Second, we research how temporal patterning of echolocation calls is affected during competitive foraging of paired bats in open and cluttered environments. Our findings show that “silent behavior”, the ceasing of emitting echolocation calls, which had previously been proposed as a mechanism to avoid acoustic interference, or to “eavesdrop” on another bat, may not be as common as has been reported

ORIENTING IN 3D SPACE: BEHAVIORAL AND NEUROPHYSIOLOGICAL STUDIES IN BIG BROWN BATS

In their natural environment, animals engage in a wide range of behavioral tasks that require them to orient to stimuli in three-dimensional space, such as navigating around obstacles, reaching for objects and escaping from predators. Echolocating bats, for example, have evolved a high-resolution 3D acoustic orienting system that allows them to localize and track small moving targets in azimuth, elevation and range. The bat’s active control over the features of its echolocation signals contributes directly to the information represented in its sonar receiver, and its adaptive adjustments in sonar signal design provide a window into the acoustic features that are important for different behavioral tasks. When bats inspect sonar objects and require accurate 3D localization of targets, they produce sonar sound groups (SSGs), which are clusters of sonar calls produced at short intervals and flanked by long interval calls. SSGs are hypothesized to enhance the bat’s range resolution, but this hypothesis has not been directly tested. We first, in Chapter 2, provide a comprehensive comparison of SSG production of bats flying in the field and in the lab under different environmental conditions. Further, in Chapter 3, we devise an experiment to specifically compare SSG production under conditions when target motion is predictable and unpredictable, with the latter mimicking natural conditions where bats chase erratically moving prey. Data from both of these studies are consistent with the hypothesis that SSGs improve the bat’s spatio-temporal resolution of target range, and provide a behavioral foundation for the analysis and interpretation of neural recording data in chapters 4 and 6. The complex orienting behaviors exhibited by animals can be understood as a feedback loop between sensing and action. A primary brain structure involved in sensorimotor integration is the midbrain superior colliculus (SC). The SC is a widely studied brain region and has been implicated in species-specific orienting behaviors. However, most studies of the SC have investigated its functional organization using synthetic 2D (azimuth and elevation) stimuli in restrained animals, leaving gaps in our knowledge of how 3D space (azimuth, elevation and distance) is represented in the CNS. In contrast, the representation of stimulus distance in the auditory systems of bats has been widely studied. Almost all of these studies have been conducted in passively listening bats, thus severing the loop between sensing and action and leaving gaps in our knowledge regarding how target distance is represented in the auditory system of actively echolocating bats. In chapters 4, 5 and 6, we attempt to fill gaps in our knowledge by recording from the SC of free flying echolocating bats engaged in a naturalistic navigation task where bats produce SSGs. In chapter 4, we provide a framework to compute time-of-arrival and direction of the instantaneous echo stimuli received at the bats ears. In chapters 5 and 6, we provide an algorithm to classify neural activity in the SC as sensory, sensorimotor and premotor and then compute spatial receptive fields of SC neurons. Our results show that neurons in the SC of the free-flying echolocating bat respond selectively to stimulus azimuth, elevation and range. Importantly, we find that SC neuron response profiles are modulated by the bat’s behavioral state, indicated by the production of SSG. Broadly, we use both behavior and electrophysiology to understand the action-perception loop that supports spatial orientation by echolocation. We believe that the results and methodological advances presented here will open doors to further studies of sensorimotor integration in freely behaving animals