Multimedia signal processing for behavioral quantification in neuroscience

While there have been great advances in quantification of the genotype of organisms, including full genomes for many species, the quantification of phenotype is at a comparatively primitive stage. Part of the reason is technical difficulty: the phenotype covers a wide range of characteristics, ranging from static morphological features, to dynamic behavior. The latter poses challenges that are in the area of multimedia signal processing. Automated analysis of video and audio recordings of animal and human behavior is a growing area of research, ranging from the behavioral phenotyping of genetically modified mice or drosophila to the study of song learning in birds and speech acquisition in human infants. This paper reviews recent advances and identifies key problems for a range of behavior experiments that use audio and video recording. This research area offers both research challenges and an application domain for advanced multimedia signal processing. There are a number of MMSP tools that now exist which are directly relevant for behavioral quantification, such as speech recognition, video analysis and more recently, wired and wireless sensor networks for surveillance. The research challenge is to adapt these tools and to develop new ones required for studying human and animal behavior in a high throughput manner while minimizing human intervention. In contrast with consumer applications, in the research arena there is less of a penalty for computational complexity, so that algorithmic quality can be maximized through the utilization of larger computational resources that are available to the biomedical researcher

Seeing sound: a new way to illustrate auditory objects and their neural correlates

This thesis develops a new method for time-frequency signal processing and examines the relevance of the new representation in studies of neural coding in songbirds. The method groups together associated regions of the time-frequency plane into objects defined by time-frequency contours. By combining information about structurally stable contour shapes over multiple time-scales and angles, a signal decomposition is produced that distributes resolution adaptively. As a result, distinct signal components are represented in their own most parsimonious forms.  Next, through neural recordings in singing birds, it was found that activity in song premotor cortex is significantly correlated with the objects defined by this new representation of sound. In this process, an automated way of finding sub-syllable acoustic transitions in birdsongs was first developed, and then increased spiking probability was found at the boundaries of these acoustic transitions. Finally, a new approach to study auditory cortical sequence processing more generally is proposed. In this approach, songbirds were trained to discriminate Morse-code-like sequences of clicks, and the neural correlates of this behavior were examined in primary and secondary auditory cortex. It was found that a distinct transformation of auditory responses to the sequences of clicks exists as information transferred from primary to secondary auditory areas. Neurons in secondary auditory areas respond asynchronously and selectively -- in a manner that depends on the temporal context of the click. This transformation from a temporal to a spatial representation of sound provides a possible basis for the songbird's natural ability to discriminate complex temporal sequences

Children at risk : their phonemic awareness development in holistic instruction

Includes bibliographical references (p. 17-19

Automated classification of humpback whale (Megaptera novaeangliae) songs using hidden Markov models

Humpback whales songs have been widely investigated in the past few decades. This study proposes a new approach for the classification of the calls detected in the songs with the use of Hidden Markov Models (HMMs). HMMs have been used once before for such task but in an unsupervised algorithm with promising results. Here HMMs were trained and two models were employed to classify the calls into their component units and subunits. The results show that classification of humpback whale songs from one year to another is possible even with limited training. The classification is fully automated apart from the labelling of the training set and the input of the initial HMM prototype models. Two different models for the song structure are considered: one based on song units and one based on subunits. The latter model is shown to achieve better recognition results with a reduced need for updating when applied to a variety of recordings from different years and different geographic locations

Mendeteksi Jenis Burung Berdasarkan Pola Suaranya

AbstrakIlmuwan biologi terutama di bidang biodifersitas, terus melakukan penelitian tentang spesies hewan yang ada di dunia. salah satu hewan yang spesiesnya memiliki banyak variasi adalah burung. Tiap jenis burung memiliki perbedaan-perbedaan, mulai dari bentuk anggota tubuhnya, prilakunya, makanannya hingga suaranya. Ilmuwan sering juga mengalami kesulitan untuk melakukan pengamatan di alam. Misalnya, untuk mengetahui spesies burung apa saja yang ada di suatu daerah, mereka harus hadir di suatu wilayah, dan menelusuri setiap pelosok. kadang kala kehadiran mereka di tempat tersebut dalam jangka waktu lama, malah mengusik burung yang ada, dan burung-burung malah pergi meninggalkan tempat, sebelum berhasil diamati. Salah satu cara untuk mendeteksi burung apa saja yang ada di suatu wilayah, tanpa harus mengusik keberadaan burung adalah dengan menggunakan alat bantu. Bisa dengan menggunakan kamera video untuk mengambil gambar lingkungan sekitar, atau dengan perekam suara, untuk merekam suara burung yang ada di sana. Untuk itu penelitian ini ditujukan untuk membuat sebuah pengklasifikasi suara burung secara otomatis. Fitur yang digunakan adalah rhythm, pitch, mean, varian, min, max, dan delta  dari suara burungnya. dari hasil klasifikasi 4 jenis burung, didapatkan hasil rata-rata akurasi terbaik sebesar 88.82%. Kata Kunci : suara burung, klasifikasi, rhythm, pitchAbstractMany of Biologi scientist, especially in the field of biodiversity, conduct research on the animal species that exist in the world. One of the animal which is largely diverse in species is bird. Each species of birds have differences, from the shape of his body, his behavior, his food to it's voice. Scientists often find it difficult to make observations in nature. For example, to determine which species of birds present in an area, they should be present in an area, and explore every corner. sometimes their presence in that place for a long time, even disturb the bird, and they leaving the place, before been observed. One way to detect any bird that is in an area, without having to disturb the presence of birds is to use the automatic tools. For example to use a video camera to take pictures of the surrounding environment, or with voice recorders to record the sound of the birds that were there. This study is aimed to create a classifier bird sound automatically. Features used are rhythm, pitch, mean, variance, min, max, and delta of the bird sound samples. of the results of the classification of four types of birds, showed the best average of accuracy is 88.82%. Key Word : bird song, classification, rhythm, pitch

Large-scale automated acoustic monitoring of birds and the challenges of field data

Modern technologies for the automated acoustic monitoring of animal communities enable species surveys that yield data in unprecedented volumes. Interpretation of these data bring new challenges related to the need of automated species identification. Coupling automated audio recording with automated species identification has enormous potential for biodiversity assessment studies, but it has posed many challenges to the effective use of techniques in real-world situations. This thesis develops new methods in the field of bioacoustics applied to automated monitoring of vocal species in terrestrial environments. Specifically, I developed automated methods to classify acoustic ecological data generated under the two most common contexts used in ecology: identification of vocalization data stored in acoustic libraries of sounds and identification of vocalizations in audio data collected from the field, through e.g., acoustic monitoring programs. The methods bring key developments across the entire pipeline for automated acoustical identification, connecting techniques from the data acquisition in the field to the ecological modelling of data identified utilizing automated classification methods. I show the performance of methods over huge datasets, compare them with alternative cutting-edge techniques and provide an ample study case of Amazonian bird communities to show the tools in practice. The methods in this thesis are available as open source and ready-to-use software capable to work directly on field data collected from acoustic monitoring efforts.Nykyaikaiset/modernit teknologiat/tekniikat eläinyhteiskuntien automattiseen akustiseen monitorointiin mahdollistavat lajitutkimuksen, joka tuottaa ennennäkemättömän määrän tutkimusaineistoa. Tällaisen tutkimusaineiston tulkinta aiheuttaa uusia haasteita (kuten) tarpeen automatisoituun lajitunnistukseen. Automatisoitu audiotallennus yhdistettynä automaattiseen lajitunnistukseen luo uusia mahdollisuuksia biodiversiteetin inventointiin/ luontotyyppien seurantatutkimukseen, mutta ne ovat myös aiheuttaneet monia haasteita menetelmän käyttämiseen todellisissa tilanteissa. Tämä tutkimus kehittää uusia menetelmiä maalla elävien ääntelevien lajien automaattiseen seurantaan bioakustiikan tutkimuksen kentälle. Kehitin ennenkaikkea automatisoituja menetelmiä kahden tyypillisimmän akustisessa ekologiassa käytetyn aineiston; lajiäänitteiden tietokantojen sekä lajiäänitteiden maastoaineiston tallenteiden, luokitteluun. Nämä menetelmät kehittävät olennaisesti koko automatisoidun akustisen tunnistuksen kenttää yhdistäen maastoaineiston automatisoidun keruun automaattisten luokitusmenetelmien avulla tunnistettujen tietojen ekologiseen mallintamiseen. Osoitan menetelmien toimivuuden (käytännössä) erittäin suurten aineistojen avulla vertaillen niitä tämänhetkisiin huipputekniikoihin sekä tarjoan laajan Amazonin lintuyhdyskuntia koskevan tapaustutkimuksen/tutkimusesimerkin osoittaen näin välineiden/menetelmien toimivuuden käytännössä. Tutkimuksessa tuotetut menetelmät ovat saatavilla avoimen lähdekoodin sekä käyttöönotettavan/toimivan ohjelmiston muodossa maastoaineiston käsittelyä varten

Musical Regularity And Rhythmic Patterns: A Quantitative Analysis Of Birdsong Structure

Birdsong is a complex, learned behavior that, like music, has meaningful units at multiple timescales. Birds perform by constructing extended presentations of their phrase repertoire. Each bird\u27s repertoire is built from small units, such as syllables, or groups of syllables with characteristic pitch, rhythm, and timbre. Like a musician each bird has its unique structure of performance that communicates its individual identity. Also contained within a bird\u27s performance, is information about its group identity and species identity. Like a musician\u27s performance, a bird\u27s singing affects the behavioral state of listeners\u27birds perform to attract mates and defend territory. Subjectively, many can appreciate birdsong as musical but what evidence is there that birds have music? What parameters can be chosen to test the presence of musicality in birdsong? Are there quantitative ways to demonstrate musicality in birdsong? In this study I test quantitatively for the presence of musical structure in birdsong by homing in on two distinct features: structural balance and groove. Music is known for its characteristic balance between complexity and regularity. Groove, in the context of genres such as jazz offers a unique, visceral parameter that is known to vary in nuanced ways. I test for musical features based on understanding of how these two parameters manifest in music. Like music, birdsong affects the behavioral state of conspecifics, but what is it in the acoustic signal that serves to affect the behavioral state of bird listeners in a desired manner? By investigating extensive song databases of birds\u27 singing performances, I developed methods that facilitate a deeper understanding of what structures are present within song performances and why they may arise. A key feature of these methods is the capacity for multimodal data processing, as well as analysis at micro and macro levels simultaneously. This facilitates an understanding of the relationship between units and performance level structure. I studied two species to test for the presence of musicality within their vocalizations. In the Australian pied butcherbird I investigated temporal regularity in phrase types and demonstrated a characteristic balance analogous to that found in music. In the thrush nightingale I studied regularity in song rhythms and found that performance nuances used in groove rhythms follow similar principles in the context of music and birdsong alike. Australian pied butcherbird song phrases are built from the rearrangement of shared motifs (syllables or stereotyped groupings of notes). If the function of these motifs is to increase the repertoire of different phrase types, then transition probabilities between phrase types should capture most of the structure of singing performances. Alternatively, phrase types can be seen as varied presentations of shared themes, as often is the case in music. If this is the case, temporal regularity in performing shared motifs should be observed beyond phrase types, as if the transitions between phrases are designed to \u27organize\u27 those motifs over longer time scales. I tested which of those two views can explain more statistical regularity during entire singing performances of wild Australian pied butcherbirds, including thousands of song syllables recorded without interruption for each bird. I found that all birds produced several highly stereotyped phrase types. Most phrase types produced by each bird had shared motifs. Throughout the performance, the temporal gap between a motif\u27s reappearance was much more regular than what was expected by chance. In contrast, regularity in the performance of phrase types was much weaker. I developed a statistical estimate of the extent to which transition probabilities between phrase types are \u27optimized\u27 to maximize regularity in the repetition of shared motifs. I found that the phrase-types syntax is selective in achieving a regular repetition of shared motifs over the entire singing performance of the bird. This effect was stronger in birds with a richer song repertoire, suggesting the intriguing possibility that birds may regulate the temporal diversity of dominant themes in their singing performance in a manner that takes their repertoire size into account. The thrush nightingale is a distant relative of the pied butcherbird so it would be surprising to find similarities in the deep structure of the two species. I test whether or not thrush nightingales distribute motifs throughout a performance uniformly as butcherbirds do. I found that thrush nightingales exhibit more regularity in their distribution of phrase types than what is expected from chance. However, I failed to find a distribution of motif types that was balanced against repertoire size. The thrush nightingale ends many of its song phrases with buzzes (or rattles). Upon closer inspection these buzzes emerge from a diversity of repetitive rhythmic patterns of clicks. These clicks are repeated at a regular pace, or in rhythmic groups of two, three, and four or more and they sound like the complex grooves of a jazz drummer. I tested whether or not these patterns contain timing relationships that coincide with small integer ratios and found a no significant bias for small integer ratios. I tested whether or not the range of rhythmic ratios used could be explained by any systematic trend. I tested whether or not thrush nightingales, like jazz drummers adjust their \u27swing ratio\u27 according to tempo. Swing ratio is a term that describes the non-isochronous manner in which jazz musicians interpret eighth note rhythms, using a \u27long-short\u27 pattern instead of equal timing between beats. Jazz drummers tend to use a longer long segment at slow tempos and more even segments at fast tempos. I found that thrush nightingales have a significant tendency to adjust the swing ratio in the same manner

The effectiveness of a multi-sensory phonological awareness and letter knowledge training programme for disadvantaged first graders

The study aimed to determine the effectiveness of a multi-sensory phonological awareness and letter knowledge programme for disadvantaged first graders. One control group and one experimental group, each consisting of 20 children, were matched for age, gender, school readiness, socio-economic status and phonological awareness. Twenty-nine sessions of phonological awareness and letter knowledge training were administered to the experimental group while the control group received vocabulary stimulation activities for the same length of time. Results indicated that the programme was highly effective in improving phonological awareness, letter knowledge, reading and spelling skills. The experimental group scored significantly higher than the control group on simple phonological awareness tasks such as segmenting the sounds in a word, letter knowledge and in their ability to read and spell real and pseudowords. The results are discussed in terms of the importance of both phonological awareness and letter knowledge in the process of literacy acquisition