Audio Caption: Listen and Tell

Increasing amount of research has shed light on machine perception of audio events, most of which concerns detection and classification tasks. However, human-like perception of audio scenes involves not only detecting and classifying audio sounds, but also summarizing the relationship between different audio events. Comparable research such as image caption has been conducted, yet the audio field is still quite barren. This paper introduces a manually-annotated dataset for audio caption. The purpose is to automatically generate natural sentences for audio scene description and to bridge the gap between machine perception of audio and image. The whole dataset is labelled in Mandarin and we also include translated English annotations. A baseline encoder-decoder model is provided for both English and Mandarin. Similar BLEU scores are derived for both languages: our model can generate understandable and data-related captions based on the dataset.Comment: accepted by ICASSP201

A Comprehensive Survey of Automated Audio Captioning

Automated audio captioning, a task that mimics human perception as well as innovatively links audio processing and natural language processing, has overseen much progress over the last few years. Audio captioning requires recognizing the acoustic scene, primary audio events and sometimes the spatial and temporal relationship between events in an audio clip. It also requires describing these elements by a fluent and vivid sentence. Deep learning-based approaches are widely adopted to tackle this problem. This current paper situates itself as a comprehensive review covering the benchmark datasets, existing deep learning techniques and the evaluation metrics in automated audio captioning

Strategies for Improving Chinese Writing Exercises of Senior Grade Students in the Primary School of the Yi Ethnic Group

There has been a strong consensus among previous researchers that first language interferes with second language acquisition. A limited number of studies have been conducted to examine this interference in writing Chinese. Chinese writing exercises for primary school students of the Yi ethnic group in rural areas tend to encourage students to incorporate their native language habits into Chinese writing practices. This enables students to apply modern Chinese word order to the Yi language paradigm. Consequently, they are prone to grammatical errors, word order errors, and collocation errors in their compositions. This study focuses on elementary school students' writing abilities in order to alleviate their fears associated with Chinese writing. It also assists them in grasping the second language writing paradigm, and enjoying writing. In addition, by analyzing the composition errors of the students in the corpus, we propose some practical recommendations from three perspectives: educators, students, and institutions. These recommendations will enhance language and writing education within ethnic communities through theoretical and practical strategies

Enhance Temporal Relations in Audio Captioning with Sound Event Detection

Automated audio captioning aims at generating natural language descriptions for given audio clips, not only detecting and classifying sounds, but also summarizing the relationships between audio events. Recent research advances in audio captioning have introduced additional guidance to improve the accuracy of audio events in generated sentences. However, temporal relations between audio events have received little attention while revealing complex relations is a key component in summarizing audio content. Therefore, this paper aims to better capture temporal relationships in caption generation with sound event detection (SED), a task that locates events' timestamps. We investigate the best approach to integrate temporal information in a captioning model and propose a temporal tag system to transform the timestamps into comprehensible relations. Results evaluated by the proposed temporal metrics suggest that great improvement is achieved in terms of temporal relation generation