Inferring Room Semantics Using Acoustic Monitoring

Having knowledge of the environmental context of the user i.e. the knowledge of the users' indoor location and the semantics of their environment, can facilitate the development of many of location-aware applications. In this paper, we propose an acoustic monitoring technique that infers semantic knowledge about an indoor space \emph{over time,} using audio recordings from it. Our technique uses the impulse response of these spaces as well as the ambient sounds produced in them in order to determine a semantic label for them. As we process more recordings, we update our \emph{confidence} in the assigned label. We evaluate our technique on a dataset of single-speaker human speech recordings obtained in different types of rooms at three university buildings. In our evaluation, the confidence\emph{ }for the true label generally outstripped the confidence for all other labels and in some cases converged to 100\% with less than 30 samples.Comment: 2017 IEEE International Workshop on Machine Learning for Signal Processing, Sept.\ 25--28, 2017, Tokyo, Japa

Audio-Based Semantic Concept Classification for Consumer Video

This paper presents a novel method for automatically classifying consumer video clips based on their soundtracks. We use a set of 25 overlapping semantic classes, chosen for their usefulness to users, viability of automatic detection and of annotator labeling, and sufficiency of representation in available video collections. A set of 1873 videos from real users has been annotated with these concepts. Starting with a basic representation of each video clip as a sequence of mel-frequency cepstral coefficient (MFCC) frames, we experiment with three clip-level representations: single Gaussian modeling, Gaussian mixture modeling, and probabilistic latent semantic analysis of a Gaussian component histogram. Using such summary features, we produce support vector machine (SVM) classifiers based on the Kullback-Leibler, Bhattacharyya, or Mahalanobis distance measures. Quantitative evaluation shows that our approaches are effective for detecting interesting concepts in a large collection of real-world consumer video clips

Studies on binaural and monaural signal analysis methods and applications

Sound signals can contain a lot of information about the environment and the sound sources present in it. This thesis presents novel contributions to the analysis of binaural and monaural sound signals. Some new applications are introduced in this work, but the emphasis is on analysis methods. The three main topics of the thesis are computational estimation of sound source distance, analysis of binaural room impulse responses, and applications intended for augmented reality audio. A novel method for binaural sound source distance estimation is proposed. The method is based on learning the coherence between the sounds entering the left and right ears. Comparisons to an earlier approach are also made. It is shown that these kinds of learning methods can correctly recognize the distance of a speech sound source in most cases. Methods for analyzing binaural room impulse responses are investigated. These methods are able to locate the early reflections in time and also to estimate their directions of arrival. This challenging problem could not be tackled completely, but this part of the work is an important step towards accurate estimation of the individual early reflections from a binaural room impulse response. As the third part of the thesis, applications of sound signal analysis are studied. The most notable contributions are a novel eyes-free user interface controlled by finger snaps, and an investigation on the importance of features in audio surveillance. The results of this thesis are steps towards building machines that can obtain information on the surrounding environment based on sound. In particular, the research into sound source distance estimation functions as important basic research in this area. The applications presented could be valuable in future telecommunications scenarios, such as augmented reality audio

단일 음향 센서를 사용하는 데이터 기반 다층 철근 콘크리트 건물 내 소음의 종류와 위치 추정

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 조선해양공학과, 2021. 2. 성우제.The construction of multi-story residential buildings triggers indoor noise. Indoor noise in residential areas has been investigated to ascertain the effect of noise on occupants and to improve their quality of life. In buildings, indoor acoustic noise transmitted from various sources travels through these structures and exerts an unpleasant effect on occupants. Inter-floor noise is identified as a severe type of indoor noise in residential areas. The identification of noise is considered a fundamental step that is essential for studying the challenges of noise pollution. By harnessing a sound level meter, long-term measurement, and site surveying, previous studies have been conducted on the identification of noise in residential areas to estimate the level, type, and position of generated noise. However, it is challenging to identify the source type and position of noise travelling through multi-story residential buildings owing to the difficulty of the human ear in intercepting these sounds. Recent studies on the identification of indoor noise are limited to noise sources and receivers on a single level of the floor, and they require multiple sensor channels to determine the time difference of arrival. Residential buildings, which are usually reinforced concrete structures, are considered to be concrete, steel, and fluid-mixed media with high structural complexity and occupants that have insufficient knowledge of the details of their properties. In this study, we propose a data-driven identification of noise in reinforced concrete buildings via the learning-based localization method using a single sensor. Actual experiments were conducted in a campus building, as well as two apartment buildings. Performance was analyzed according to several source types and positions that apply the deep convolutional neural network (CNN)-based supervised learning. The validations against the datasets obtained in three buildings verified the generalizability of the proposed method. In addition, noise identification data transferred within different floor sections in a single building and between similar buildings were presented in this study. Although indoor noise identification is emphasized in this work, the proposed method can be beneficial for other noise identification methods that employ a single sensor.공동주택의 증가로 건물 내 이웃 간의 소음 문제가 사회적으로 대두되고 있다. 거주자에게 노출된 소음은 거주자의 건강 문제에 직결될 수도 있으므로 건물 내 소음에 관한 여러 연구가 진행되어 왔다. 다층 건물 내에서 발생한 소음은 건물의 구조를 따라 다른 층으로 전달되며 이러한 층간소음은 주변 이웃에게 고통으로 다가올 수 있다. 소음원의 규명은 소음을 다룰 때 선행되어야 하는 바 건물 내 소음의 준위, 종류, 위치 파악에 관련된 연구들이 진행되어 왔다. 소음의 준위는 소음측정기를 사용하여 측정이 가능하나 건물의 구조를 따라 전달된 소음의 종류와 위치를 판별하는 것은 추정이 필요한 문제이며 사람의 청력에 의존하여서 풀기도 어렵다. 최근 연구된 관련 연구를 살펴보면 건물 내 소음의 종류를 분류하는 연구는 거의 다뤄지지 않았고, 소음원 위치 추정 연구의 경우 동일 층에 소음원과 여러 채널의 수신기가 위치한 경우를 다중측량 (multilateration) 을 통하여 제한적으로 다뤘다. 일반적으로 현대 거주용 건축물의 대부분은 철근 콘크리트 구조이며 층간의 소음 전달 환경은 콘크리트, 철근, 유체가 혼재하는 복잡한 환경이다. 일반인 거주자가 이러한 환경에서의 소음 전달 환경을 파악하고 소음의 전달 모델을 세워 소음을 규명하는 것은 어렵다. 본 논문은 모바일 장치 (mobile device) 의 단일 음향 센서로 측정한 소음과 합성곱 신경망을 활용하여 데이터 기반 (data-driven) 의 건물 내 소음 규명 방법을 제안하고 한 개의 캠퍼스 건물과 두 아파트 건물에서 진행한 실험을 통하여 이 기법의 유용성과 보편성을 보였다. 또한 한 층간에서 학습한 소음 규명 지식을 동일 건물의 다른 층간에서의 소음 규명에, 한 건물에서 학습한 소음 규명 지식을 다른 건물 내 소음 규명에 활용 할 수 있음을 보였다. 제안하는 기법은 소음 전달 환경 파악 및 모델을 얻기 어려운 분야에서의 적용에도 유용할 것으로 기대한다.Abstract I Contents iii List of Figures vi List of Tables ix 1 Introduction 2 1.1 Backgrounds 2 1.2 Approach 5 1.2.1 Data-driven noise identification 5 1.2.2 Source type classification and localization 7 1.2.3 Knowledge transfer 10 1.3 Contributions 15 1.4 Outline of the Dissertation 16 2 Source type classification and localization of acoustic noises in a reinforced concrete structure 28 2.1 Introduction 29 2.1.1 Motivation 29 2.1.2 Related literature 29 2.1.3 Approach 30 2.1.4 Contributions of this chapter 31 2.2 Campus building inter-floor noise dataset 32 2.2.1 Selecting source type and source position 32 2.2.2 Generating and collecting inter-floor noise 33 2.3 Supervised learning of inter-floor noises 36 2.3.1 Convolutional neural networks for acoustic scene classification 36 2.3.2 Network architecture 36 2.3.3 Evaluation 40 2.3.4 Source type classification results 41 2.3.5 Localizationresults....................... 41 2.4 Source type classification and localization of inter-floor noises generated on unlearned positions 47 2.4.1 Source type classification of inter-floor noises from unlearned positions 48 2.4.2 Localization of inter-floor noises from unlearned positions 50 2.5 Summary 52 2.6 Acknowledgments 53 3 Knowledge transfer between reinforced concrete structures 61 3.1 Introduction 62 3.1.1 Motivation 62 3.1.2 Related Literature 62 3.1.3 Approach 63 3.1.4 Contributions of this chapter 63 3.2 Apartment building inter-floor noise dataset 64 3.3 Inter-floor noise classification 70 3.3.1 Onset detection 70 3.3.2 Convolutional neural network-based classifier 71 3.3.3 Network training 75 3.3.4 Source type classification and localization tasks 75 3.4 Performance Evaluation 79 3.4.1 Source type classification results in a single apartment building 79 3.4.2 Localization results in a single apartment building 80 3.4.3 Results of knowledge transfer between the apartment buildings 81 3.5 Summary 87 3.6 Acknowledgments 94 4 Conclusions 96 4.1 Findings and limitations 97 4.2 Applications 97 4.2.1 Marine structures 98 4.2.2 Mobile application 98 4.3 Future study 100 4.3.1 Learning with building structure representation 100 4.3.2 Learning with data measured at multiple receiver locations 100 4.3.3 Task oriented algorithm 101 A Precision, recall, and F1 score of the classification results 102 B Data analysis 105 C Using a one-dimensional convolutional neural network and feature visualization 112 Abstract (In Korean) 124Docto

AI augmented Edge and Fog computing: trends and challenges

In recent years, the landscape of computing paradigms has witnessed a gradual yet remarkable shift from monolithic computing to distributed and decentralized paradigms such as Internet of Things (IoT), Edge, Fog, Cloud, and Serverless. The frontiers of these computing technologies have been boosted by shift from manually encoded algorithms to Artificial Intelligence (AI)-driven autonomous systems for optimum and reliable management of distributed computing resources. Prior work focuses on improving existing systems using AI across a wide range of domains, such as efficient resource provisioning, application deployment, task placement, and service management. This survey reviews the evolution of data-driven AI-augmented technologies and their impact on computing systems. We demystify new techniques and draw key insights in Edge, Fog and Cloud resource management-related uses of AI methods and also look at how AI can innovate traditional applications for enhanced Quality of Service (QoS) in the presence of a continuum of resources. We present the latest trends and impact areas such as optimizing AI models that are deployed on or for computing systems. We layout a roadmap for future research directions in areas such as resource management for QoS optimization and service reliability. Finally, we discuss blue-sky ideas and envision this work as an anchor point for future research on AI-driven computing systems