The Impact of Acoustic Imaging Geometry on the Fidelity of Seabed Bathymetric Models

Attributes derived from digital bathymetric models (DBM) are a powerful means of analyzing seabed characteristics. Those models however are inherently constrained by the method of seabed sampling. Most bathymetric models are derived by collating a number of discrete corridors of multibeam sonar data. Within each corridor the data are collected over a wide range of distances, azimuths and elevation angles and thus the quality varies significantly. That variability therefore becomes imprinted into the DBM. Subsequent users of the DBM, unfamiliar with the original acquisition geometry, may potentially misinterpret such variability as attributes of the seabed. This paper examines the impact on accuracy and resolution of the resultant derived model as a function of the imaging geometry. This can be broken down into the range, angle, azimuth, density and overlap attributes. These attributes in turn are impacted by the sonar configuration including beam widths, beam spacing, bottom detection algorithms, stabilization strategies, platform speed and stability. Superimposed over the imaging geometry are residual effects due to imperfect integration of ancillary sensors. As the platform (normally a surface vessel), is moving with characteristic motions resulting from the ocean wave spectrum, periodic residuals in the seafloor can become imprinted that may again be misinterpreted as geomorphological information

신체 임베딩을 활용한 오토인코더 기반 컴퓨터 비전 모형의 성능 개선

학위논문(박사) -- 서울대학교대학원 : 공과대학 산업공학과, 2021.8. 박종헌.Deep learning models have dominated the field of computer vision, achieving state-of-the-art performance in various tasks. In particular, with recent increases in images and videos of people being posted on social media, research on computer vision tasks for analyzing human visual information is being used in various ways. This thesis addresses classifying fashion styles and measuring motion similarity as two computer vision tasks related to humans. In real-world fashion style classification problems, the number of samples collected for each style class varies according to the fashion trend at the time of data collection, resulting in class imbalance. In this thesis, to cope with this class imbalance problem, generalized few-shot learning, in which both minority classes and majority classes are used for learning and evaluation, is employed. Additionally, the modalities of the foreground images, cropped to show only the body and fashion item parts, and the fashion attribute information are reflected in the fashion image embedding through a variational autoencoder. The K-fashion dataset collected from a Korean fashion shopping mall is used for the model training and evaluation. Motion similarity measurement is used as a sub-module in various tasks such as action recognition, anomaly detection, and person re-identification; however, it has attracted less attention than the other tasks because the same motion can be represented differently depending on the performer's body structure and camera angle. The lack of public datasets for model training and evaluation also makes research challenging. Therefore, we propose an artificial dataset for model training, with motion embeddings separated from the body structure and camera angle attributes for training using an autoencoder architecture. The autoencoder is designed to generate motion embeddings for each body part to measure motion similarity by body part. Furthermore, motion speed is synchronized by matching patches performing similar motions using dynamic time warping. The similarity score dataset for evaluation was collected through a crowdsourcing platform utilizing videos of NTU RGB+D 120, a dataset for action recognition. When the proposed models were verified with each evaluation dataset, both outperformed the baselines. In the fashion style classification problem, the proposed model showed the most balanced performance, without bias toward either the minority classes or the majority classes, among all the models. In addition, In the motion similarity measurement experiments, the correlation coefficient of the proposed model to the human-measured similarity score was higher than that of the baselines.컴퓨터 비전은 딥러닝 학습 방법론이 강점을 보이는 분야로, 다양한 태스크에서 우수한 성능을 보이고 있다. 특히, 사람이 포함된 이미지나 동영상을 딥러닝을 통해 분석하는 태스크의 경우, 최근 소셜 미디어에 사람이 포함된 이미지 또는 동영상 게시물이 늘어나면서 그 활용 가치가 높아지고 있다. 본 논문에서는 사람과 관련된 컴퓨터 비전 태스크 중 패션 스타일 분류 문제와 동작 유사도 측정에 대해 다룬다. 패션 스타일 분류 문제의 경우, 데이터 수집 시점의 패션 유행에 따라 스타일 클래스별 수집되는 샘플의 양이 달라지기 때문에 이로부터 클래스 불균형이 발생한다. 본 논문에서는 이러한 클래스 불균형 문제에 대처하기 위하여, 소수 샘플 클래스와 다수 샘플 클래스를 학습 및 평가에 모두 사용하는 일반화된 퓨샷러닝으로 패션 스타일 분류 문제를 설정하였다. 또한 변분 오토인코더 기반의 모델을 통해, 신체 및 패션 아이템 부분만 잘라낸 전경 이미지 모달리티와 패션 속성 정보 모달리티가 패션 이미지의 임베딩 학습에 반영되도록 하였다. 학습 및 평가를 위한 데이터셋으로는 한국 패션 쇼핑몰에서 수집된 K-fashion 데이터셋을 사용하였다. 한편, 동작 유사도 측정은 행위 인식, 이상 동작 감지, 사람 재인식 같은 다양한 분야의 하위 모듈로 활용되고 있지만 그 자체가 연구된 적은 많지 않은데, 이는 같은 동작을 수행하더라도 신체 구조 및 카메라 각도에 따라 다르게 표현될 수 있다는 점으로 부터 기인한다. 학습 및 평가를 위한 공개 데이터셋이 많지 않다는 점 또한 연구를 어렵게 하는 요인이다. 따라서 본 논문에서는 학습을 위한 인공 데이터셋을 수집하여 오토인코더 구조를 통해 신체 구조 및 카메라 각도 요소가 분리된 동작 임베딩을 학습하였다. 이때, 각 신체 부위별로 동작 임베딩을 생성할 수 있도록하여 신체 부위별로 동작 유사도 측정이 가능하도록 하였다. 두 동작 사이의 유사도를 측정할 때에는 동적 시간 워핑 기법을 사용, 비슷한 동작을 수행하는 구간끼리 정렬시켜 유사도를 측정하도록 함으로써, 동작 수행 속도의 차이를 보정하였다. 평가를 위한 유사도 점수 데이터셋은 행위 인식 데이터셋인 NTU-RGB+D 120의 영상을 활용하여 크라우드 소싱 플랫폼을 통해 수집되었다. 두 가지 태스크의 제안 모델을 각각의 평가 데이터셋으로 검증한 결과, 모두 비교 모델 대비 우수한 성능을 기록하였다. 패션 스타일 분류 문제의 경우, 모든 비교군에서 소수 샘플 클래스와 다수 샘플 클래스 중 한 쪽으로 치우치지 않는 가장 균형잡힌 추론 성능을 보여주었고, 동작 유사도 측정의 경우 사람이 측정한 유사도 점수와 상관계수에서 비교 모델 대비 더 높은 수치를 나타내었다.Chapter 1 Introduction 1 1.1 Background and motivation 1 1.2 Research contribution 5 1.2.1 Fashion style classication 5 1.2.2 Human motion similarity 9 1.2.3 Summary of the contributions 11 1.3 Thesis outline 13 Chapter 2 Literature Review 14 2.1 Fashion style classication 14 2.1.1 Machine learning and deep learning-based approaches 14 2.1.2 Class imbalance 15 2.1.3 Variational autoencoder 17 2.2 Human motion similarity 19 2.2.1 Measuring the similarity between two people 19 2.2.2 Human body embedding 20 2.2.3 Datasets for measuring the similarity 20 2.2.4 Triplet and quadruplet losses 21 2.2.5 Dynamic time warping 22 Chapter 3 Fashion Style Classication 24 3.1 Dataset for fashion style classication: K-fashion 24 3.2 Multimodal variational inference for fashion style classication 28 3.2.1 CADA-VAE 31 3.2.2 Generating multimodal features 33 3.2.3 Classier training with cyclic oversampling 36 3.3 Experimental results for fashion style classication 38 3.3.1 Implementation details 38 3.3.2 Settings for experiments 42 3.3.3 Experimental results on K-fashion 44 3.3.4 Qualitative analysis 48 3.3.5 Eectiveness of the cyclic oversampling 50 Chapter 4 Motion Similarity Measurement 53 4.1 Datasets for motion similarity 53 4.1.1 Synthetic motion dataset: SARA dataset 53 4.1.2 NTU RGB+D 120 similarity annotations 55 4.2 Framework for measuring motion similarity 58 4.2.1 Body part embedding model 58 4.2.2 Measuring motion similarity 67 4.3 Experimental results for measuring motion similarity 68 4.3.1 Implementation details 68 4.3.2 Experimental results on NTU RGB+D 120 similarity annotations 72 4.3.3 Visualization of motion latent clusters 78 4.4 Application 81 4.4.1 Real-world application with dancing videos 81 4.4.2 Tuning similarity scores to match human perception 87 Chapter 5 Conclusions 89 5.1 Summary and contributions 89 5.2 Limitations and future research 91 Appendices 93 Chapter A NTU RGB+D 120 Similarity Annotations 94 A.1 Data collection 94 A.2 AMT score analysis 95 Chapter B Data Cleansing of NTU RGB+D 120 Skeletal Data 100 Chapter C Motion Sequence Generation Using Mixamo 102 Bibliography 104 국문초록 123박

Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources

We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible. It particularly allows questions to be asked about the contents of an image, even when the image itself does not contain the whole answer. The method constructs a textual representation of the semantic content of an image, and merges it with textual information sourced from a knowledge base, to develop a deeper understanding of the scene viewed. Priming a recurrent neural network with this combined information, and the submitted question, leads to a very flexible visual question answering approach. We are specifically able to answer questions posed in natural language, that refer to information not contained in the image. We demonstrate the effectiveness of our model on two publicly available datasets, Toronto COCO-QA and MS COCO-VQA and show that it produces the best reported results in both cases.Comment: Accepted to IEEE Conf. Computer Vision and Pattern Recognitio