Towards End-to-end Car License Plate Location and Recognition in Unconstrained Scenarios

Benefiting from the rapid development of convolutional neural networks, the performance of car license plate detection and recognition has been largely improved. Nonetheless, challenges still exist especially for real-world applications. In this paper, we present an efficient and accurate framework to solve the license plate detection and recognition tasks simultaneously. It is a lightweight and unified deep neural network, that can be optimized end-to-end and work in real-time. Specifically, for unconstrained scenarios, an anchor-free method is adopted to efficiently detect the bounding box and four corners of a license plate, which are used to extract and rectify the target region features. Then, a novel convolutional neural network branch is designed to further extract features of characters without segmentation. Finally, recognition task is treated as sequence labelling problems, which are solved by Connectionist Temporal Classification (CTC) directly. Several public datasets including images collected from different scenarios under various conditions are chosen for evaluation. A large number of experiments indicate that the proposed method significantly outperforms the previous state-of-the-art methods in both speed and precision

Vehicle make and model recognition for intelligent transportation monitoring and surveillance.

Vehicle Make and Model Recognition (VMMR) has evolved into a significant subject of study due to its importance in numerous Intelligent Transportation Systems (ITS), such as autonomous navigation, traffic analysis, traffic surveillance and security systems. A highly accurate and real-time VMMR system significantly reduces the overhead cost of resources otherwise required. The VMMR problem is a multi-class classification task with a peculiar set of issues and challenges like multiplicity, inter- and intra-make ambiguity among various vehicles makes and models, which need to be solved in an efficient and reliable manner to achieve a highly robust VMMR system. In this dissertation, facing the growing importance of make and model recognition of vehicles, we present a VMMR system that provides very high accuracy rates and is robust to several challenges. We demonstrate that the VMMR problem can be addressed by locating discriminative parts where the most significant appearance variations occur in each category, and learning expressive appearance descriptors. Given these insights, we consider two data driven frameworks: a Multiple-Instance Learning-based (MIL) system using hand-crafted features and an extended application of deep neural networks using MIL. Our approach requires only image level class labels, and the discriminative parts of each target class are selected in a fully unsupervised manner without any use of part annotations or segmentation masks, which may be costly to obtain. This advantage makes our system more intelligent, scalable, and applicable to other fine-grained recognition tasks. We constructed a dataset with 291,752 images representing 9,170 different vehicles to validate and evaluate our approach. Experimental results demonstrate that the localization of parts and distinguishing their discriminative powers for categorization improve the performance of fine-grained categorization. Extensive experiments conducted using our approaches yield superior results for images that were occluded, under low illumination, partial camera views, or even non-frontal views, available in our real-world VMMR dataset. The approaches presented herewith provide a highly accurate VMMR system for rea-ltime applications in realistic environments.\\ We also validate our system with a significant application of VMMR to ITS that involves automated vehicular surveillance. We show that our application can provide law inforcement agencies with efficient tools to search for a specific vehicle type, make, or model, and to track the path of a given vehicle using the position of multiple cameras

An Adaptive Approach for Multi-National Vehicle License Plate Recognition Using Multi-Level Deep Features and Foreground Polarity Detection Model

License plate recognition system (LPR) plays a vital role in intelligent transport systems to build up smart environments. Numerous country specific methods have been proposed successfully for an LPR system, but there is a need to find a generalized solution that is independent of license plate layout. The proposed architecture is comprised of two important LPR stages: (i) License plate character segmentation (LPCS) and (ii) License plate character recognition (LPCR). A foreground polarity detection model is proposed by using a Red-Green-Blue (RGB) channel-based color map in order to segment and recognize the LP characters effectively at both LPCS and LPCR stages respectively. Further, a multi-channel CNN framework with layer aggregation module is proposed to extract deep features, and support vector machine is used to produce target labels. Multi-channel processing with merged features from different-level convolutional layers makes output feature map more expressive. Experimental results show that the proposed method is capable of achieving high recognition rate for multinational vehicles license plates under various illumination conditions. Document type: Articl

Learning to grasp in unstructured environments with deep convolutional neural networks using a Baxter Research Robot

Recent advancements in Deep Learning have accelerated the capabilities of robotic systems in terms of visual perception, object manipulation, automated navigation, and human-robot collaboration. The capability of a robotic system to manipulate objects in unstructured environments is becoming an increasingly necessary skill. Due to the dynamic nature of these environments, traditional methods, that require expert human knowledge, fail to adapt automatically. After reviewing the relevant literature a method was proposed to utilise deep transfer learning techniques to detect object grasps from coloured depth images. A grasp describes how a robotic end-effector can be arranged to securely grasp an object and successfully lift it without slippage. In this study, a ResNet-50 convolutional neural network (CNN) model is trained on the Cornell grasp dataset. The training was completed within 30 hours using a workstation PC with accelerated GPU support via an NVIDIA Titan X. The trained grasp detection model was further evaluated with a Baxter research robot and a Microsoft Kinect-v2 and a successful grasp detection accuracy of 93.91% was achieved on a diverse set of novel objects. Physical grasping trials were conducted on a set of 8 different objects. The overall system achieves an average grasp success rate of 65.0% while performing the grasp detection in under 25 milliseconds. The results analysis concluded that the objects with reasonably straight edges and moderately pronounced heights above the table are easily detected and grasped by the system

Automated license plate recognition: a survey on methods and techniques

With the explosive growth in the number of vehicles in use, automated license plate recognition (ALPR) systems are required for a wide range of tasks such as law enforcement, surveillance, and toll booth operations. The operational specifications of these systems are diverse due to the differences in the intended application. For instance, they may need to run on handheld devices or cloud servers, or operate in low light and adverse weather conditions. In order to meet these requirements, a variety of techniques have been developed for license plate recognition. Even though there has been a notable improvement in the current ALPR methods, there is a requirement to be filled in ALPR techniques for a complex environment. Thus, many approaches are sensitive to the changes in illumination and operate mostly in daylight. This study explores the methods and techniques used in ALPR in recent literature. We present a critical and constructive analysis of related studies in the field of ALPR and identify the open challenge faced by researchers and developers. Further, we provide future research directions and recommendations to optimize the current solutions to work under extreme conditions

단일 음향 센서를 사용하는 데이터 기반 다층 철근 콘크리트 건물 내 소음의 종류와 위치 추정

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 조선해양공학과, 2021. 2. 성우제.The construction of multi-story residential buildings triggers indoor noise. Indoor noise in residential areas has been investigated to ascertain the effect of noise on occupants and to improve their quality of life. In buildings, indoor acoustic noise transmitted from various sources travels through these structures and exerts an unpleasant effect on occupants. Inter-floor noise is identified as a severe type of indoor noise in residential areas. The identification of noise is considered a fundamental step that is essential for studying the challenges of noise pollution. By harnessing a sound level meter, long-term measurement, and site surveying, previous studies have been conducted on the identification of noise in residential areas to estimate the level, type, and position of generated noise. However, it is challenging to identify the source type and position of noise travelling through multi-story residential buildings owing to the difficulty of the human ear in intercepting these sounds. Recent studies on the identification of indoor noise are limited to noise sources and receivers on a single level of the floor, and they require multiple sensor channels to determine the time difference of arrival. Residential buildings, which are usually reinforced concrete structures, are considered to be concrete, steel, and fluid-mixed media with high structural complexity and occupants that have insufficient knowledge of the details of their properties. In this study, we propose a data-driven identification of noise in reinforced concrete buildings via the learning-based localization method using a single sensor. Actual experiments were conducted in a campus building, as well as two apartment buildings. Performance was analyzed according to several source types and positions that apply the deep convolutional neural network (CNN)-based supervised learning. The validations against the datasets obtained in three buildings verified the generalizability of the proposed method. In addition, noise identification data transferred within different floor sections in a single building and between similar buildings were presented in this study. Although indoor noise identification is emphasized in this work, the proposed method can be beneficial for other noise identification methods that employ a single sensor.공동주택의 증가로 건물 내 이웃 간의 소음 문제가 사회적으로 대두되고 있다. 거주자에게 노출된 소음은 거주자의 건강 문제에 직결될 수도 있으므로 건물 내 소음에 관한 여러 연구가 진행되어 왔다. 다층 건물 내에서 발생한 소음은 건물의 구조를 따라 다른 층으로 전달되며 이러한 층간소음은 주변 이웃에게 고통으로 다가올 수 있다. 소음원의 규명은 소음을 다룰 때 선행되어야 하는 바 건물 내 소음의 준위, 종류, 위치 파악에 관련된 연구들이 진행되어 왔다. 소음의 준위는 소음측정기를 사용하여 측정이 가능하나 건물의 구조를 따라 전달된 소음의 종류와 위치를 판별하는 것은 추정이 필요한 문제이며 사람의 청력에 의존하여서 풀기도 어렵다. 최근 연구된 관련 연구를 살펴보면 건물 내 소음의 종류를 분류하는 연구는 거의 다뤄지지 않았고, 소음원 위치 추정 연구의 경우 동일 층에 소음원과 여러 채널의 수신기가 위치한 경우를 다중측량 (multilateration) 을 통하여 제한적으로 다뤘다. 일반적으로 현대 거주용 건축물의 대부분은 철근 콘크리트 구조이며 층간의 소음 전달 환경은 콘크리트, 철근, 유체가 혼재하는 복잡한 환경이다. 일반인 거주자가 이러한 환경에서의 소음 전달 환경을 파악하고 소음의 전달 모델을 세워 소음을 규명하는 것은 어렵다. 본 논문은 모바일 장치 (mobile device) 의 단일 음향 센서로 측정한 소음과 합성곱 신경망을 활용하여 데이터 기반 (data-driven) 의 건물 내 소음 규명 방법을 제안하고 한 개의 캠퍼스 건물과 두 아파트 건물에서 진행한 실험을 통하여 이 기법의 유용성과 보편성을 보였다. 또한 한 층간에서 학습한 소음 규명 지식을 동일 건물의 다른 층간에서의 소음 규명에, 한 건물에서 학습한 소음 규명 지식을 다른 건물 내 소음 규명에 활용 할 수 있음을 보였다. 제안하는 기법은 소음 전달 환경 파악 및 모델을 얻기 어려운 분야에서의 적용에도 유용할 것으로 기대한다.Abstract I Contents iii List of Figures vi List of Tables ix 1 Introduction 2 1.1 Backgrounds 2 1.2 Approach 5 1.2.1 Data-driven noise identification 5 1.2.2 Source type classification and localization 7 1.2.3 Knowledge transfer 10 1.3 Contributions 15 1.4 Outline of the Dissertation 16 2 Source type classification and localization of acoustic noises in a reinforced concrete structure 28 2.1 Introduction 29 2.1.1 Motivation 29 2.1.2 Related literature 29 2.1.3 Approach 30 2.1.4 Contributions of this chapter 31 2.2 Campus building inter-floor noise dataset 32 2.2.1 Selecting source type and source position 32 2.2.2 Generating and collecting inter-floor noise 33 2.3 Supervised learning of inter-floor noises 36 2.3.1 Convolutional neural networks for acoustic scene classification 36 2.3.2 Network architecture 36 2.3.3 Evaluation 40 2.3.4 Source type classification results 41 2.3.5 Localizationresults....................... 41 2.4 Source type classification and localization of inter-floor noises generated on unlearned positions 47 2.4.1 Source type classification of inter-floor noises from unlearned positions 48 2.4.2 Localization of inter-floor noises from unlearned positions 50 2.5 Summary 52 2.6 Acknowledgments 53 3 Knowledge transfer between reinforced concrete structures 61 3.1 Introduction 62 3.1.1 Motivation 62 3.1.2 Related Literature 62 3.1.3 Approach 63 3.1.4 Contributions of this chapter 63 3.2 Apartment building inter-floor noise dataset 64 3.3 Inter-floor noise classification 70 3.3.1 Onset detection 70 3.3.2 Convolutional neural network-based classifier 71 3.3.3 Network training 75 3.3.4 Source type classification and localization tasks 75 3.4 Performance Evaluation 79 3.4.1 Source type classification results in a single apartment building 79 3.4.2 Localization results in a single apartment building 80 3.4.3 Results of knowledge transfer between the apartment buildings 81 3.5 Summary 87 3.6 Acknowledgments 94 4 Conclusions 96 4.1 Findings and limitations 97 4.2 Applications 97 4.2.1 Marine structures 98 4.2.2 Mobile application 98 4.3 Future study 100 4.3.1 Learning with building structure representation 100 4.3.2 Learning with data measured at multiple receiver locations 100 4.3.3 Task oriented algorithm 101 A Precision, recall, and F1 score of the classification results 102 B Data analysis 105 C Using a one-dimensional convolutional neural network and feature visualization 112 Abstract (In Korean) 124Docto