Automated license plate recognition: a survey on methods and techniques

With the explosive growth in the number of vehicles in use, automated license plate recognition (ALPR) systems are required for a wide range of tasks such as law enforcement, surveillance, and toll booth operations. The operational specifications of these systems are diverse due to the differences in the intended application. For instance, they may need to run on handheld devices or cloud servers, or operate in low light and adverse weather conditions. In order to meet these requirements, a variety of techniques have been developed for license plate recognition. Even though there has been a notable improvement in the current ALPR methods, there is a requirement to be filled in ALPR techniques for a complex environment. Thus, many approaches are sensitive to the changes in illumination and operate mostly in daylight. This study explores the methods and techniques used in ALPR in recent literature. We present a critical and constructive analysis of related studies in the field of ALPR and identify the open challenge faced by researchers and developers. Further, we provide future research directions and recommendations to optimize the current solutions to work under extreme conditions

학습된 모델 갱신 기법을 사용한 강인한 물체 추적

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·컴퓨터공학부, 2021. 2. 이경무.본 학위 논문에서는 물체 추적 알고리즘들에서 사용되는 모델 갱신 기법의 문제를 해결하기 위한 방법론을 제안한다. 기존 물체 추적 알고리즘들은 물체 추적 문제를 객체 검출을 통한 추적 문제 (tracking-by-detection) 로 간주하여 왔으며, 이들은 특정 물체를 검출할 수 있는 검출기 모델을 주어진 비디오의 첫번째 프레임에서 학습하여 이 모델을 사용하여 비디오의 차후 프레임들에서 목표 물체를 검출하는 방식으로 물체 추적 문제를 해결하여 왔다. 하지만 이러한 모델은 물체의 변형, 크기 변화, 가려짐, 조명 변화, 배경 물체의 등장 등의 다양한 상황변화와 물체의 외양변화에 따라 추적에 어려움이 존재한다. 이러한 문제들을 해결하기 위해 기존 물체 추적 알고리즘들은 추적 도중에 물체의 변화한 외양과 배경 물체들에 대한 새로운 정보를 추적 과정에 반영하기 위해 모델 갱신 기법을 활용하여 왔다. 하지만 이러한 기법들에서 모델 갱신 과정은 소수의 학습 표본을 사용한 최적화 문제의 해결을 통해 주로 이루어지며, 경험적으로 얻어진 정규화 기법을 활용하기 때문에 모델의 과적합 문제와 오류 누적 문제가 물체 추적 과정에서 지속되는 문제점이 있다. 전술한 문제점들을 해결하기 위해 본 논문에서는 물체 추적 문제에서 사용되는 모델 갱신 기법에 대한 새로운 접근법들을 제시한다. 이에 대해 세 가지의 모델 갱신 기법들을 제안하며 각각은: (1) 강화학습 기법에 기반한 표본 선택기법, (2) 메타학습을 기반으로 한 피쳐 공간의 갱신기법, (3) 적응적 컨티뉴얼 메타학습 기반의 갱신기법이다. 제안한 방법론들은 심층 신경망 구조에 기반한 메타러너를 도입하여 다양한 장면변화와 상황변화에 대해 학습 과정에서의 과적합 문제와 오류 누적 문제를 줄이고자 하였으며, 메타러너는 경량화된 구조로 설계되어 전체 물체 추적 프레임워크가 실시간 속도로 동작할 수 있게 하였다. 첫번째로, 정책 네트워크를 메타러너로 활용하는 강화학습 기반의 표본 선택기법을 제안한다. 정책 네트워크는 주어진 장면에서 목표 물체를 검출하기 위해 여러 표본 중 사용하기에 가장 적합한 표본을 선택하는 의사결정을 학습한다. 다음으로, 메타러너 네트워크를 활용한 메타학습 기법을 제안하며, 여기서 메타러너는 손실함수의 그래디언트 정보를 활용해 목표 물체에 특화된 피쳐 공간을 구축한다. 메타러너 네트워크는 물체 추적기에 대해 적응적인 가중치와 채널 어텐션의 형태로 새로운 정보를 제공한다. 마지막으로, 컨티뉴얼 메타학습 기반의 기법에서는 초기 갱신과정과 온라인 갱신과정 두 가지 모두를 적응형 컨티뉴얼 메타학습 프레임워크로 모델한다. 메타러너는 물체 추적기가 새로운 학습 표본을 배울지, 아니면 기존 지식을 유지할지를 선택할 수 있도록 적응적으로 학습과정을 제어하는 역할을 학습한다. 제안하는 기법들을 물체 추적 알고리즘들에 적용해본 결과 유의미한 성능 향상을 얻을 수 있었으며, 면밀한 실험적 분석과 구성요소별 분석을 통해 유효성을 검증하였다. 또한 저명하면서 널리 사용되는 물체 추적 벤치마크들을 활용한 비교실험을 통해 다른 최신 물체추적 알고리즘들과 비교해서도 실시간 속도로 효율적으로 동작하면서 우수한 성능을 보여줌을 확인하였다.In this dissertation, we address the model adaptation problem of visual tracking algorithms. Conventional tracking algorithms regard the visual tracking problem as a tracking-by-detection problem, which can be solved by formulating a target-specific detection model at the initial frame of a given video, and evaluating the model for the subsequent video frames. However, various challenges are associated with the model due to changes in circumstances such as target deformation, scale change, occlusion, illumination change, background clutter, etc. To deal with the aforementioned challenges, conventional tracking algorithms incorporate a model adaptation strategy to provide the model with new information regarding the target appearance and background distractor objects. Nonetheless, since these approaches are often conducted on a handful of self-labeled training examples through solving an optimization task involving hand-crafted regularization schemes, the risk of overfitting and error accumulation persist throughout the course of the tracking process. In order to address the aforementioned problems, we introduce novel approaches to the model adaptation strategy for the visual tracking problem. Three types of model adaptation approaches are proposed, based on the following: (1) reinforcement learning based exemplar selection, (2) deep meta-learning based feature space update, (3) deep adaptive continual meta-learning based adaptation. The proposed approaches introduce deep neural network based meta-learners that can handle various scenes and circumstances with reduced overfitting and error accumulation, while the meta-learners are designed to be light-weight and can achieve real-time speeds for the overall visual tracking framework. First, we propose a deep reinforcement learning based exemplar selection method that incorporates a policy network for its meta-learner. The policy network is trained to make decisions on selecting the adequate target exemplar that can be used to locate the target given a scene. Next, a deep meta-learning based method, which utilizes a meta-learner network to construct the target-specific feature space using the loss gradient information, is proposed. The meta-learner network provides the tracker with new information in the form of adaptive weights and channel attention. Finally, a deep continual meta-learning based method simultaneously models the initial and online adaptations under the adaptive continual meta-learning framework. The meta-learner is trained to adaptively regulate the learning process where the tracker can choose between learning new examples and retaining the previous knowledge. Applying the proposed methods to visual tracking algorithms, significant performance gains are achieved and the effectiveness is validated by the extensive experimental evaluations and component-wise ablation analyses. Additionally, comparisons on well-known, widely used visual tracking benchmarks demonstrate the competitive performance against other state-of-the-art tracking algorithms, while efficiently running at real-time speeds.1 Introduction 1 2 Model Selection by Reinforcement Learning 7 2.1 Introduction 8 2.2 Related Work 10 2.3 Tracking with Reinforced Decisions 13 2.3.1 Proposed Tracking Algorithm 13 2.3.2 Reinforcement Learning Overview and Application to Visual Tracking 16 2.3.3 Network Architectures 19 2.3.4 Training the Policy Network 20 2.4 Experiments 21 2.4.1 Implementation Details 21 2.4.2 Evaluation on OTB dataset 23 2.5 Summary 31 3 Model Update by Meta-Learning 33 3.1 Introduction 34 3.2 Related Work 36 3.3 Tracking with Meta-Learner 39 3.3.1 Overview of Proposed Method 39 3.3.2 Network Implementation and Training 44 3.4 Experimental Results 47 3.4.1 Evaluation Environment 47 3.4.2 Experiments and Analysis 47 3.5 Summary 57 4 Model Update by Continual Meta-Learning 59 4.1 Introduction 61 4.2 Related Work 63 4.3 Tracking with Adaptive Continual Meta-Learner 67 4.3.1 Meta-Training with Simulated Episodes 67 4.3.2 Proposed Tracking Algorithm 73 4.3.3 Baseline Tracking Algorithm: TACT 75 4.4 Experiments 81 4.4.1 Implementation Details 81 4.4.2 Quantitative Evaluation 83 4.4.3 Analysis 88 4.5 Summary 93 5 Conclusion 95 5.1 Summary and Contributions of the Dissertation 95 5.2 Future Work 97 Bibliography 99 국문초록 114Docto

License Plate Recognition using Convolutional Neural Networks Trained on Synthetic Images

In this thesis, we propose a license plate recognition system and study the feasibility of using synthetic training samples to train convolutional neural networks for a practical application. First we develop a modular framework for synthetic license plate generation; to generate different license plate types (or other objects) only the first module needs to be adapted. The other modules apply variations to the training samples such as background, occlusions, camera perspective projection, object noise and camera acquisition noise, with the aim to achieve enough variation of the object that the trained networks will also recognize real objects of the same class. Then we design two convolutional neural networks of low-complexity for license plate detection and character recognition. Both are designed for simultaneous classification and localization by branching the networks into a classification and a regression branch and are trained end-to-end simultaneously over both branches, on only our synthetic training samples. To recognize real license plates, we design a pipeline for scale invariant license plate detection with a scale pyramid and a fully convolutional application of the license plate detection network in order to detect any number of license plates and of any scale in an image. Before character classification is applied, potential plate regions are un-skewed based on the detected plate location in order to achieve an as optimal representation of the characters as possible. The character classification is also performed with a fully convolutional sweep to simultaneously find all characters at once. Both the plate and the character stages apply a refinement classification where initial classifications are first centered and rescaled. We show that this simple, yet effective trick greatly improves the accuracy of our classifications, and at a small increase of complexity. To our knowledge, this trick has not been exploited before. To show the effectiveness of our system we first apply it on a dataset of photos of Italian license plates to evaluate the different stages of our system and which effect the classification thresholds have on the accuracy. We also find robust training parameters and thresholds that are reliable for classification without any need for calibration on a validation set of real annotated samples (which may not always be available) and achieve a balanced precision and recall on the set of Italian license plates, both in excess of 98%. Finally, to show that our system generalizes to new plate types, we compare our system to two reference system on a dataset of Taiwanese license plates. For this, we only modify the first module of the synthetic plate generation algorithm to produce Taiwanese license plates and adjust parameters regarding plate dimensions, then we train our networks and apply the classification pipeline, using the robust parameters, on the Taiwanese reference dataset. We achieve state-of-the-art performance on plate detection (99.86% precision and 99.1% recall), single character detection (99.6%) and full license reading (98.7%)

Multimodal sentiment analysis in real-life videos

This thesis extends the emerging field of multimodal sentiment analysis of real-life videos, taking two components into consideration: the emotion and the emotion's target. The emotion component of media is traditionally represented as a segment-based intensity model of emotion classes. This representation is replaced here by a value- and time-continuous view. Adjacent research fields, such as affective computing, have largely neglected the linguistic information available from automatic transcripts of audio-video material. As is demonstrated here, this text modality is well-suited for time- and value-continuous prediction. Moreover, source-specific problems, such as trustworthiness, have been largely unexplored so far. This work examines perceived trustworthiness of the source, and its quantification, in user-generated video data and presents a possible modelling path. Furthermore, the transfer between the continuous and discrete emotion representations is explored in order to summarise the emotional context at a segment level. The other component deals with the target of the emotion, for example, the topic the speaker is addressing. Emotion targets in a video dataset can, as is shown here, be coherently extracted based on automatic transcripts without limiting a priori parameters, such as the expected number of targets. Furthermore, alternatives to purely linguistic investigation in predicting targets, such as knowledge-bases and multimodal systems, are investigated. A new dataset is designed for this investigation, and, in conjunction with proposed novel deep neural networks, extensive experiments are conducted to explore the components described above. The developed systems show robust prediction results and demonstrate strengths of the respective modalities, feature sets, and modelling techniques. Finally, foundations are laid for cross-modal information prediction systems with applications to the correction of corrupted in-the-wild signals from real-life videos

3D-3D Deformable Registration and Deep Learning Segmentation based Neck Diseases Analysis in MRI

Whiplash, cervical dystonia (CD), neck pain and work-related upper limb disorder (WRULD) are the most common diseases in the cervical region. Headaches, stiffness, sensory disturbance to the legs and arms, optical problems, aching in the back and shoulder, and auditory and visual problems are common symptoms seen in patients with these diseases. CD patients may also suffer tormenting spasticity in some neck muscles, with the symptoms possibly being acute and persisting for a long time, sometimes a lifetime. Whiplash-associated disorders (WADs) may occur due to sudden forward and backward movements of the head and neck occurring during a sporting activity or vehicle or domestic accident. These diseases affect private industries, insurance companies and governments, with the socio-economic costs significantly related to work absences, long-term sick leave, early disability and disability support pensions, health care expenses, reduced productivity and insurance claims. Therefore, diagnosing and treating neck-related diseases are important issues in clinical practice. The reason for these afflictions resulting from accident is the impairment of the cervical muscles which undergo atrophy or pseudo-hypertrophy due to fat infiltrating into them. These morphological changes have to be determined by identifying and quantifying their bio-markers before applying any medical intervention. Volumetric studies of neck muscles are reliable indicators of the proper treatments to apply. Radiation therapy, chemotherapy, injection of a toxin or surgery could be possible ways of treating these diseases. However, the dosages required should be precise because the neck region contains some sensitive organs, such as nerves, blood vessels and the trachea and spinal cord. Image registration and deep learning-based segmentation can help to determine appropriate treatments by analyzing the neck muscles. However, this is a challenging task for medical images due to complexities such as many muscles crossing multiple joints and attaching to many bones. Also, their shapes and sizes vary greatly across populations whereas their cross-sectional areas (CSAs) do not change in proportion to the heights and weights of individuals, with their sizes varying more significantly between males and females than ages. Therefore, the neck's anatomical variabilities are much greater than those of other parts of the human body. Some other challenges which make analyzing neck muscles very difficult are their compactness, similar gray-level appearances, intra-muscular fat, sliding due to cardiac and respiratory motions, false boundaries created by intramuscular fat, low resolution and contrast in medical images, noise, inhomogeneity and background clutter with the same composition and intensity. Furthermore, a patient's mode, position and neck movements during the capture of an image create variability. However, very little significant research work has been conducted on analyzing neck muscles. Although previous image registration efforts form a strong basis for many medical applications, none can satisfy the requirements of all of them because of the challenges associated with their implementation and low accuracy which could be due to anatomical complexities and variabilities or the artefacts of imaging devices. In existing methods, multi-resolution- and heuristic-based methods are popular. However, the above issues cause conventional multi-resolution-based registration methods to be trapped in local minima due to their low degrees of freedom in their geometrical transforms. Although heuristic-based methods are good at handling large mismatches, they require pre-segmentation and are computationally expensive. Also, current deformable methods often face statistical instability problems and many local optima when dealing with small mismatches. On the other hand, deep learning-based methods have achieved significant success over the last few years. Although a deeper network can learn more complex features and yields better performances, its depth cannot be increased as this would cause the gradient to vanish during training and result in training difficulties. Recently, researchers have focused on attention mechanisms for deep learning but current attention models face a challenge in the case of an application with compact and similar small multiple classes, large variability, low contrast and noise. The focus of this dissertation is on the design of 3D-3D image registration approaches as well as deep learning-based semantic segmentation methods for analyzing neck muscles. In the first part of this thesis, a novel object-constrained hierarchical registration framework for aligning inter-subject neck muscles is proposed. Firstly, to handle large-scale local minima, it uses a coarse registration technique which optimizes a new edge position difference (EPD) similarity measure to align large mismatches. Also, a new transformation based on the discrete periodic spline wavelet (DPSW), affine and free-form-deformation (FFD) transformations are exploited. Secondly, to avoid the monotonous nature of using transformations in multiple stages, affine registration technique, which uses a double-pushing system by changing the edges in the EPD and switching the transformation's resolutions, is designed to align small mismatches. The EPD helps in both the coarse and fine techniques to implement object-constrained registration via controlling edges which is not possible using traditional similarity measures. Experiments are performed on clinical 3D magnetic resonance imaging (MRI) scans of the neck, with the results showing that the EPD is more effective than the mutual information (MI) and the sum of squared difference (SSD) measures in terms of the volumetric dice similarity coefficient (DSC). Also, the proposed method is compared with two state-of-the-art approaches with ablation studies of inter-subject deformable registration and achieves better accuracy, robustness and consistency. However, as this method is computationally complex and has a problem handling large-scale anatomical variabilities, another 3D-3D registration framework with two novel contributions is proposed in the second part of this thesis. Firstly, a two-stage heuristic search optimization technique for handling large mismatches,which uses a minimal user hypothesis regarding these mismatches and is computationally fast, is introduced. It brings a moving image hierarchically closer to a fixed one using MI and EPD similarity measures in the coarse and fine stages, respectively, while the images do not require pre-segmentation as is necessary in traditional heuristic optimization-based techniques. Secondly, a region of interest (ROI) EPD-based registration framework for handling small mismatches using salient anatomical information (AI), in which a convex objective function is formed through a unique shape created from the desired objects in the ROI, is proposed. It is compared with two state-of-the-art methods on a neck dataset, with the results showing that it is superior in terms of accuracy and is computationally fast. In the last part of this thesis, an evaluation study of recent U-Net-based convolutional neural networks (CNNs) is performed on a neck dataset. It comprises 6 recent models, the U-Net, U-Net with a conditional random field (CRF-Unet), attention U-Net (A-Unet), nested U-Net or U-Net++, multi-feature pyramid (MFP)-Unet and recurrent residual U-Net (R2Unet) and 4 with more comprehensive modifications, the multi-scale U-Net (MS-Unet), parallel multi-scale U-Net (PMSUnet), recurrent residual attention U-Net (R2A-Unet) and R2A-Unet++ in neck muscles segmentation, with analyses of the numerical results indicating that the R2Unet architecture achieves the best accuracy. Also, two deep learning-based semantic segmentation approaches are proposed. In the first, a new two-stage U-Net++ (TS-UNet++) uses two different types of deep CNNs (DCNNs) rather than one similar to the traditional multi-stage method, with the U-Net++ in the first stage and the U-Net in the second. More convolutional blocks are added after the input and before the output layers of the multi-stage approach to better extract the low- and high-level features. A new concatenation-based fusion structure, which is incorporated in the architecture to allow deep supervision, helps to increase the depth of the network without accelerating the gradient-vanishing problem. Then, more convolutional layers are added after each concatenation of the fusion structure to extract more representative features. The proposed network is compared with the U-Net, U-Net++ and two-stage U-Net (TS-UNet) on the neck dataset, with the results indicating that it outperforms the others. In the second approach, an explicit attention method, in which the attention is performed through a ROI evolved from ground truth via dilation, is proposed. It does not require any additional CNN, as does a cascaded approach, to localize the ROI. Attention in a CNN is sensitive with respect to the area of the ROI. This dilated ROI is more capable of capturing relevant regions and suppressing irrelevant ones than a bounding box and region-level coarse annotation, and is used during training of any CNN. Coarse annotation, which does not require any detailed pixel wise delineation that can be performed by any novice person, is used during testing. This proposed ROI-based attention method, which can handle compact and similar small multiple classes with objects with large variabilities, is compared with the automatic A-Unet and U-Net, and performs best

Artificial Intelligence Technology

This open access book aims to give our readers a basic outline of today’s research and technology developments on artificial intelligence (AI), help them to have a general understanding of this trend, and familiarize them with the current research hotspots, as well as part of the fundamental and common theories and methodologies that are widely accepted in AI research and application. This book is written in comprehensible and plain language, featuring clearly explained theories and concepts and extensive analysis and examples. Some of the traditional findings are skipped in narration on the premise of a relatively comprehensive introduction to the evolution of artificial intelligence technology. The book provides a detailed elaboration of the basic concepts of AI, machine learning, as well as other relevant topics, including deep learning, deep learning framework, Huawei MindSpore AI development framework, Huawei Atlas computing platform, Huawei AI open platform for smart terminals, and Huawei CLOUD Enterprise Intelligence application platform. As the world’s leading provider of ICT (information and communication technology) infrastructure and smart terminals, Huawei’s products range from digital data communication, cyber security, wireless technology, data storage, cloud computing, and smart computing to artificial intelligence