포토리소그래피 검사 시스템의 이미지 분할을 위한 새로운 깊은 아키텍처

학위논문(박사) -- 서울대학교대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2021.8. 홍성수.In semiconductor manufacturing, defect detection is critical to maintain high yield. Typically, the defects of semiconductor wafer may be generated from the manufacturing process. Most computer vision systems used in semiconductor photolithography process inspection still have adopt to image processing algorithm, which often occur inspection faults due to sensitivity to external environment changes. Therefore, we intend to tackle this problem by means of converging the advantages of image processing algorithm and deep learning. In this dissertation, we propose Image Segmentation Detector (ISD) to extract the enhanced feature-maps under the situations where training dataset is limited in the specific industry domain, such as semiconductor photolithography inspection. ISD is used as a novel backbone network of state-of-the-art Mask R-CNN framework for image segmentation. ISD consists of four dense blocks and four transition layers. Especially, each dense block in ISD has the shortcut connection and the concatenation of the feature-maps produced in layer with dynamic growth rate for more compactness. ISD is trained from scratch without using recently approached transfer learning method. Additionally, ISD is trained with image dataset pre-processed by means of our designed image filter to extract the better enhanced feature map of Convolutional Neural Network (CNN). In ISD, one of the key design principles is the compactness, plays a critical role for addressing real-time problem and for application on resource bounded devices. To empirically demonstrate the model, this dissertation uses the existing image obtained from the computer vision system embedded in the currently operating semiconductor manufacturing equipment. ISD achieves consistently better results than state-of-the-art methods at the standard mean average precision which is the most common metric used to measure the accuracy of the instance detection. Significantly, our ISD outperforms baseline method DenseNet, while requiring only 1/4 parameters. We also observe that ISD can achieve comparable better results in performance than ResNet, with only much smaller 1/268 parameters, using no extra data or pre-trained models. Our experimental results show that ISD can be useful to many future image segmentation research efforts in diverse fields of semiconductor industry which is requiring real-time and good performance with only limited training dataset.반도체 제조에서 결함 검출은 높은 수율을 유지하는데 중요합니다. 전형적으로, 반도체 웨이퍼의 결함은 제조 공정에서 발생하고 있습니다. 반도체 포토리소그래피 공정 검사에 사용되는 대부분의 컴퓨터 비전 시스템들은 여전히 외부 환경 변화에 민감한 이미지 처리 알고리즘을 사용하고 있어서 검사 오류가 자주 발생하고 있습니다. 따라서, 이미지 처리 알고리즘의 장점과 딥 러닝의 장점을 융합하여 이 문제를 해결하려고 합니다. 이 논문에서 우리는 반도체 포토리소그래피 검사와 같이 훈련 데이터 세트가 제한된 상황에서 향상된 기능 맵을 추출하기 위해 이미지 분할 검출기(Image Segmentation Detector, 이하 ISD)를 제안합니다. ISD는 이미지 분할을 위한 최신 Mask R-CNN 프레임 워크의 새로운 백본 네트워크로 사용합니다. ISD는 4 개의 조밀한 블록과 4 개의 전환 레이어로 구성합니다. 특히, ISD의 각 조밀한 블록은 보다 컴팩트함을 위해 단축 연결 및 동적 성장률을 가지고 레이어에서 생성된 피쳐 맵을 결합하고 있습니다. ISD는 최근 적용하고 있는 전이 학습 방법을 사용하지 않고 처음부터 훈련합니다. 또한, ISD는 합성곱 신경망(Convolutional Neural Network, 이하 CNN)의 향상된 기능 맵을 추출하기 위해 우리가 설계한 이미지 필터를 통해 사전 처리된 이미지 데이터 세트로 훈련을 합니다. ISD의 설계 핵심 원칙 중 하나는 소형화로 실시간 문제를 해결하고 리소스에 제한이 있는 장치에 적용하는데 중요한 역할을 하게 합니다. 모델을 실증적으로 입증하기 위해 이 논문에서는 현재 운영 중인 반도체 제조 장비에 내장된 컴퓨터 비전 시스템에서 획득한 실제 이미지를 사용합니다. ISD는 가장 일반적인 성능 측정 지표인 평균 정밀도에서 최첨단 백본 네트워크 보다 일관되게 더 나은 성능을 얻습니다. 특히, ISD는 베이스 라인으로 삼은 DenseNet 보다 파라미터들이 4배 더 적지만, 성능이 우수 합니다. 우리는 또한 ISD가 Mask R-CNN 백본 네트워크로 주로 사용하는 ResNet 보다 268배 훨씬 더 적은 파라미터들을 가지고, 추가 데이터 또는 사전 훈련된 모델을 사용하지 않고, 성능에서 비슷하거나 더 나은 결과를 얻을 수 있음을 관찰합니다. 우리의 실험 결과들은 ISD가 제한된 훈련 데이터 세트만으로 실시간 및 우수한 성능을 요구하는 반도체 산업의 다양한 분야들에서 많은 미래의 이미지 분할 연구 노력에 유용할 수 있음을 보여줍니다.Chapter 1. Introduction １ 1.1. Background and Motivation ４ Chapter 2. Related Work １２ 2.1. Inspection Method １２ 2.2. Instance Segmentation １６ 2.3. Backbone Structure ２４ 2.4. Enhanced Feature Map ３５ 2.5. Detection Performance Evaluation ４７ 2.6. Learning Network Model from Scratch ５０ Chapter 3. Proposed Method ５２ 3.1. ISD Architecture ５２ 3.2. Pre-processing ６３ 3.3. Model Training ７１ 3.4. Training Objective ７３ 3.5. Setting and Configurations ７５ Chapter 4. Experimental Evaluation ７８ 4.1. Classification Results on ISD ８１ 4.2. Comparison with Pre-processing ８５ 4.3. Image Segmentation Results on ISD ９４ 4.3.1. Results on Suck-back State ９４ 4.3.2. Results on Dispensing State １０４ 4.4. Comparison with State-of-the-art Methods １１３ Chapter 5. Conclusion １２１ Bibliography １２７ 초록 １４６박

Semiconductor Defect Pattern Classification by Self-Proliferation-and-Attention Neural Network

Semiconductor manufacturing is on the cusp of a revolution: the Internet of Things (IoT). With IoT we can connect all the equipment and feed information back to the factory so that quality issues can be detected. In this situation, more and more edge devices are used in wafer inspection equipment. This edge device must have the ability to quickly detect defects. Therefore, how to develop a high-efficiency architecture for automatic defect classification to be suitable for edge devices is the primary task. In this paper, we present a novel architecture that can perform defect classification in a more efficient way. The first function is self-proliferation, using a series of linear transformations to generate more feature maps at a cheaper cost. The second function is self-attention, capturing the long-range dependencies of feature map by the channel-wise and spatial-wise attention mechanism. We named this method as self-proliferation-and-attention neural network. This method has been successfully applied to various defect pattern classification tasks. Compared with other latest methods, SP&A-Net has higher accuracy and lower computation cost in many defect inspection tasks

A deep learning-based approach for defect classification with context information in semiconductor manufacturing

This thesis presents some methodological and experimental contributions to a deep learning-based approach for the automatic classifi cation of microscopic defects in silicon wafers with context information. Canonical image classifi cation approaches have the limitation of utilizing only the information contained in the images. This work overcomes this limitation by using some context information about the defects to improve the current automatic classifi cation system

JujubeNet: A high-precision lightweight jujube surface defect classification network with an attention mechanism

Surface Defect Detection (SDD) is a significant research content in Industry 4.0 field. In the real complex industrial environment, SDD is often faced with many challenges, such as small difference between defect imaging and background, low contrast, large variation of defect scale and diverse types, and large amount of noise in defect images. Jujubes are naturally growing plants, and the appearance of the same type of surface defect can vary greatly, so it is more difficult than industrial products produced according to the prescribed process. In this paper, a ConvNeXt-based high-precision lightweight classification network JujubeNet is presented to address the practical needs of Jujube Surface Defect (JSD) classification. In the proposed method, a Multi-branching module using Depthwise separable Convolution (MDC) is designed to extract more feature information through multi-branching and substantially reduces the number of parameters in the model by using depthwise separable convolutions. What’s more, in our proposed method, the Convolutional Block Attention Module (CBAM) is introduced to make the model concentrate on different classes of JSD features. The proposed JujubeNet is compared with other mainstream networks in the actual production environment. The experimental results show that the proposed JujubeNet can achieve 99.1% classification accuracy, which is significantly better than the current mainstream classification models. The FLOPS and parameters are only 30.7% and 30.6% of ConvNeXt-Tiny respectively, indicating that the model can quickly and effectively classify JSD and is of great practical value

The use of image processing to determine cell defects in polycrystalline solar modules

This research aims to use image processingtodetermine cell defects in polycrystalline solar modules. Image processing is a process of enhancing images for differentapplications. One domain that seems to not yet utilise the use of image processing, is photovoltaics. An increased use of fossil fuels is damaging the earth and a call to protect the earth has resulted in the emergence of pollutant-free technologies such as polycrystalline photovoltaic (PV) cells, which are connected to make up solar modules. However, defects often affect the performance of PV cells and consequently solar modules. Electroluminescence (EL) images are used to examine polycrystalline solar (PV) modules to determine if the modules are defective. The main research question that this research addressed is“How can an image processing technique be used to effectively identify defective polycrystalline PV cells from EL images of such cells?“. The experimental research methodology was used to address the main research question. The initial investigation into the problem revealed that certain sectors within industry, as well as the Physics Department at Nelson Mandela University(NMU), do not currently utiliseimage processing when examining EL images of solar modules. The current process is a tedious, manual process whereby solar modules are manually inspected. An analysis of the current processes enabled the identification of ways in which to automatically examine EL images of solar modules. An analysis of literatureprovided a better understanding of the different techniques that are used to examine solar modules, and it was identified how image processing can be applied to EL images. Further analysis of literatureprovided a better understanding of image processing and how image classification experiments using Deep Learning (DL) as an image processing technique can be used to address the main research question. The outcome of the experiments conducted in this research weredifferentadaptive models(LeNet, MobileNet, Xception)that can classify EL images of PV cellsaccording to known standardsused by the Physics Department at NMU. The known standards yielded four classes; normal, uncritical, critical and very critical, which were used for the classification of EL images of PV cells. The adaptive models were evaluated to obtain the precision, recall and F1–scoreof the models.The precession, recall, and F1–score were required to determine how effective the models were in identifying defective PV cells from EL images.The results indicated that an image processing technique canbe used to identify defective polycrystalline PV cells from EL images of such cells. However, further research needs to be conducted to improve the effectiveness of the adaptive models

Gaze-Based Human-Robot Interaction by the Brunswick Model

We present a new paradigm for human-robot interaction based on social signal processing, and in particular on the Brunswick model. Originally, the Brunswick model copes with face-to-face dyadic interaction, assuming that the interactants are communicating through a continuous exchange of non verbal social signals, in addition to the spoken messages. Social signals have to be interpreted, thanks to a proper recognition phase that considers visual and audio information. The Brunswick model allows to quantitatively evaluate the quality of the interaction using statistical tools which measure how effective is the recognition phase. In this paper we cast this theory when one of the interactants is a robot; in this case, the recognition phase performed by the robot and the human have to be revised w.r.t. the original model. The model is applied to Berrick, a recent open-source low-cost robotic head platform, where the gazing is the social signal to be considered

Handbook of Digital Face Manipulation and Detection

This open access book provides the first comprehensive collection of studies dealing with the hot topic of digital face manipulation such as DeepFakes, Face Morphing, or Reenactment. It combines the research fields of biometrics and media forensics including contributions from academia and industry. Appealing to a broad readership, introductory chapters provide a comprehensive overview of the topic, which address readers wishing to gain a brief overview of the state-of-the-art. Subsequent chapters, which delve deeper into various research challenges, are oriented towards advanced readers. Moreover, the book provides a good starting point for young researchers as well as a reference guide pointing at further literature. Hence, the primary readership is academic institutions and industry currently involved in digital face manipulation and detection. The book could easily be used as a recommended text for courses in image processing, machine learning, media forensics, biometrics, and the general security area

Handbook of Digital Face Manipulation and Detection

This open access book provides the first comprehensive collection of studies dealing with the hot topic of digital face manipulation such as DeepFakes, Face Morphing, or Reenactment. It combines the research fields of biometrics and media forensics including contributions from academia and industry. Appealing to a broad readership, introductory chapters provide a comprehensive overview of the topic, which address readers wishing to gain a brief overview of the state-of-the-art. Subsequent chapters, which delve deeper into various research challenges, are oriented towards advanced readers. Moreover, the book provides a good starting point for young researchers as well as a reference guide pointing at further literature. Hence, the primary readership is academic institutions and industry currently involved in digital face manipulation and detection. The book could easily be used as a recommended text for courses in image processing, machine learning, media forensics, biometrics, and the general security area

Proceedings of the Conference on Production Systems and Logistics: CPSL 2022

[no abstract available