Automatic Robot Path Planning for Visual Inspection from Object Shape

Visual inspection is a crucial yet time-consuming task across various industries. Numerous established methods employ machine learning in inspection tasks, necessitating specific training data that includes predefined inspection poses and training images essential for the training of models. The acquisition of such data and their integration into an inspection framework is challenging due to the variety in objects and scenes involved and due to additional bottlenecks caused by the manual collection of training data by humans, thereby hindering the automation of visual inspection across diverse domains. This work proposes a solution for automatic path planning using a single depth camera mounted on a robot manipulator. Point clouds obtained from the depth images are processed and filtered to extract object profiles and transformed to inspection target paths for the robot end-effector. The approach relies on the geometry of the object and generates an inspection path that follows the shape normal to the surface. Depending on the object size and shape, inspection paths can be defined as single or multi-path plans. Results are demonstrated in both simulated and real-world environments, yielding promising inspection paths for objects with varying sizes and shapes. Code and video are open-source available at: https://github.com/CuriousLad1000/Auto-Path-PlannerComment: 8 page

A GEO-DATABASE FOR 3D-AIDED MULTI-EPOCH DOCUMENTATION OF BRIDGE INSPECTIONS

The recent collapse of bridges in Italy has prompted numerous studies on monitoring and maintenance. Many structures in Italy have been in service for over 50 years, necessitating new approaches to ensure their safety. To address this issue, Italy's Consiglio Superiore dei Lavori Pubblici (Superior Council of Public Works) has developed the Guidelines for Risk Classification and Management, proposing a multi-level approach to bridge management within a complex geomorphological environment. The guidelines outline a multi-level process that includes surveying the structures, conducting detailed inspections, and assigning risk classes based on hazard, exposure, and vulnerability. Current inspection processes are time-consuming and costly. Therefore, alternative monitoring technologies are crucial. Unmanned aerial vehicles equipped with cameras, laser technologies, and GPS systems offer flexible and cost-effective solutions for visual inspection. These technologies enable the collection of both quantitative and qualitative data, such as size, material properties, and overall condition. In this context, efficient data management and exploration systems are necessary to handle the vast amount of geo-referenced information. Multi-epoch databases play a crucial role in documenting the conditions of bridges and supporting a maintenance and structural health monitoring workflow. These databases can be utilized within a Bridge Management System to aid road managers in decision-making processes. Additionally, 3D exploration platforms provide visual analysis and highlight areas of interest within the structure. This work presents a multi-epoch geo-database that adheres to the Italian guidelines, offering optimized data management and queryability for 2D and 3D information. The entire process is designed using open-source and reproducible solutions

Excision And Recovery: Visual Defect Obfuscation Based Self-Supervised Anomaly Detection Strategy

Due to scarcity of anomaly situations in the early manufacturing stage, an unsupervised anomaly detection (UAD) approach is widely adopted which only uses normal samples for training. This approach is based on the assumption that the trained UAD model will accurately reconstruct normal patterns but struggles with unseen anomalous patterns. To enhance the UAD performance, reconstruction-by-inpainting based methods have recently been investigated, especially on the masking strategy of suspected defective regions. However, there are still issues to overcome: 1) time-consuming inference due to multiple masking, 2) output inconsistency by random masking strategy, and 3) inaccurate reconstruction of normal patterns when the masked area is large. Motivated by this, we propose a novel reconstruction-by-inpainting method, dubbed Excision And Recovery (EAR), that features single deterministic masking based on the ImageNet pre-trained DINO-ViT and visual obfuscation for hint-providing. Experimental results on the MVTec AD dataset show that deterministic masking by pre-trained attention effectively cuts out suspected defective regions and resolve the aforementioned issues 1 and 2. Also, hint-providing by mosaicing proves to enhance the UAD performance than emptying those regions by binary masking, thereby overcomes issue 3. Our approach achieves a high UAD performance without any change of the neural network structure. Thus, we suggest that EAR be adopted in various manufacturing industries as a practically deployable solution.Comment: 10 pages, 5 figures, 5 table

Visualised inspection system for monitoring environmental anomalies during daily operation and maintenance

PurposeVisual inspection and human judgement form the cornerstone of daily operations and maintenance (O&M) services activities carried out by facility managers nowadays. Recent advances in technologies such as building information modelling (BIM), distributed sensor networks, augmented reality (AR) technologies and digital twins present an immense opportunity to radically improve the way daily O&M is conducted. This paper aims to describe the development of an AR-supported automated environmental anomaly detection and fault isolation method to assist facility managers in addressing problems that affect building occupants’ thermal comfort.Design/methodology/approachThe developed system focusses on the detection of environmental anomalies related to the thermal comfort of occupants within a building. The performance of three anomaly detection algorithms in terms of their ability to detect indoor temperature anomalies is compared. Based on the fault tree analysis (FTA), a decision-making tree is developed to assist facility management (FM) professionals in identifying corresponding failed assets according to the detected anomalous symptoms. The AR system facilitates easy maintenance by highlighting the failed assets hidden behind walls/ceilings on site to the maintenance personnel. The system can thus provide enhanced support to facility managers in their daily O&M activities such as inspection, recording, communication and verification.FindingsTaking the indoor temperature inspection as an example, the case study demonstrates that the O&M management process can be improved using the proposed AR-enhanced inspection system. Comparative analysis of different anomaly detection algorithms reveals that the binary segmentation-based change point detection is effective and efficient in identifying temperature anomalies. The decision-making tree supported by FTA helps formalise the linkage between temperature issues and the corresponding failed assets. Finally, the AR-based model enhanced the maintenance process by visualising and highlighting the hidden failed assets to the maintenance personnel on site.Originality/valueThe originality lies in bringing together the advances in augmented reality, digital twins and data-driven decision-making to support the daily O&M management activities. In particular, the paper presents a novel binary segmentation-based change point detection for identifying temperature anomalous symptoms, a decision-making tree for matching the symptoms to the failed assets, and an AR system for visualising those assets with related information.EPSRC, Innovate U

RE-MOVE: An Adaptive Policy Design Approach for Dynamic Environments via Language-Based Feedback

Reinforcement learning-based policies for continuous control robotic navigation tasks often fail to adapt to changes in the environment during real-time deployment, which may result in catastrophic failures. To address this limitation, we propose a novel approach called RE-MOVE (\textbf{RE}quest help and \textbf{MOVE} on), which uses language-based feedback to adjust trained policies to real-time changes in the environment. In this work, we enable the trained policy to decide \emph{when to ask for feedback} and \emph{how to incorporate feedback into trained policies}. RE-MOVE incorporates epistemic uncertainty to determine the optimal time to request feedback from humans and uses language-based feedback for real-time adaptation. We perform extensive synthetic and real-world evaluations to demonstrate the benefits of our proposed approach in several test-time dynamic navigation scenarios. Our approach enable robots to learn from human feedback and adapt to previously unseen adversarial situations

Performance evaluation of an improved deep CNN-based concrete crack detection algorithm

This study uses a novel directional lighting approach to produce a computationally efficient five-channel Visual Geometry Group-16 (VGG-16) convolutional neural network (CNN) model for concrete crack detection and classification in low-light environments. The first convolutional layer of the proposed model copies the weights for the first three channels from the pre-trained model. In contrast, the additional two channels are set to the average of the existing weights along the channels. The model employs transfer learning and fine-tuning approaches to enhance accuracy and efficiency. It utilizes variations in patterned lighting to produce five channels. Each channel represents a grayscale version of the images captured using directed lighting in the right, below, left, above, and diffused directions, respectively. The model is evaluated on concrete crack samples with crack widths ranging from 0.07 mm to 0.3 mm. The modified five-channel VGG-16 model outperformed the traditional three-channel model, showing improvements ranging from 6.5 to 11.7 percent in true positive rate, false positive rate, precision, F1 score, accuracy, and Matthew’s correlation coefficient. These performance improvements are achieved with no significant change in evaluation time. This study provides useful information for constructing custom CNN models for civil engineering problems. Furthermore, it introduces a novel technique to identify cracks in concrete buildings using directed illumination in low-light conditions

Stereo vision-based autonomous navigation for oil and gas pressure vessel inspection using a low-cost UAV

It is vital to visually inspect pressure vessels regularly in the oil and gas company to maintain their integrity. Compared with visual inspection conducted by sending engineers and ground vehicles into the pressure vessel, utilising an autonomous Unmanned Aerial Vehicle (UAV) can overcome many limitations including high labour intensity, low efficiency and high risk to human health. This work focuses on enhancing some existing technologies to support low-cost UAV autonomous navigation for visual inspection of oil and gas pressure vessels. The UAV can gain the ability to follow the planned trajectory autonomously to record videos with a stereo camera in the pressure vessel, which is a GPS-denied and low-illumination environment. Particularly, the ORB-SLAM3 is improved by adopting the image contrast enhancement technique to locate the UAV in this challenging scenario. What is more, a vision hybrid Proportional-Proportional-Integral-Derivative (P-PID) position tracking controller is integrated to control the movement of the UAV. The ROS-Gazebo-PX4 simulator is customised deeply to validate the developed stereo vision-based autonomous navigation approach. It is verified that compared with the ORB-SLAM3, the numbers of ORB feature points and effective matching points obtained by the improved ORB-SLAM3 are increased by more than 400% and 600%, respectively. Thereby, the improved ORB-SLAM3 is effective and robust enough for UAV self-localisation, and the developed stereo vision-based autonomous navigation approach can be deployed for pressure vessel visual inspectio

제조 시스템에서의 예측 모델링을 위한 지능적 데이터 획득

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 산업공학과, 2021. 2. 조성준.Predictive modeling is a type of supervised learning to find the functional relationship between the input variables and the output variable. Predictive modeling is used in various aspects in manufacturing systems, such as automation of visual inspection, prediction of faulty products, and result estimation of expensive inspection. To build a high-performance predictive model, it is essential to secure high quality data. However, in manufacturing systems, it is practically impossible to acquire enough data of all kinds that are needed for the predictive modeling. There are three main difficulties in the data acquisition in manufacturing systems. First, labeled data always comes with a cost. In many problems, labeling must be done by experienced engineers, which is costly. Second, due to the inspection cost, not all inspections can be performed on all products. Because of time and monetary constraints in the manufacturing system, it is impossible to obtain all the desired inspection results. Third, changes in the manufacturing environment make data acquisition difficult. A change in the manufacturing environment causes a change in the distribution of generated data, making it impossible to obtain enough consistent data. Then, the model have to be trained with a small amount of data. In this dissertation, we overcome this difficulties in data acquisition through active learning, active feature-value acquisition, and domain adaptation. First, we propose an active learning framework to solve the high labeling cost of the wafer map pattern classification. This makes it possible to achieve higher performance with a lower labeling cost. Moreover, the cost efficiency is further improved by incorporating the cluster-level annotation into active learning. For the inspection cost for fault prediction problem, we propose a active inspection framework. By selecting products to undergo high-cost inspection with the novel uncertainty estimation method, high performance can be obtained with low inspection cost. To solve the recipe transition problem that frequently occurs in faulty wafer prediction in semiconductor manufacturing, a domain adaptation methods are used. Through sequential application of unsupervised domain adaptation and semi-supervised domain adaptation, performance degradation due to recipe transition is minimized. Through experiments on real-world data, it was demonstrated that the proposed methodologies can overcome the data acquisition problems in the manufacturing systems and improve the performance of the predictive models.예측 모델링은 지도 학습의 일종으로, 학습 데이터를 통해 입력 변수와 출력 변수 간의 함수적 관계를 찾는 과정이다. 이런 예측 모델링은 육안 검사 자동화, 불량 제품 사전 탐지, 고비용 검사 결과 추정 등 제조 시스템 전반에 걸쳐 활용된다. 높은 성능의 예측 모델을 달성하기 위해서는 양질의 데이터가 필수적이다. 하지만 제조 시스템에서 원하는 종류의 데이터를 원하는 만큼 획득하는 것은 현실적으로 거의 불가능하다. 데이터 획득의 어려움은 크게 세가지 원인에 의해 발생한다. 첫번째로, 라벨링이 된 데이터는 항상 비용을 수반한다는 점이다. 많은 문제에서, 라벨링은 숙련된 엔지니어에 의해 수행되어야 하고, 이는 큰 비용을 발생시킨다. 두번째로, 검사 비용 때문에 모든 검사가 모든 제품에 대해 수행될 수 없다. 제조 시스템에는 시간적, 금전적 제약이 존재하기 때문에, 원하는 모든 검사 결과값을 획득하는 것이 어렵다. 세번째로, 제조 환경의 변화가 데이터 획득을 어렵게 만든다. 제조 환경의 변화는 생성되는 데이터의 분포를 변형시켜, 일관성 있는 데이터를 충분히 획득하지 못하게 한다. 이로 인해 적은 양의 데이터만으로 모델을 재학습시켜야 하는 상황이 빈번하게 발생한다. 본 논문에서는 이런 데이터 획득의 어려움을 극복하기 위해 능동 학습, 능동 피쳐값 획득, 도메인 적응 방법을 활용한다. 먼저, 웨이퍼 맵 패턴 분류 문제의 높은 라벨링 비용을 해결하기 위해 능동학습 프레임워크를 제안한다. 이를 통해 적은 라벨링 비용으로 높은 성능의 분류 모델을 구축할 수 있다. 나아가, 군집 단위의 라벨링 방법을 능동학습에 접목하여 비용 효율성을 한차례 더 개선한다. 제품 불량 예측에 활용되는 검사 비용 문제를 해결하기 위해서는 능동 검사 방법을 제안한다. 제안하는 새로운 불확실성 추정 방법을 통해 고비용 검사 대상 제품을 선택함으로써 적은 검사 비용으로 높은 성능을 얻을 수 있다. 반도체 제조의 웨이퍼 불량 예측에서 빈번하게 발생하는 레시피 변경 문제를 해결하기 위해서는 도메인 적응 방법을 활용한다. 비교사 도메인 적응과 반교사 도메인 적응의 순차적인 적용을 통해 레시피 변경에 의한 성능 저하를 최소화한다. 본 논문에서는 실제 데이터에 대한 실험을 통해 제안된 방법론들이 제조시스템의 데이터 획득 문제를 극복하고 예측 모델의 성능을 높일 수 있음을 확인하였다.1. Introduction 1 2. Literature Review 9 2.1 Review of Related Methodologies 9 2.1.1 Active Learning 9 2.1.2 Active Feature-value Acquisition 11 2.1.3 Domain Adaptation 14 2.2 Review of Predictive Modelings in Manufacturing 15 2.2.1 Wafer Map Pattern Classification 15 2.2.2 Fault Detection and Classification 16 3. Active Learning for Wafer Map Pattern Classification 19 3.1 Problem Description 19 3.2 Proposed Method 21 3.2.1 System overview 21 3.2.2 Prediction model 25 3.2.3 Uncertainty estimation 25 3.2.4 Query wafer selection 29 3.2.5 Query wafer labeling 30 3.2.6 Model update 30 3.3 Experiments 31 3.3.1 Data description 31 3.3.2 Experimental design 31 3.3.3 Results and discussion 34 4. Active Cluster Annotation for Wafer Map Pattern Classification 42 4.1 Problem Description 42 4.2 Proposed Method 44 4.2.1 Clustering of unlabeled data 46 4.2.2 CNN training with labeled data 48 4.2.3 Cluster-level uncertainty estimation 49 4.2.4 Query cluster selection 50 4.2.5 Cluster-level annotation 50 4.3 Experiments 51 4.3.1 Data description 51 4.3.2 Experimental setting 51 4.3.3 Clustering results 53 4.3.4 Classification performance 54 4.3.5 Analysis for label noise 57 5. Active Inspection for Fault Prediction 60 5.1 Problem Description 60 5.2 Proposed Method 65 5.2.1 Active inspection framework 65 5.2.2 Acquisition based on Expected Prediction Change 68 5.3 Experiments 71 5.3.1 Data description 71 5.3.2 Fault prediction models 72 5.3.3 Experimental design 73 5.3.4 Results and discussion 74 6. Adaptive Fault Detection for Recipe Transition 76 6.1 Problem Description 76 6.2 Proposed Method 78 6.2.1 Overview 78 6.2.2 Unsupervised adaptation phase 81 6.2.3 Semi-supervised adaptation phase 83 6.3 Experiments 85 6.3.1 Data description 85 6.3.2 Experimental setting 85 6.3.3 Performance degradation caused by recipe transition 86 6.3.4 Effect of unsupervised adaptation 87 6.3.5 Effect of semi-supervised adaptation 88 7. Conclusion 91 7.1 Contributions 91 7.2 Future work 94Docto

A Systematic Literature Survey of Unmanned Aerial Vehicle Based Structural Health Monitoring

Unmanned Aerial Vehicles (UAVs) are being employed in a multitude of civil applications owing to their ease of use, low maintenance, affordability, high-mobility, and ability to hover. UAVs are being utilized for real-time monitoring of road traffic, providing wireless coverage, remote sensing, search and rescue operations, delivery of goods, security and surveillance, precision agriculture, and civil infrastructure inspection. They are the next big revolution in technology and civil infrastructure, and it is expected to dominate more than $45 billion market value. The thesis surveys the UAV assisted Structural Health Monitoring or SHM literature over the last decade and categorize UAVs based on their aerodynamics, payload, design of build, and its applications. Further, the thesis presents the payload product line to facilitate the SHM tasks, details the different applications of UAVs exploited in the last decade to support civil structures, and discusses the critical challenges faced in UASHM applications across various domains. Finally, the thesis presents two artificial neural network-based structural damage detection models and conducts a detailed performance evaluation on multiple platforms like edge computing and cloud computing