A Survey on Audio-Video based Defect Detection through Deep Learning in Railway Maintenance

Within Artificial Intelligence, Deep Learning (DL) represents a paradigm that has been showing unprecedented performance in image and audio processing by supporting or even replacing humans in defect and anomaly detection. The Railway sector is expected to benefit from DL applications, especially in predictive maintenance applications, where smart audio and video sensors can be leveraged yet kept distinct from safety-critical functions. Such separation is crucial, as it allows for improving system dependability with no impact on its safety certification. This is further supported by the development of DL in other transportation domains, such as automotive and avionics, opening for knowledge transfer opportunities and highlighting the potential of such a paradigm in railways. In order to summarize the recent state-of-the-art while inquiring about future opportunities, this paper reviews DL approaches for the analysis of data generated by acoustic and visual sensors in railway maintenance applications that have been published until August 31st, 2021. In this paper, the current state of the research is investigated and evaluated using a structured and systematic method, in order to highlight promising approaches and successful applications, as well as to identify available datasets, current limitations, open issues, challenges, and recommendations about future research directions

A novel data augmentation method for improved visual crack detection using generative adversarial networks

Condition monitoring and inspection are core activities for assessing and evaluating the health of critical infrastructure spanning from road networks to nuclear power stations. Defect detection on visual inspections of such assets is a field that enjoys increasing attention. However, data-based models are prone to a lack of available data depicting cracks of various modalities and present a great data imbalance. This paper introduces a novel data augmentation technique by deploying the CycleGan Generative Adversarial Network (GAN). The proposed model is deployed between different image datasets depicting cracks, with a nuclear application as the main industrial example. The aim of this network is to improve the segmentation accuracy on these datasets using deep convolutional neural networks. The proposed GAN generates realistic images that are challenging to segment and under-represented in the original datasets. Different deep networks are trained with the augmented datasets while introducing no labelling overhead. A comparison is drawn between the performance of the different neural networks on the original data and their augmented counterparts. Extensive experiments suggest that the proposed augmentation method results in superior crack detection in challenging cases across all datasets. This is reflected by the respective increase in the quantitative evaluation metrics

Augmented classification for electrical coil winding defects

A green revolution has accelerated over the recent decades with a look to replace existing transportation power solutions through the adoption of greener electrical alternatives. In parallel the digitisation of manufacturing has enabled progress in the tracking and traceability of processes and improvements in fault detection and classification. This paper explores electrical machine manufacture and the challenges faced in identifying failures modes during this life cycle through the demonstration of state-of-the-art machine vision methods for the classification of electrical coil winding defects. We demonstrate how recent generative adversarial networks can be used to augment training of these models to further improve their accuracy for this challenging task. Our approach utilises pre-processing and dimensionality reduction to boost performance of the model from a standard convolutional neural network (CNN) leading to a significant increase in accuracy

A Literature Review of Fault Diagnosis Based on Ensemble Learning

The accuracy of fault diagnosis is an important indicator to ensure the reliability of key equipment systems. Ensemble learning integrates different weak learning methods to obtain stronger learning and has achieved remarkable results in the field of fault diagnosis. This paper reviews the recent research on ensemble learning from both technical and field application perspectives. The paper summarizes 87 journals in recent web of science and other academic resources, with a total of 209 papers. It summarizes 78 different ensemble learning based fault diagnosis methods, involving 18 public datasets and more than 20 different equipment systems. In detail, the paper summarizes the accuracy rates, fault classification types, fault datasets, used data signals, learners (traditional machine learning or deep learning-based learners), ensemble learning methods (bagging, boosting, stacking and other ensemble models) of these fault diagnosis models. The paper uses accuracy of fault diagnosis as the main evaluation metrics supplemented by generalization and imbalanced data processing ability to evaluate the performance of those ensemble learning methods. The discussion and evaluation of these methods lead to valuable research references in identifying and developing appropriate intelligent fault diagnosis models for various equipment. This paper also discusses and explores the technical challenges, lessons learned from the review and future development directions in the field of ensemble learning based fault diagnosis and intelligent maintenance

Classification, Localization, and Quantification of Structural Damage in Concrete Structures using Convolutional Neural Networks

Applications of Machine Learning (ML) algorithms in Structural Health Monitoring (SHM) have recently become of great interest owing to their superior ability to detect damage in engineering structures. ML algorithms used in this domain are classified into two major subfields: vibration-based and image-based SHM. Traditional condition survey techniques based on visual inspection have been the most widely used for monitoring concrete structures in service. Inspectors visually evaluate defects based on experience and engineering judgment. However, this process is subjective, time-consuming, and hampered by difficult access to numerous parts of complex structures. Accordingly, the present study proposes a nearly automated inspection model based on image processing, signal processing, and deep learning for detecting defects and identifying damage locations in typically inaccessible areas of concrete structures. The work conducted in this thesis achieved excellent damage localization and classification performance and could offer a nearly automated inspection platform for the colossal backlog of ageing civil engineering structures

A Digital Triplet for Utilizing Offline Environments to Train Condition Monitoring Systems for Rolling Element Bearings

Manufacturing competitiveness is related to making a quality product while incurring the lowest costs. Unexpected downtime caused by equipment failure negatively impacts manufacturing competitiveness due to the ensuing defects and delays caused by the downtime. Manufacturers have adopted condition monitoring (CM) techniques to reduce unexpected downtime to augment maintenance strategies. The CM adoption has transitioned maintenance from Breakdown Maintenance (BM) to Condition-Based Maintenance (CbM) to anticipate impending failures and provide maintenance actions before equipment failure. CbM is the umbrella term for maintenance strategies that use condition monitoring techniques such as Preventive Maintenance (PM) and Predictive Maintenance (PdM). Preventive Maintenance involves providing periodic checks based on either time or sensory input. Predictive Maintenance utilizes continuous or periodic sensory inputs to determine the machine health state to predict the equipment failure. The overall goal of the work is to improve bearing diagnostic and prognostic predictions for equipment health by utilizing surrogate systems to generate failure data that represents production equipment failure, thereby providing training data for condition monitoring solutions without waiting for real world failure data. This research seeks to address the challenges of obtaining failure data for CM systems by incorporating a third system into monitoring strategies to create a Digital Triplet (DTr) for condition monitoring to increase the amount of possible data for condition monitoring. Bearings are a critical component in rotational manufacturing systems with wide application to other industries outside of manufacturing, such as energy and defense. The reinvented DTr system considers three components: the physical, surrogate, and digital systems. The physical system represents the real-world application in production that cannot fail. The surrogate system represents a physical component in a test system in an offline environment where data is generated to fill in gaps from data unavailable in the real-world system. The digital system is the CM system, which provides maintenance recommendations based on the ingested data from the real world and surrogate systems. In pursuing the research goal, a comprehensive bearing dataset detailing these four failure modes over different collection operating parameters was created. Subsequently, the collections occurred under different operating conditions, such as speed-varying, load-varying, and steadystate. Different frequency and time measures were used to analyze and identify differentiating criteria between the different failure classes over the differing operating conditions. These empirical observations were recreated using simulations to filter out potential outliers. The outputs of the physical model were combined with knowledge from the empirical observations to create ”spectral deltas” to augment existing bearing data and create new failure data that resemble similar frequency criteria to the original data. The primary verification occurred on a laboratory-bearing test stand. A conjecture is provided on how to scale to a larger system by analyzing a larger system from a local manufacturer. From the subsequent analysis of machine learning diagnosis and prognosis models, the original and augmented bearing data can complement each other during model training. The subsequent data substitution verifies that bearing data collected under different operating conditions and sizes can be substituted between different systems. Ostensibly, the full formulation of the digital triplet system is that bearing data generated at a smaller size can be scaled to train predictive failure models for larger bearing sizes. Future work should consider implementing this method for other systems outside of bearings, such as gears, non-rotational equipment, such as pumps, or even larger complex systems, such as computer numerically controlled machine tools or car engines. In addition, the method and process should not be restricted to only mechanical systems and could be applied to electrical systems, such as batteries. Furthermore, an investigation should consider further data-driven approximations to specific bearing characteristics related to the stiffness and damping parameters needed in modeling. A final consideration is for further investigation into the scalability quantities within the data and how to track these changes through different system levels

A New Generative Adversarial Network for Improving Classification Performance for Imbalanced Data

Data is a common issue in many industries, particularly in fields such as fraud detection and medical diagnosis. Imbalanced data refers to datasets where the distribution of classes is not equal, resulting in an over- representation of one class and an under-representation of another. This can lead to biassed and inaccurate machine learning models, as the algorithm may be inclined to favour the majority class and overlook important patterns in the minority class. Various sectors have utilised deep neural networks for data synthesis. However, according to research papers in these fields, balanced data outperforms imbalanced data when it comes to deep neural networks. Although deep generative approaches, such as Generative Adversarial Networks (GANs), are an efficient method of augmenting high-dimensional data, there is a lack of research on their effectiveness with credit card or breast cancer data and the current methods demonstrate limitations. Our research focuses on obtaining a great number of sets of data that are valid and resemble the minority class, in this case, fraudulent or malignant samples. Having more data like this can be used to train a binary classifier so it's effective against fraud or cancer diagnosis. To overcome challenges opposed to existing methods we have developed a novel GAN-based method called K-CGAN, which has been tested on credit card fraud and breast cancer data. K- CGAN is designed to generate synthetic data that resembles the minority class, effectively balancing the dataset and improving the performance of binary classifiers. Our research demonstrates the effectiveness of K-CGAN in handling complex data imbalance problems often encountered in practical applications. In addition, the experiments performed on different datasets indicate that K-CGAN can be used for various purposes. The application of machine learning algorithms in various industries has become increasingly popular in recent years. However, the quality and quantity of available data are crucial factors that directly impact the accuracy and reliability of these models. The scarcity and imbalance of datasets in certain domains pose challenges for researchers and practitioners, and the need for effective solutions is more pressing than ever. In this context, K- CGAN provides a promising approach to address data imbalance and improve the performance of machine learning models. Our results show that K-CGAN can be applied to different datasets with different characteristics, making it a valuable tool for data scientists and practitioners in various fields

Text Similarity Between Concepts Extracted from Source Code and Documentation

Context: Constant evolution in software systems often results in its documentation losing sync with the content of the source code. The traceability research field has often helped in the past with the aim to recover links between code and documentation, when the two fell out of sync. Objective: The aim of this paper is to compare the concepts contained within the source code of a system with those extracted from its documentation, in order to detect how similar these two sets are. If vastly different, the difference between the two sets might indicate a considerable ageing of the documentation, and a need to update it. Methods: In this paper we reduce the source code of 50 software systems to a set of key terms, each containing the concepts of one of the systems sampled. At the same time, we reduce the documentation of each system to another set of key terms. We then use four different approaches for set comparison to detect how the sets are similar. Results: Using the well known Jaccard index as the benchmark for the comparisons, we have discovered that the cosine distance has excellent comparative powers, and depending on the pre-training of the machine learning model. In particular, the SpaCy and the FastText embeddings offer up to 80% and 90% similarity scores. Conclusion: For most of the sampled systems, the source code and the documentation tend to contain very similar concepts. Given the accuracy for one pre-trained model (e.g., FastText), it becomes also evident that a few systems show a measurable drift between the concepts contained in the documentation and in the source code.</p