Automatic Multi-Label Image Classification Model for Construction Site Images

학위논문(석사)--서울대학교 대학원 :공과대학 건축학과,2019. 8. 박문서.최근 이미지 분석 기술이 발전함에 따라 건설 현장에서 다양한 방면에서 현장에서 수집된 사진을 활용하여 건설 프로젝트를 관리하고자 하는 시도가 이루어지고 있다. 특히 촬영 장비의 발전되자 건설 현장에서 생산되는 사진의 수가 급증하여 건설 현장 사진의 잠재적인 활용도는 더욱 더 높아지고 있다. 하지만 이렇게 생산되는 많은 양의 사진은 대부분 제대로 분류되지 않은 상태로 보관되고 있기 때문에 현장 사진으로부터 필요한 프로젝트 정보를 추출하는 것은 매우 어려운 실정이다. 현재 현장에서 사진을 분류하는 방식은 사용자가 직접 개별 사진을 검토한 뒤 분류하기 때문에 많은 시간과 노력이 요구되고, 이미지 분류를 위한 특징을 직접적으로 추출하는 기존의 이미지 분석 기술 역시 복잡한 건설 현장 사진의 특징을 범용적으로 학습하는 데는 한계가 있다. 이에 본 연구에서는 건설 현장 사진의 모습이 매우 다양하고, 동적으로 변하는 것에 대응하기 위해 이미지 분류에서 높은 성능을 보이고 합성곱 신경망(Deep Convolutional Neural Network) 알고리즘을 적용하여 개별 건설 현장 사진에 적합한 키워드를 자동으로 할당할 수 있는 모델을 개발하고자 한다. 합성곱 신경망 모델은 모델 구조가 깊어짐에 따라 높은 차원의 항상성(invariant) 특징도 효과적으로 학습할 수 있는 특징이 있기 때문에 복잡한 건설 현장 사진 분류 문제에 적합하다. 따라서 본 연구에서는 합성곱 신경망 모델을 토대로 현장에서 필요한 사진을 빠르고 정확하게 찾을 수 있도록 각 사진에 적합한 키워드를 자동으로 할당하는 모델을 개발하였다. 특히, 건설 현장 사진의 대부분이 하나 이상의 레이블과 연관이 있다는 점에 기반하여 다중 레이블 분류 모델을 적용하였다. 이를 통해 일차적으로는 건설 사진에서 프로젝트와 관련된 다양한 정보를 추출하여 건설 현장 사진의 활용도를 개선하고, 나아가 사진 데이터를 활용하여 효율적인 건설 관리를 도모하고자 한다. 본 연구의 진행 순서는 다음과 같다. 우선 모델을 학습시키기 위해서 실제 건설 현장 및 오픈소스 검색엔진을 통하여 총 6개 공종의 사진을 수집하고, 하위 분류 범위를 포함한 총 10개 레이블의 데이터셋을 구성하여 학습을 진행했다. 또한 구체적인 모델 선택을 위해 대표적인 합성곱 신경망 모델을 비교 검토하여 가장 우수한 성능을 보인 ResNet 18을 최종 모델로 선택했다. 실험 결과 평균 91%의 정확도를 보이며 건설 현장 사진을 자동으로 분류할 수 있는 가능성을 확인하였다. 또한 본 연구는 최근 타 분야 이미지 분석에서 좋은 성과를 보인 합성곱 신경망을 활용하여 건설 현장 사진을 자동으로 분류할 수 있다는 가능성을 확인했다는 점과, 건설 현장 사진 분류 문제에 다중 레이블 분류를 적용한 첫 연구라는 점에서 의의가 있다. 실제 현장에서는 사진을 자동으로 분류할 수 있게 됨에 따라 기존에 번거로운 수동 사진 분류 작업을 줄이고, 건설 현장 사진의 활용도를 높일 수 있을 것으로 기대된다. 하지만 본 연구는 각 레이블 간에 연관성이나 의존성을 고려하지 않기 때문에 추후 연구에서는 각 사진 간의 계층적 관계를 모델에 추가적으로 학습시켜 정확도를 높이고, 학습 레이블도 더 낮은 단계의 키워드까지 포함하여 현장 사진으로부터 보다 다양한 정보를 얻을 수 있도록 모델을 개선하는 것을 목표로 하고 있다.Activity recognition in construction performs as the prerequisite step in the process for various tasks and thus is critical for successful project management. In the last several years, the computer vision community has blossomed, taking advantage of the exploding amount of construction images and deploying the visual analytics technology for cumbersome construction tasks. However, the current annotation practice itself, which is a critical preliminary step for prompt image retrieval and image understanding, is remained as both time-consuming and labor-intensive. Because previous attempts to make the process more efficient were inappropriate to handle dynamic nature of construction images and showed limited performance in classifying construction activities, this research aims to develop a model which is not only robust to a wide range of appearances but also multi-composition of construction activity images. The proposed model adopts a deep convolutional neural network model to learn high dimensional feature with less human-engineering and annotate multi-labels of semantic information in the images. The result showed that our model was capable of distinguishing different trades of activities at different stages of the activity. The average accuracy of 83% and a maximum accuracy of 91% holds promise in an actual implementation of automated activity recognition for construction operations. Ultimately, it demonstrated a potential method to provide automated and reliable procedure to monitor construction activity.Chapter 1. Introduction 1 1.1. Research Background 1 1.2. Research Objectives and Scope 5 1.3. Research Outline 7 Chapter 2. Preliminary Study 9 2.1. Challenges with Construction Activity Image Classification Task 10 2.2. Applications of Traditional Vision-based Algorithms in Construction Domain 13 2.3. Convolutional Neural Network-based Image Classification in Construction Domain 18 2.4. Summary 21 Chapter 3. Development of Construction Image Classification Model 22 3.1. Customized Construction Image Dataset Preparation 23 3.1.1. Construction Activity Classification System 23 3.1.2. Dataset Collection 24 3.1.3. Data Pre-Processing 25 3.2. Construction Image Classification Model Framework 27 3.2.1. Multi-label Image Classification 27 3.2.2. Base CNN Model Selection 28 3.2.3. Proposed ResNet Model Architecture 29 3.3. Model Training and Validation 33 3.3.1. Transfer Learning 33 3.3.2. Loss Computation and Model Optimization 33 3.3.3. Model Performance Indicator 35 3.4. Summary 37 Chapter 4. Experiment Results and Discussion 38 4.1. Experiment Results 38 4.2. Analysis of Experiment Results 42 4.3. Summary 44 Chapter 5. Conclusion 45 5.1. Research Summary 45 5.2. Research Contributions 46 5.3. Limitations and Further Study 47 References 49 Appendix 57 Abstract in Korean 63Maste

Human-Tool-Interaction-Based Action Recognition Framework for Automatic Construction Operation Monitoring

Monitoring activities on a construction jobsite is one of the most important tasks that a construction management team performs every day. Construction management teams monitor activities to ensure that a construction project progresses as scheduled and that the construction crew works properly in a safe working environment. However, site monitoring is often time-consuming. Various automated or semi-automated tracking approaches such as radio frequency identification, Global Positioning System, ultrawide band, barcode, and laser scanning have been introduced to better monitor activities on the construction site. However, deploying and maintaining such techniques require a high level of involvement by very specific well-trained professionals and could be costly. As an alternative way to monitor sites, object recognition and tracking have the advantage of requiring low human involvement and intervention. However, it is still a challenge to recognize construction crew activities with existing methods, which have a high false recognition rate. This research proposes a new approach for recognizing construction personnel activity from still images or video frames. The new approach mimics the human thinking process with the assumption that a construction worker performs a certain activity with a specific body pose using a specific tool. The new approach consists of two recognition tasks, construction worker pose recognition and tool recognition. The two recognition tasks are connected in sequence with an interactive spatial relationship. The proposed method was developed into a computer application using Matlab. It was compared against a benchmark method that only uses construction worker body pose for activity recognition. The benchmark method was also developed into a computer application with Matlab. The proposed method and the benchmark method were tested with the same sample set containing 500 images of over 10 different construction activities. The experimental results show that the proposed framework achieved a higher reliability (precision value), a lower sensitivity (recall value), and an overall better performance (F₁ score) than the benchmark method

Characterizing construction equipment activities in long video sequences of earthmoving operations via kinematic features

This thesis presents a fast and scalable method for activity analysis of construction equipment involved in earthmoving operations from highly varying long-sequence videos obtained from fixed cameras. A common approach to characterize equipment activities consists of detecting and tracking the equipment within the video volume, recognizing interest points and describing them locally, followed by a bag-of-words representation for classifying activities. While successful results have been achieved in each aspect of detection, tracking, and activity recognition, the highly varying degree of intra-class variability in resources, occlusions and scene clutter, the difficulties in defining visually-distinct activities, together with long computational time have challenged scalability of current solutions. In this thesis, we present a new end-to-end automated method to recognize the equipment activities by simultaneously detecting and tracking features, and characterizing the spatial kinematics of features via a decision tree. The method is tested on an unprecedented dataset of 5hr-long real-world videos of interacting pairs of excavators and trucks. The Experimental results show that the method is capable of activity recognition with accuracy of 88.91% with a computational time less than 1- to-1 ratio for each video length. The benefits of the proposed method for root-cause assessment of performance deviations are discussed

Audio-Based Productivity Forecasting of Construction Cyclic Activities

Due to its high cost, project managers must be able to monitor the performance of construction heavy equipment promptly. This cannot be achieved through traditional management techniques, which are based on direct observation or on estimations from historical data. Some manufacturers have started to integrate their proprietary technologies, but construction contractors are unlikely to have a fleet of entirely new and single manufacturer equipment for this to represent a solution. Third party automated approaches include the use of active sensors such as accelerometers and gyroscopes, passive technologies such as computer vision and image processing, and audio signal processing. Hitherto, most studies with these technologies have aimed to activity identification or to identifying active and idle times. Given that most actions performed with construction machinery involve cyclic activities, cycle time estimation is much more relevant. In this study, hardware and software requirements were optimized toward that goal. This approach had three facets: first, signal spectral analysis was performed through the short-time Fourier transform (STFT) and the continuous wavelet transform (CWT) for comparison; second, audio and active sensor data have been submitted to a machine learning framework for activity classification accuracy comparison; and, third, Bayesian statistical models were used to include historical data for cycle time estimation enhancement. As a result, audio signals have been used along with a Markov-chain-based filter to achieve cycle time estimation with an accuracy of over 81% for up to five days of single-machine operation

Data-Driven Simulation Modeling of Construction and Infrastructure Operations Using Process Knowledge Discovery

Within the architecture, engineering, and construction (AEC) domain, simulation modeling is mainly used to facilitate decision-making by enabling the assessment of different operational plans and resource arrangements, that are otherwise difficult (if not impossible), expensive, or time consuming to be evaluated in real world settings. The accuracy of such models directly affects their reliability to serve as a basis for important decisions such as project completion time estimation and resource allocation. Compared to other industries, this is particularly important in construction and infrastructure projects due to the high resource costs and the societal impacts of these projects. Discrete event simulation (DES) is a decision making tool that can benefit the process of design, control, and management of construction operations. Despite recent advancements, most DES models used in construction are created during the early planning and design stage when the lack of factual information from the project prohibits the use of realistic data in simulation modeling. The resulting models, therefore, are often built using rigid (subjective) assumptions and design parameters (e.g. precedence logic, activity durations). In all such cases and in the absence of an inclusive methodology to incorporate real field data as the project evolves, modelers rely on information from previous projects (a.k.a. secondary data), expert judgments, and subjective assumptions to generate simulations to predict future performance. These and similar shortcomings have to a large extent limited the use of traditional DES tools to preliminary studies and long-term planning of construction projects. In the realm of the business process management, process mining as a relatively new research domain seeks to automatically discover a process model by observing activity records and extracting information about processes. The research presented in this Ph.D. Dissertation was in part inspired by the prospect of construction process mining using sensory data collected from field agents. This enabled the extraction of operational knowledge necessary to generate and maintain the fidelity of simulation models. A preliminary study was conducted to demonstrate the feasibility and applicability of data-driven knowledge-based simulation modeling with focus on data collection using wireless sensor network (WSN) and rule-based taxonomy of activities. The resulting knowledge-based simulation models performed very well in properly predicting key performance measures of real construction systems. Next, a pervasive mobile data collection and mining technique was adopted and an activity recognition framework for construction equipment and worker tasks was developed. Data was collected using smartphone accelerometers and gyroscopes from construction entities to generate significant statistical time- and frequency-domain features. The extracted features served as the input of different types of machine learning algorithms that were applied to various construction activities. The trained predictive algorithms were then used to extract activity durations and calculate probability distributions to be fused into corresponding DES models. Results indicated that the generated data-driven knowledge-based simulation models outperform static models created based upon engineering assumptions and estimations with regard to compatibility of performance measure outputs to reality

Using Cost Simulation and Computer Vision to Inform Probabilistic Cost Estimates

Cost estimating is a critical task in the construction process. Building cost estimates using historical data from previously performed projects have long been recognized as one of the better methods to generate precise construction bids. However, the cost and productivity data are typically gathered at the summary level for cost-control purposes. The possible ranges of production rates and costs associated with the construction activities lack accuracy and comprehensiveness. In turn, the robustness of cost estimates is minimal. Thus, this study proposes exploring a range of cost and productivity data to better inform potential outcomes of cost estimates by using probabilistic cost simulation and computer vision techniques for activity production rate analysis. Chapter two employed the Monte Carlo Simulation approach to computing a range of cost outcomes to find the optimal construction methods for large-scale concrete construction. The probabilistic cost simulation approach helps the decision-makers better understand the probable cost consequences of different construction methods and to make more informed decisions based on the project characteristics. Chapter three experimented with the computer vision-based skeletal pose estimation model and recurrent neural network to recognize human activities. The activity recognition algorithm was employed to help interpret the construction activities into productivity information for automated labor productivity tracking. Chapter four implemented computer vision-based object detection and object tracking algorithms to automatically track the construction productivity data. The productivity data collected was used to inform the probabilistic cost estimates. The Monte Carlo Simulation was adopted to explore potential cost outcomes and sensitive cost factors in the overall construction project. The study demonstrated how the computer vision techniques and probabilistic cost simulation optimize the reliability of the cost estimates to support construction decision-making. Advisor: Philip Baruth

Analysis of the Performance of HOG and CNNs for Detecting Construction Equipment and Personal Protective Equipment

The construction industry remains one of the most dangerous working environments in terms of fatalities and accidents. High numbers of accidents and loss-time injuries, leads to a decrease in productivity in this industry. Therefore, new technologies are being developed to improve the safety of construction sites. Object detection on construction sites has a huge impact on the construction industry. Many researchers studied productivity, safety, and project progress. However, few efforts have been made to improve the robustness of the related datasets for detection purposes. In the meantime, it is noticed that the lack of a custom dataset leads to low accuracy and also an increase in the cost and time of training dataset preparation. In this research, we first investigated the generation of synthetic images using 3D models of construction equipment to use them as the datasets for training purposes, namely: excavators, loaders and trucks, and then sensitivity analysis is applied. We compared the performance of CNNs and other conventional methods for classifying construction equipment. In the second part, the detection of personal protective equipment for construction workers was studied. For this purpose, several object detection architectures from the TensorFlow object detection model zoo have been evaluated to find the best and most robust detection model. The dataset used in this study contains real images from construction sites. The performance evaluation of trained object detectors are measured in terms of mean average precision. The test results from this study showed that (1) synthetic images have a significant effect on the final detection results; and (2) comparing various object detection architectures, Faster_rcnn_resnet101 was the most suitable model in terms of accuracy of detection

Towards a Cyber-Physical Gaming System for Training in the Construction and Engineering Industry

Antidepressants are among the most commonly detected human pharmaceuticals in the aquatic environment. Since their mode of action is by modulating the neurotransmitters serotonin, dopamine, and norepinephrine, aquatic invertebrates who possess transporters and receptors sensitive to activation by these pharmaceuticals are potentially affected by them. We review the various types of antidepressants, their occurrence and concentrations in aquatic environments, and the actions of neurohormones modulated by antidepressants in molluscs and crustaceans. Recent studies on the effects of antidepressants on these two important groups show that molluscan reproductive and locomotory systems are affected by antidepressants at environmentally relevant concentrations. In particular, antidepressants affect spawning and larval release in bivalves and disrupt locomotion and reduce fecundity in snails. In crustaceans, antidepressants affect freshwater amphipod activity patterns, marine amphipod photo- and geotactic behavior, crayfish aggression, and daphnid reproduction and development. We note with interest the occurrence of non-monotonic dose responses curves in many studies on effects of antidepressants on aquatic animals, often with effects at low concentrations, but not at higher concentrations, and we suggest future experiments consider testing a broader range of concentrations. Furthermore, we consider invertebrate immune responses, genomic and transcriptomic sequencing of invertebrate genes, and the ever-present and overwhelming question of how contaminant mixtures could affect the action of neurohormones as topics for future study. In addressing the question, if antidepressants affect aquatic invertebrates at concentrations currently found in the environment, there is strong evidence to suggest the answer is yes. Furthermore, the examples highlighted in this review provide compelling evidence that the effects could be quite multifaceted across a variety of biological systems

A deep learning approach towards railway safety risk assessment

Railway stations are essential aspects of railway systems, and they play a vital role in public daily life. Various types of AI technology have been utilised in many fields to ensure the safety of people and their assets. In this paper, we propose a novel framework that uses computer vision and pattern recognition to perform risk management in railway systems in which a convolutional neural network (CNN) is applied as a supervised machine learning model to identify risks. However, risk management in railway stations is challenging because stations feature dynamic and complex conditions. Despite extensive efforts by industry associations and researchers to reduce the number of accidents and injuries in this field, such incidents still occur. The proposed model offers a beneficial method for obtaining more accurate motion data, and it detects adverse conditions as soon as possible by capturing fall, slip and trip (FST) events in the stations that represent high-risk outcomes. The framework of the presented method is generalisable to a wide range of locations and to additional types of risks