3D Matching of resource vision tracking trajectories

Issues related to management and workforce play a key role in the productivity gap of construction and manufacturing. Both issues are directly related to the way productivity is measured. Current measurement methods tend to be ineffective because they are labour intensive, costly and prone to human errors whereas they are mainly reactive processes initiated after the detection of a negatively influencing factor. So far, research efforts in automating the measuring process have not achieved full automation because they require prior knowledge of the type of tasks performed in specific working zones. This is associated with the lack of depth information. For this purpose, this paper proposes a computationally efficient computer vision method for matching construction workers across different frames based on epipolar geometry, template and motion matching methods. The main result of this process is to provide a method for the acquisition of the 4D features (x, y, z, t) that compose the detailed profile of a construction activity in terms of both time and space.This is the author accepted manuscript. The final version is available from the American Society of Civil Engineers via https://doi.org/10.1061/9780784479827.17

3D Matching of Resource Vision Tracking Trajectories

Three-dimensional (3D) paths of resources have been proposed in construction management, as an efficient way for measuring labor productivity. These paths are either extracted by using sensors such as global positioning system (GPS), radio frequency identification (RFID), and ultra-wideband (UWB), or based on cameras placed at jobsites for surveillance purposes. However, the tag-based methods are seriously limited by privacy conflicts since they are not welcome from the personnel. On the other hand, the computer vision based methods have not achieved full automation in measuring labour productivity because they require prior knowledge of the type of tasks performed in specific working zones. This is associated with the lack of depth information. For this purpose, this paper proposes a computationally efficient computer vision method for matching construction workers across different frames. Entity matching is a process that corresponds to a compulsory step prior to the calculation of the 3D position. The proposed matching method, is based on epipolar geometry, template and motion similarity features. The main result of this process is to provide a method for the acquisition of the 3D paths that compose the detailed profile of a construction activity in terms of both time and space.This is the author accepted manuscript. The final version is available from the American Society of Civil Engineers via https://doi.org/10.1061/9780784479827.17

THE IMPORTANCE OF VISION-BASED TECHNOLOGIES FOR PROGRESS MONITORING AND PRODUCTIVITY ASSESSMENT OF EARTHMOVING OPERATIONS

Praćenje progresa izvedbe zemljanih radova uz točnu procjenu produktivnosti građevinskih strojeva omogućuje detaljan uvid u tijek izvedbe, ranu detekciju slabe produktivnosti, kao i svih ostalih mogućih nedostataka, povratnu informaciju ispravnosti donesnih odluka te precizniji iskaz potrebnog vremena i troška aktivnosti. Ranim uočavanjem, i alarmiranjem, svih rizičnih, nepovoljnih radnji, pružene su mogućnosti za pravodobno poduzimanje odgovarajućih, korektivnih mjera i provedbu poboljšanja. Bežične tehnologije pružaju znatan potencijal za primjenu u svrhu praćenja progresa rada i procjene produktivnosti. Međutim, dosadašnja istraživanja ukazuju na nedostatke i ograničenja. Potrebna su daljnja istraživanja njihovih velikih potencijala, a moguće rješenje problematike i kompleksnosti u praćenju progresa rada i procjene produktivnosti predstavlja integracija različitih bežičnih tehnologija. Vizualne tehnologije su, pritom, neizostavno područje bežičnih tehnologija za razvoj odgovarajuće, pouzdane, vjerodostojne, brze i ekonomične metodologije za praćenje progresa izvedbe zemljanih radova uz točnu procjenu produktivnosti građevinskih strojeva.Monitoring the progress of earthmoving operations while accurately estimating productivity of construction equipment provides a detailed insight into the performance, early detection of low productivity, as well as any other possible defects. It also provides feedback on the correctness of the decisions made and a more accurate report of the time and cost necessary for the activity. Early detection and warning of all the risky, unfavorable actions provides opportunities to timely take appropriate corrective measures and make improvements. Wireless technologies offer considerable potential for application in order to monitor work progress and evaluate productivity. However, the previous studies indicate shortcomings and limitations. Future research attention on their considerable potential is needed. The integration of various wireless technologies is a possible solution to the problem and complexity of monitoring work progress and productivity estimates. In this matter, vision-based technologies are an indispensable field of wireless technologies for the development of an appropriate, reliable, credible, fast and cost-effective methodology of monitoring the progress of earthmoving works with accurate estimation of the productivity of construction machines

Estimation of the excavator actual productivity at the construction site using video analysis

Current estimates of the actual productivity of heavy construction machinery at a construction site are not supported by an appropriate and widely used methodology. Recently, for the purpose of estimating the actual productivity of heavy construction machinery, visionbased technologies are used. This paper emphasizes the importance of estimating actual productivity and presents a way (i.e. a research framework) to achieve it. Therefore, the aim of this paper is to propose a simple research framework (SRF) for quick and practical estimates of excavator actual productivity and cycle time at a construction site. The excavator actual productivity refers to the maximum possible productivity in real construction site conditions. The SRF includes the use of a video camera and the analysis of recorded videos using an advanced computer program. In cases of continuous application of SRF, a clear and transparent base for monitoring and control of earthworks can be obtained at an observed construction site

Characterizing construction equipment activities in long video sequences of earthmoving operations via kinematic features

This thesis presents a fast and scalable method for activity analysis of construction equipment involved in earthmoving operations from highly varying long-sequence videos obtained from fixed cameras. A common approach to characterize equipment activities consists of detecting and tracking the equipment within the video volume, recognizing interest points and describing them locally, followed by a bag-of-words representation for classifying activities. While successful results have been achieved in each aspect of detection, tracking, and activity recognition, the highly varying degree of intra-class variability in resources, occlusions and scene clutter, the difficulties in defining visually-distinct activities, together with long computational time have challenged scalability of current solutions. In this thesis, we present a new end-to-end automated method to recognize the equipment activities by simultaneously detecting and tracking features, and characterizing the spatial kinematics of features via a decision tree. The method is tested on an unprecedented dataset of 5hr-long real-world videos of interacting pairs of excavators and trucks. The Experimental results show that the method is capable of activity recognition with accuracy of 88.91% with a computational time less than 1- to-1 ratio for each video length. The benefits of the proposed method for root-cause assessment of performance deviations are discussed

Automatic estimation of excavator actual and relative cycle times in loading operations

This paper proposes a framework to automatically determine the productivity and operational effectiveness of an excavator. The method estimates the excavator\u27s actual, theoretical, and relative cycle times in the loading operation. Firstly, a supervised learning algorithm is proposed to recognize excavator activities using motion data obtained from four inertial measurement units (IMUs) installed on different moving parts of the machine. The classification algorithm is offline trained using a dataset collected via an excavator operated by two operators with different levels of competence in different operating conditions. Then, an approach is presented to estimate the cycle time based on the sequence of activities detected using the trained classification model. Since operating conditions can significantly influence the cycle time, the actual cycle time cannot solely reveal the machine\u27s performance. Hence, a benchmark or reference is required to analyze the actual cycle time. In the second step, the theoretical cycle time of an excavator is automatically estimated based on the operating conditions, such as swing angle and digging depth. Furthermore, two schemes are presented to estimate the swing angle and digging depth based on the recognized excavator activities. In the third step, the relative cycle time is obtained by dividing the theoretical cycle time by the actual cycle time. Finally, the results of the method are demonstrated by the implementation on two case studies which are operated by inexperienced and experienced operators. The obtained relative cycle time can effectively monitor the performance of an excavator in loading operations. The proposed method can be highly beneficial for worksite managers to monitor the performance of each machine in worksites

A Hybrid Kinematic-Acoustic System for Automated Activity Detection of Construction Equipment

Automatically recognizing and tracking construction equipment activities is the first step towards performance monitoring of a job site. Recognizing equipment activities helps construction managers to detect the equipment downtime/idle time in a real-time framework, estimate the productivity rate of each equipment based on its progress, and efficiently evaluate the cycle time of each activity. Thus, it leads to project cost reduction and time schedule improvement. Previous studies on this topic have been based on single sources of data (e.g., kinematic, audio, video signals) for automated activity-detection purposes. However, relying on only one source of data is not appropriate, as the selected data source may not be applicable under certain conditions and fails to provide accurate results. To tackle this issue, the authors propose a hybrid system for recognizing multiple activities of construction equipment. The system integrates two major sources of data-audio and kinematic-through implementing a robust data fusion procedure. The presented system includes recording audio and kinematic signals, preprocessing data, extracting several features, as well as dimension reduction, feature fusion, equipment activity classification using Support Vector Machines (SVM), and smoothing labels. The proposed system was implemented in several case studies (i.e., ten different types and equipment models operating at various construction job sites) and the results indicate that a hybrid system is capable of providing up to 20% more accurate results, compared to cases using individual sources of data

The impact of visual production management on construction project controls: a case-based reasoning

Over the past decade, production control theories such as the Last Planner System by Ballard, G., (2000) have emerged that stabilize workflows by shielding the direct work from upstream variation and uncertainty. Although theories have been well documented, yet their full-scale implementation is not realized, and the root-causes for this are not entirely understood. A large body of empirical observations suggest that successful implementation of control mechanisms requires dedicated facilitators and engages practitioners in a relatively deep learning process. Sustaining this level of commitment for the duration of a project is difficult, and in its absence, project teams revert to traditional project control practices. These barriers are in part attributed to the people and organizational processes involved in implementing lean principles, however there is a growing recognition among researchers that the functional aspects of production control techniques need close re-examination to understand better, predict, and analyze reliability in performance, and preserve effective and timely flow of information both to and from the workforce. To address these knowledge gaps, Lin and Golparvar-Fard (2016) proposes a visual project control system that a) improves understanding of how construction performance can be captured, communicated, and analyzed in form of a production system; b) predicts the reliability of the weekly work plan and look-ahead schedule, supports root-cause assessment on plan failure at both project and task-levels; c) facilitates information flows; and d) decentralizes decision-making. The web-based system which is built on visual data analytics maps the current state of production on construction sites in 3D and exposes waste at both project and task-levels and it then forecasts reliability in the future state of production to highlight potential issues in a location-driven scheme. The platform also supports collaborative decision making that eliminates root causes of waste and provides visual interfaces between people and information that enable effective pull flow, decentralize work tracking, and facilitate in-process quality control and hand-overs among contractors. To ensure their implementation does not take away from actual productivity, it extends the value of 4D Building Information Models (BIM) commonly used for constructability review as a benchmark for performance. It also leverages images and videos frequently collected by project participants or professional services via consumer-grade, time-lapse, smartphone cameras and Unmanned Aerial Vehicles (UAVs) to visually document actual performance. To better understand, assess, and improve the performance of this visual production system, a case-based reasoning study is conducted in this thesis using case studies based on two real-world construction projects. The use of simple and effective visuals of work-in-progress and at risk locations on construction sites offered through visual production management is assessed to better understand if such systems can improve reliability of short-term planning, enhance situational awareness, enable easy and quick root-cause assessment of plan failures, and facilitate flow of information onsite and during coordination sessions. The lessons learned and areas for further development in theory and technology are discussed in detail