Laser beams-based localization methods for Boom-type roadheader using underground camera non-uniform blur model

The efficiency of automatic underground tunneling is significantly depends on the localization accuracy and reliable for the Boom-type roadheader. In comparison with other underground equipment positioning methods, vision-based measurement has gained attention for its advantages of noncontact and no accumulated error. However, the harsh underground environment, especially the geometric errors brought by the vibration of the machine body to the underground camera model, has a certain influence on the accuracy and stability for the vision-based underground localization. In this paper, a laser beams-based localization methods for the machine body of Boom-type roadheader is presented, which can tackle the dense-dust, low illumination environment with the stray lights interference. Taking mining vibration into consideration, an underground camera non-uniform blur model that incorporate the two-layer glasses refraction effect was established to eliminate vibration errors. The blur model explicitly reveals the change of imaging optical path under the influence of vibration and double layer explosion-proof glass. On the basis of this, the underground laser beams extraction and positioning are presents, which is with well environmental adaptability, and the improved 2P3L (two-points-three-lines) localization model from line correspondences are developed. Experimental evaluation are designed to verify the performance of the proposed method, and the deblurring algorithm is investigated and evaluated. The results show that the proposed methods is effective to restore the blurred laser beams image that caused by the vibration, and can meet the precision need of roadheader body localization for roadway construction in coal mine

Kinematic Modeling Of An Automated Laser Line Scanning System

This research work describes the geometric coordinate transformation in an automated laser line scanning system caused by movements required for scanning a component surface. The elements of an automated laser scanning system (robot, laser line scanner, and the component coordinate system) function as a mechanical linkage to obtain a trajectory on a component surface. This methodology solves the forward kinematics, derives the component surface, and uses inverse kinematic equations to characterize the movement of the entire automated scanning system on point trajectory. To reach a point on the component, joint angles of robot have been calculated. As a result, trajectory path is obtained. This obtained robot poses on point trajectory of the component surface can be used as an input for future work that aims to develop optimal scan paths to collect “best” point cloud data sets. This work contributes in laser scanning inspection of component surfaces in manufacturing, remanufacturing, and reverse engineering applications

Camera Marker Networks for Pose Estimation and Scene Understanding in Construction Automation and Robotics.

The construction industry faces challenges that include high workplace injuries and fatalities, stagnant productivity, and skill shortage. Automation and Robotics in Construction (ARC) has been proposed in the literature as a potential solution that makes machinery easier to collaborate with, facilitates better decision-making, or enables autonomous behavior. However, there are two primary technical challenges in ARC: 1) unstructured and featureless environments; and 2) differences between the as-designed and the as-built. It is therefore impossible to directly replicate conventional automation methods adopted in industries such as manufacturing on construction sites. In particular, two fundamental problems, pose estimation and scene understanding, must be addressed to realize the full potential of ARC. This dissertation proposes a pose estimation and scene understanding framework that addresses the identified research gaps by exploiting cameras, markers, and planar structures to mitigate the identified technical challenges. A fast plane extraction algorithm is developed for efficient modeling and understanding of built environments. A marker registration algorithm is designed for robust, accurate, cost-efficient, and rapidly reconfigurable pose estimation in unstructured and featureless environments. Camera marker networks are then established for unified and systematic design, estimation, and uncertainty analysis in larger scale applications. The proposed algorithms' efficiency has been validated through comprehensive experiments. Specifically, the speed, accuracy and robustness of the fast plane extraction and the marker registration have been demonstrated to be superior to existing state-of-the-art algorithms. These algorithms have also been implemented in two groups of ARC applications to demonstrate the proposed framework's effectiveness, wherein the applications themselves have significant social and economic value. The first group is related to in-situ robotic machinery, including an autonomous manipulator for assembling digital architecture designs on construction sites to help improve productivity and quality; and an intelligent guidance and monitoring system for articulated machinery such as excavators to help improve safety. The second group emphasizes human-machine interaction to make ARC more effective, including a mobile Building Information Modeling and way-finding platform with discrete location recognition to increase indoor facility management efficiency; and a 3D scanning and modeling solution for rapid and cost-efficient dimension checking and concise as-built modeling.PHDCivil EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113481/1/cforrest_1.pd

Excavator Pose Estimation for Safety Monitoring by Fusing Computer Vision and RTLS Data

The construction industry is considered as a hazardous industry because of its high number of accidents and fatality rates. Safety is one of the main requirements on construction sites since an insecure site drops the morale of the workers, which can also result in lower productivity. To address safety issues, many proactive methods have been introduced by researchers and equipment manufacturers. Studying these methods shows that most of them are using radio-based technologies that perform based on the locations of the attached sensors to the moving objects, which could be expensive and impractical for the large fleet of available construction equipment. Safety monitoring is a sensitive task and avoiding collisions requires a detailed information of the articulated equipment (e.g. excavators) and the motion of each part of that equipment. Therefore, it is necessary to install the location sensors on each moving part of the equipment for estimating its pose, which is a difficult, time consuming, and expensive task. On the other hand, the application of Computer Vision (CV) techniques is growing and becoming more practical and affordable. However, most of the available CV-based techniques evaluate the proximity of the resources by considering each object as a single point regardless of its shape and pose. Moreover, the process of manually collecting and annotating a large image dataset of different pieces of equipment is one of the most time consuming tasks. Furthermore, relying on a single source of data may not only decrease the accuracy of the pose estimation system because of missing data or calculation errors, but it may also increase the computation time. Moreover, when there are multiple objects and equipment in the field of view of each camera, CV-based algorithms are under a higher risk of false recognition of the equipment and their parts. Therefore, fusing the cameras’ data with data from Real-Time Location System (RTLS) can help the pose estimation system by limiting the search area for the parts’ detectors, and consequently reducing the processing time and improving the accuracy by reducing the false detections. This research aims to estimate the excavator pose by fusing CV and RTLS data for safety monitoring and has the following objectives: (1) improving the CV training by developing a method to automatically generate and annotate around-view synthetic images of equipment and their parts using the 3D model of the equipment and the real images of the construction sites as background; (2) developing a guideline for applying stereo vision system in construction sites using regular surveillance cameras with long baseline at a high level; (3) improving the accuracy and speed of CV detection by fusing RTLS data with cameras’ data; and (4) estimating the 3D pose of the equipment for detecting potential collisions based on a pair of Two Dimensional (2D) skeletons of the parts from the views of two cameras. To support these objectives, a comprehensive database of the synthetic images of the excavator and its parts are generated, and multiple detectors from multiple views are trained for each part of the excavator using the image database. Moreover, the RTLS data, providing the location of the equipment, are linked with the corresponding video frames from two cameras to fuse the location data with the video data. Knowing the overall size of the equipment and its location provided by the RTLS system, a virtual cylinder defined around the equipment is projected on the video frames to limit the search scope of the object detection algorithm within the projected cylinder, resulting in a faster processing time and higher detection accuracy. Additionally, knowing the equipment ID assigned to each RTLS device and the cameras’ locations and heights, it is possible to select the suitable detectors for each equipment. After detecting a part, the background of the detected bounding box are removed to estimate the location and orientation of each part. The final skeleton of the excavator is derived by connecting the start and end points of the parts to their adjacent parts knowing the kinematic information of the excavator. Estimating the skeleton of the excavator from each camera view on one hand, and knowing the extrinsic and intrinsic parameters of all available cameras on the construction site, on the other hand, are used for estimating the 3D pose by triangulating the estimated skeleton from each camera. In order to use the available collision avoidance systems, the 3D pose of the excavator is sent to the game environment and the potential collisions are detected followed by generating a warning. The contributions of this research are: (1) developing a method for creating and annotating the synthetic images of the construction equipment and their parts using the equipment 3D models and the real images of the construction sites; (2) creating and training the HOG-based excavator’s parts detectors using the database of the synthetic images developed earlier and automatically produced negative samples from the other excavator parts in addition to the real images of different construction sites while the target object is cut from these; (3) developing a data fusion framework after calibrating two regular surveillance cameras with the long baseline to integrate the RTLS data received from GPS with the video data from the cameras to decrease the processing efforts for detecting excavator parts while increasing the detection accuracy by limiting the search scope for the detectors; (4) developing a clustering technique to subtract parts’ background and extracting the 2D skeleton of the excavator in each camera’s view and to estimate the 3D pose of the excavator; and (5) transferring the 3D pose data of the excavator to the game environment using TCP/IP connection and visualizing the near real-time pose of the excavator in the game engine for detecting the potential collisions