Road distresses such as potholes can have a negative economic and social impact. The timeous detection and identification of potholes could expedite the maintenance and repair of potholes. The research team previously investigated and reported on the Visual Surveying Platform, which is a system that automatically detects and geo-tags potholes, with a detection accuracy of approximately 82%. At this level of accuracy, errors consisting largely of false positives could result in repair teams responding to non-existent potholes. In order to incorporate the detection system into the existing workflow of one of the municipalities in the Gauteng area, the detection accuracy needed to be improved. The research team modified the system to include a ?human-in-the-loop? mode of operation, where the detection system performs a more suggestive function. The mobile detection system automatically detects potholes in real-time and presents the detections to an operator for validation. The validated detections are then introduced into the operational workflow of the maintenance and repair teams. The ?human-in-the-loop? system and the operational workflow are described in detail in this paper.Paper presented at the 34th Annual Southern African Transport Conference 6-9 July 2015 "Working Together to Deliver - Sakha Sonke", CSIR International Convention Centre, Pretoria, South Africa.The Minister of Transport, South AfricaTransportation Research Board of the US

Chiwewe, Tapiwa Moses

Luvhengo, T.

Motloutsi, T.

Naidoo, T.

Tyatyantsi, A.

Paper presented at the 34th Annual Southern African Transport Conference 6-9 July 2015 "Working Together to Deliver - Sakha Sonke", CSIR International Convention Centre, Pretoria, South Africa.Road distresses such as potholes can have a negative economic and social impact. The timeous detection and identification of potholes could expedite the maintenance and repair of potholes. The research team previously investigated and reported on the Visual Surveying Platform, which is a system that automatically detects and geo-tags potholes, with a detection accuracy of approximately 82%. At this level of accuracy, errors consisting largely of false positives could result in repair teams responding to non-existent potholes. In order to incorporate the detection system into the existing workflow of one of the municipalities in the Gauteng area, the detection accuracy needed to be improved. The research team modified the system to include a ?human-in-the-loop? mode of operation, where the detection system performs a more suggestive function. The mobile detection system automatically detects potholes in real-time and presents the detections to an operator for validation. The validated detections are then introduced into the operational workflow of the maintenance and repair teams. The ?human-in-the-loop? system and the operational workflow are described in detail in this paper.The Minister of Transport, South AfricaTransportation Research Board of the US

UPSpace at the University of Pretoria

A HYBRID HUMAN MACHINE SYSTEM FOR THE DETECTION AND MANAGEMENT OF POTHOLES ON ASPHALT ROAD SURFACES  T Naidoo, T Chiwewe, T Luvhengo, T Motloutsi, A Tyatyantsi Council for Scientific and Industrial Research Meiring Naude Road, Pretoria, 0001, South Africa Tel: 012 841-2285; Email: tnaidoo@csir.co.za   ABSTRACT  Road distresses such as potholes can have a negative economic and social impact. The timeous detection and identification of potholes could expedite the maintenance and repair of potholes. The research team previously investigated and reported on the Visual Surveying Platform, which is a system that automatically detects and geo-tags potholes, with a detection accuracy of approximately 82%. At this level of accuracy, errors consisting largely of false positives could result in repair teams responding to non-existent potholes. In order to incorporate the detection system into the existing workflow of one of the municipalities in the Gauteng area, the detection accuracy needed to be improved. The research team modified the system to include a “human-in-the-loop” mode of operation, where the detection system performs a more suggestive function. The mobile detection system automatically detects potholes in real-time and presents the detections to an operator for validation. The validated detections are then introduced into the operational workflow of the maintenance and repair teams. The “human-in-the-loop” system and the operational workflow are described in detail in this paper.  Keywords – interactive machine learning, computer vision, image processing, geospatial information system, potholes, road maintenance.   1 INTRODUCTION  1.1.1. Background Roads are key to the day to day lives of people and must be inspected and repaired in a timely manner in order to ensure traffic safety, good riding quality and smooth traffic flow. In addition the continuous monitoring of road surface conditions would lead to an accumulation of condition data, gathered over time. The accumulated data could be used to support preventative maintenance, decision making, planning and investment by making use of data mining techniques.    In this research, road surface distresses are defined as "any indication of poor or unfavorable pavement performance or signs of impending failure; any unsatisfactory performance of a pavement short of failure” and may be grouped as: fractures in the form of cracking; distortions in the form of deformation; and disintegrations in the form of stripping, raveling or spalling. Potholes are one particular type of pavement distress that have become a cause of concern for the South African government, 145ISBN Number: 978-1-920017-63-7Proceedings of the 34th Southern African Transport Conference (SATC 2015)which launched a new initiative with a budget of R22 billion over three years for the repair of potholes, BuaNews (2011).   Potholes are generally caused by the ingress of water into the pavement structure of the road. The occurrence of potholes can be mitigated by routine road maintenance such as the sealing of surface cracks. To prevent pothole growth, existing potholes must be detected and repaired timeously. In addition to this, it is imperative to repair potholes in the correct manner in order to prevent the potholes from forming again.   Most current pavement inspection systems are manual, and are carried out by trained personnel who drive along the roads while making visual observations and measurements. The work is tedious and is often carried out under heavily trafficked conditions, which poses a risk to the safety of the inspectors. The manual method of inspection has clear drawbacks. As described by Mustaffara et al (2008), manual inspections are:  • Time consuming, costly and labor intensive. • Sometimes subjective and generate inconsistencies and inaccuracies in the determination of the pavement conditions. • Inflexible and does not provide an absolute measure of the surface. • Has poor repeatability since the assessment of given pavement section may be differ from one survey to the next. • Could pose a serious safety hazard to the surveyors due to high speed and high volume traffic, and may disrupt traffic flow.  An automated road inspection system can be developed to overcome these drawbacks and meet the needs of an efficient and economical inspection system.   1.2. Related Work There have been various approaches to automated sensing methods for the purpose of monitoring the condition of roads. Most methods make use of 3D laser scanners, cameras or accelerometers as sensors. The methods can be classified as being either (1) 3D reconstruction based, (2) vibration based or (3) computer vision based.   1.2.1. 3D reconstruction based approaches  With this approach, a 3D laser scanner, stereo camera, structured light based camera or a time-of-flight camera can be used to perform depth sensing in order to generate 3D point clouds. The point clouds are used to digitally reconstruct the road surface which is analyzed using custom algorithms. 3D laser scanners are relatively costly, which can limit widespread use. Camera based 3D reconstruction involves high computational effort and suffers from a limited range for depth sensing, as well as limited resolution as compared to the use of 3D laser scanners. 3D reconstruction has been investigated for the purpose of detecting road surface distresses by Li et al (2010), Chang et al (2005), Salari et al (2011).  1.2.2. Vibration based approaches  Vibration based approaches attempt to ‘feel’ what the road surface is like through the use of vibrations sensors such as accelerometers. This type of approach requires minimal storage, is cost effective and can be done in real time. Some of the drawbacks are that the readings are subjective to the vehicles mechanical responses 146and in the case of certain distresses, such as potholes, the vehicle must drive over the distress in order for it to be detected. Eriksson et al (2008) use accelerometer and GPS data to detect and geotag potholes respectively. The same idea is used in the BusNet project by De Zoysa et al (2007). Chen et al (2014) explored the use of crowdsourcing in detecting potholes and evaluating the roughness of road surfaces, where participating vehicles are fitted with an accelerometer and GPS device and the sensed data is sent to a central server for processing.  1.2.3. Vision based approaches  In vision based approaches, images of road surfaces are acquired using a suitable camera and analyzed to detect road surface distresses. The sensors used are relatively inexpensive; however this method generates vast amounts of data that must be intensively processed, using advanced image processing and computer vision algorithms. Machine learning techniques, such as artificial neural networks and support vector machines, have been used to detect and classify road surface distresses. Koch et al (2011) have investigated pothole detection in individual images, as well as in video sequences, Koch et al (2014). Their approach involved image segmentation, shape extraction and texture comparison to detect potholes. Rajab et al (2008) used image processing to estimate the area of potholes in images.   1.3. Scope of Paper This paper presents the further developments on the Visual Surveying Platform (VSP) as a project that aims to develop an integrated system to manage and maintain the condition of road infrastructure. The focus is on the pothole detection module (PDM) of the VSP platform. In previous work, a vision based system to automatically detect and geo-tag potholes with an accuracy of approximately 82%, was presented by Naidoo et al (2014). In recent developments the researchers have sought to deploy the system for use in a municipality within the Gauteng region. As part of the deployment, the researchers considered introducing a “human-in-the-loop” mode of detecting potholes. This introduction was primarily to cater for instances where there are false positive detections and to avoid deploying repair teams to places where there are actually no potholes. The system performs the automated detection of potholes and a human operator performs a validation step. The researchers have also made advances in the method for automatically detecting potholes.  This paper begins with a discussion of the integrated operational workflow involving the VSP and a typical existing maintenance system. An overview of the hybrid “human-in-the-loop” method of detection is provided. The technical details of the automatic detection of potholes and the latest results that were obtained are discussed.    1472 A DESCRIPTION OF THE HYBRID SYSTEM INTEGRATED WITH THE OPERATIONAL WORKFLOW  In a previous implementation of the system by Naidoo et al (2014), the analysis of the video data to detect potholes was performed off-line and without any operator interaction. The detection accuracy of the full autonomous system was approximately 82%. However, the detections contained many false positives; which are cases where the system classified a detection as a pothole, when in actual fact it was not a pothole. In order to address this in a real operational environment, the system was modified to include a human operator in the loop. The operator would validate the detections made by the hybrid system.   Figure 1 illustrates the operation of the hybrid system. The mobile sensor system is attached to a vehicle and driven through the area that is to be inspected. The mobile system contains a processor, camera and GPS unit. The camera captures a video stream of the environment. The video stream is analyzed in real-time to detect potholes. The details of the detection algorithm are described in Section 3. The detections are presented to an operator in the vehicle via a Graphical User Interface (GUI). The operator uses the GUI to decide whether a detection that is suggested by the system is a valid pothole or not. The validated pothole detections are then compiled into reports containing images and information about the location and estimated size of the potholes. The reports are then posted via 3G to a remote server as a JavaScript Object Notation (JSON) message. The remote server resides at the municipality where the reports that are received via the server are integrated into the existing financial management systems. Work orders are created and handed to the repair crews.   Figure 1 – typical operational workflow of the VSP system 1483 POTHOLE DETECTION ALGORITHM  The key steps in the general image analysis pipeline that was used is illustrated in Figure 2.   Figure 2 – general image analysis pipeline  The frames are extracted from the video stream. The full colour red, green and blue (RGB) input frames are then converted into the hue, saturation and value (HSV) colour space and the independent channels are extracted. The saturation channel is used by the following road detection and feature extraction stages.  The scene that is captured by the camera is a complex scene containing vegetation, sky, road signs, pedestrians and vehicles amongst other things. The first step to perform in the detection of potholes is to segment out the areas of the image that are unlikely to contain potholes. Therefore, the road data from the image is first extracted and the following pothole detection stages are applied to the road only. The road extraction serves two purposes. Firstly, it reduces the space in which the pothole detector has to be applied, which leads to performance improvements. Secondly, it helps in increasing the accuracy of the detector, since the dimensionality of the feature space searched for pothole features is reduced to only that part of the space that is relevant.  A heuristic method of detecting the road was used. The algorithm makes use of an estimation window and colour information. Given the fixed position of the camera, there are areas of the field of view (FOV) that are more likely to contain large areas of road. A fixed trapezoidal area was used as the estimation window within this area. Figure 3(a) below shows an RGB frame that was captured by the VSP sensor during a pilot study. Figure 3(b) shows the fixed trapezoidal window that was used as the estimation window.    (a) Original image (b) Estimation window Figure 3 - (a) original RGB image, (b) trapezoidal estimation window    149After the conversion of the RGB image into the HSV colour space, the mean and standard deviation of the saturation channel are calculated, using the area of the image within the estimation window. The road mask is then generated by selecting areas of the saturation channel of the whole image, whose saturation value is close to the mean that was calculated. In this way a simple Gaussian kernel filter is applied to the saturation channel of the image to obtain a mask to extract the road. Figure 4(a) shows the road mask that was derived, as described above, and Figure 4(b) shows the road segmented from the original RGB image using the road mask.     (a) Road mask (b) Detected road Figure 4 - (a) road mask, (b) road segmentation using the road mask.  The Local Binary Pattern (LBP) feature extractor was applied to the saturation channel of the HSV image. The LBP feature extractor makes use of textural relationships within the image, Ma et al (2008). These features were chosen to make use of the fact that potholes are irregularly defined objects of interest that occur in a uniform background.  The LBP features are then used to train a cascade of weak classifiers, similar to the method proposed by Viola et al (2001). The method involves training a group of statistical models such that they are each able weakly classify the data. The group of classifiers is then combined using the method of boosting to produce a single stronger classifier. The LBP feature extractor was applied to 1773 patches associated with potholes image data and 2133 patches associated with non-pothole image data.  During testing, the video data was first processed manually and then automatically by the algorithm. The manual and automatic detections were then compared. The videos contained 86 separate potholes and 3463 frames in total. Table 1 shows the confusion matrix for the classification algorithm. The true negative values were taken to be those frames that did not contain any false positives. Table 2 shows the performance metrics for the classification algorithm. The values in Table 2 range from 0 to 1. The performance indicates that the automated detection alone had an accuracy of approximately 90% with this limited set of test data. The researchers expect that this value may drop slightly with a much larger test set, but it would still be sufficient for the proposed hybrid “human-in-the-loop” pothole detection system.    150Table 1 – confusion matrix for the algorithms performance  Predicted Positive Negative Actual Positive 62 337 Negative 24 3169  Table 2 – performance metrics for the classification algorithm Accuracy 0.9 Precision 0.16 Recall 0.73 F1 Score 0.26  The autonomous classification results are presented to a human operator who then performs a final validation of the pothole detections. The validated results would also be used to re-train the classification algorithms and the re-training would improve the suggestive capability of the system over time.   4 CONCLUSION  The researchers were able to improve the automated detection accuracy of the system by improving the algorithm design. In order to further improve the accuracy of the overall system, the researchers modified the fully autonomous system to include a “human-in-the-loop” mode to perform the final validation of the pothole detections. Work orders are generated based on the validated potholes. The validation results are also used to re-train the classification algorithms. This means that the suggestive capability of the system would improve over time. The system will undergo an extensive pilot deployment during the next phase of the project.   5 REFERENCES  BUA NEWS (2011) SA's R22bn pothole repair programme. [Online] Available from: http://www.southafrica.info/business/economy/development/potholes-190411.htm#.VVjnvvmqpBc [Accessed: 21 October 2013]  CHANG, K.T., CHANG, J.R. and LIU, J.K., 2005. Detection of pavement distresses using 3D laser scanning technology, Proceedings of the 2005 ASCE International Conference on Computing in Civil Engineering 2005, pp. 1085-1095.  CHEN, K., LU, M., TAN, G. and WU, J., 2014. CRSM: Crowdsourcing based road surface monitoring, Proceedings - 2013 IEEE International Conference on High Performance Computing and Communications, HPCC 2013 and 2013 IEEE International Conference on Embedded and Ubiquitous Computing, EUC 2013 2014, pp. 2151-2158. 151DE ZOYSA, K., KEPPITIYAGAMA, C., SENEVIRATNE, G.P. and SHIHAN, W.W.A.T., 2007. A public transport system based sensor network for road surface condition monitoring, Proceedings of the 2007 Workshop on Networked Systems for Developing Regions, NSDR'07 2007.  ERIKSSON, J., GIROD, L., HULL, B., NEWTON, R., MADDEN, S. and BALAKRISHNAN, H., 2008. The Pothole Patrol: Using a mobile sensor network for road surface monitoring, MobiSys'08 - Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services 2008, pp. 29-39. KOCH, C. and BRILAKIS, I., 2014. Improving pothole recognition through vision tracking for automated pavement assessment, EG-ICE 2011, European Group for Intelligent Computing in Engineering 2014.  KOCH, C. and BRILAKIS, I., 2011. Pothole detection in asphalt pavement images. Advanced Engineering Informatics, 25(3), pp. 507-515.  LI, Q., YAO, M., YAO, X. and XU, B., 2010. A real-time 3D scanning system for pavement distortion inspection. Measurement Science and Technology, 21(1),.  MUSTAFFARA, M., LINGB, T. and PUANB, O., 2008. Automated pavement imaging program (APIP) for pavement cracks classification and quantification-a photogrammetric approach. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 37(B4), pp. 367-372.  NAIDOO, T., JOUBERT, D., CHIWEWE, T., TYATYANTSI, A., RANCATI, B. and MBIZENI, A., 2014. Visual surveying platform for the automated detection of road surface distresses, Proceedings of SPIE - The International Society for Optical Engineering 2014.  RAJAB, M.I., ALAWI, M.H, SAIF, M.A, 2008. Application of image processing to measure road distresses. WSEAS Transactions on Information Science and Applications, vol.5, pp1-7, 2008.  SALARI, E. and BAO, G., 2011. Pavement distress detection and severity analysis, Proceedings of SPIE - The International Society for Optical Engineering 2011. VIOLA, P., and JONES, M., 2001. Rapid object detection using a boosted cascade of simple features, Computer Vision and Pattern Recognition, vol. 1, pp. 511-518, 2001.  MA, C., TAN, T., and YAN, Q.,, 2008. Cascade boosting LBP feature based classifiers for face recognition, in 3rd International Conference on Intelligent System and Knowledge Engineering, Beijing, China, 2008.   152

A hybrid human machine system for the detection and management of potholes on asphalt road surfaces

https://repository.up.ac.za/bitstream/2263/57732/1/Naidoo_Hybrid_2015.pdf

A hybrid human machine system for the detection and management of potholes on asphalt road surfaces

Abstract

Similar works

Full text

Available Versions

UPSpace at the University of Pretoria