Application of computer vision for roller operation management

Compaction is the last and possibly the most important phase in construction of asphalt concrete (AC) pavements. Compaction densifies the loose (AC) mat, producing a stable surface with low permeability. The process strongly affects the AC performance properties. Too much compaction may cause aggregate degradation and low air void content facilitating bleeding and rutting. On the other hand too little compaction may result in higher air void content facilitating oxidation and water permeability issues, rutting due to further densification by traffic and reduced fatigue life. Therefore, compaction is a critical issue in AC pavement construction.;The common practice for compacting a mat is to establish a roller pattern that determines the number of passes and coverages needed to achieve the desired density. Once the pattern is established, the roller\u27s operator must maintain the roller pattern uniformly over the entire mat.;Despite the importance of uniform compaction to achieve the expected durability and performance of AC pavements, having the roller operator as the only mean to manage the operation can involve human errors.;With the advancement of technology in recent years, the concept of intelligent compaction (IC) was developed to assist the roller operators and improve the construction quality. Commercial IC packages for construction rollers are available from different manufacturers. They can provide precise mapping of a roller\u27s location and provide the roller operator with feedback during the compaction process.;Although, the IC packages are able to track the roller passes with impressive results, there are also major hindrances. The high cost of acquisition and potential negative impact on productivity has inhibited implementation of IC.;This study applied computer vision technology to build a versatile and affordable system to count and map roller passes. An infrared camera is mounted on top of the roller to capture the operator view. Then, in a near real-time process, image features were extracted and tracked to estimate the incremental rotation and translation of the roller. Image featured are categorized into near and distant features based on the user defined horizon. The optical flow is estimated for near features located in the region below the horizon. The change in roller\u27s heading is constantly estimated from the distant features located in the sky region. Using the roller\u27s rotation angle, the incremental translation between two frames will be calculated from the optical flow. The roller\u27s incremental rotation and translation will put together to develop a tracking map.;During system development, it was noted that in environments with thermal uniformity, the background of the IR images exhibit less featured as compared to images captured with optical cameras which are insensitive to temperature. This issue is more significant overnight, since nature elements are not able to reflect the heat energy from sun. Therefore to improve roller\u27s heading estimation where less features are available in the sky region a unique methodology that allows heading detection based on the asphalt mat edges was developed for this research. The heading measurements based on the slope of the asphalt hot edges will be added to the pool of the headings measured from sky region. The median of all heading measurements will be used as the incremental roller\u27s rotation for the tracking analysis.;The record of tracking data is used for QC/QA purposes and verifying the proper implementation of the roller pattern throughout a job constructed under the roller pass specifications.;The system developed during this research was successful in mapping roller location for few projects tested. However the system should be independently validated

Vision-Based Bridge Deformation Monitoring

This is the final version of the article. Available from Frontiers Media via the DOI in this record.Optics-based tracking of civil structures is not new, due to historical application in surveying, but automated applications capable of tracking at rates that capture dynamic effects are now a hot research topic in structural health monitoring. Recent innovations show promise of true non-contacting monitoring capability avoiding the need for physically attached sensor arrays. The paper reviews recent experience using the Imetrum Dynamic Monitoring Station (DMS) commercial optics-based tracking system on Humber Bridge and Tamar Bridge, aiming to show both the potential and limitations. In particular, the paper focuses on the challenges to field application of such a system resulting from camera instability, nature of the target (artificial or structural feature), and illumination. The paper ends with evaluation of a non-proprietary system using a consumer-grade camera for cable vibration monitoring to emphasize the potential for lower cost systems where if performance specifications can be relaxed.The GPS system at Humber was created by Dr. Ki Koo with support from EPSRC grant EP/F035403/1. DH was supported via the Marie Curie Fellowship programme and as such the research leading to these results has received funding from the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 330195

Update urban basemap by using the LiDAR mobile mapping system : the case of Abu Dhabi municipal system

Basemaps are the main resource used in urban planning and in building and infrastructure asset management. These maps are used by citizens and by private and public stakeholders. Therefore, accurate, up-to-date geoinformation of reference are needed to provide a good service. In general, basemaps have been updated by aerial photogrammetry or field surveying, but these methods are not always possible and alternatives need to be sought. Current limitations and challenges that face traditional field surveys include areas with extreme weather, deserts or artic environments, and flight restrictions due to proximity with other countries if there is not an agreement. In such cases, alternatives for large-scale are required. This thesis proposes the use of a mobile mapping system (MMS) to update urban basemaps. Most urban features can be extracted from point cloud using commercial software or open libraries. However, there are some exceptions: manhole covers, or hidden elements even with captures from defferent perspective, the most common building corners. Therefore, the main objective of this study was to establish a methodology for extracting manholes automatically and for completing hidden corners of buildings, so that urban basemaps can be updated. The algorithm developed to extract manholes is based on time, intensity and shape detection parameters, whereas additional information from satellite images is used to complete buildings. Each municipality knows the materials and dimensions of its manholes. Taking advantage of this knowledge, the point cloud is filtered to classify points according to the set of intensity values associated with the manhole material. From the classified points, the minimum bounding rectangles (MBR) are obtained and finally the shape is adjusted and drawn. We use satellite imagery to automatically digitize the layout of building footprints with automated software tools. Then, the visible corners of buildings from the LiDAR point cloud are imported and a fitting process is performed by comparing them with the corners of the building from the satellite image. Two methods are evaluated to establish which is the most suitable for adjustment in these conditions. In the first method, the differences in X and Y directions are measured in the corners, where LiDAR and satellite data are available, and is often computed as the average of the offsets. In the second method, a Helmert 2D transformation is applied. MMS involves Global Navigation Satellite Systems (GNSS) and Inertial Measurement Units (IMU) to georeference point clouds. Their accuracy depends on the acquisition environment. In this study, the influence of the urban pattern is analysed in three zones with varied urban characteristics: different height buildings, open areas, and areas with a low and high level of urbanization. To evaluate the efficiency of the proposed algorithms, three areas were chosen with varying urban patterns in Abu Dhabi. In these areas, 3D urban elements (light poles, street signs, etc) were automatically extracted using commercial software. The proposed algorithms were applied to the manholes and buildings. The completeness and correctness ratio, and geometric accuracy were calculated for all urban elements in the three areas. The best success rates (>70%) were for light poles, street signs and road curbs, regardless of the height of the buildings. The worst rate was obtained for the same features in peri-urban areas, due to high vegetation. In contrast, the best results for trees were found in theses areas. Our methodology demonstrates the great potential and efficiency of mobile LiDAR technology in updating basemaps; a process that is required to achieve standard accuracy in large scale maps. The cost of the entire process and the time required for the proposed methodology was calculated and compared with the traditional method. It was found that mobile LiDAR could be a standard cost-efficient procedure for updating maps.La cartografía de referencia es la principal herramienta en planificación urbanística, y gestión de infraestructuras y edificios, al servicio de ciudadanos, empresas y administración. Por esta razón, debe estar actualizada y ser lo más precisa posible. Tradicionalmente, la cartografía se actualiza mediante fotogrametría aérea o levantamientos terrestres. No obstante, deben buscarse alternativas válidas para escalas grandes, porque no siempre es posible emplear estas técnicas debido a las limitaciones y retos actuales a los que se enfrenta la medición tradicional en algunas zonas del planeta, con meteorología extrema o restricciones de vuelo por la proximidad a la frontera con otros países. Esta tesis propone el uso del sistema Mobile Mapping System (MMS) para actualizar la cartografía urbana de referencia. La mayoría de los elementos pueden extraerse empleando software comercial o librerías abiertas, excepto los registros de servicios. Los elementos ocultos son otro de los inconvenientes encontrados en el proceso de creación o actualización de la cartografía, incluso si se dispone de capturas desde diferentes puntos de vista. El caso más común es el de las esquinas de edificios. Por ello, el principal objetivo de este estudio es establecer una metodología de extracción automática de los registros y completar las esquinas ocultas de los edificios para actualizar cartografía urbana. El algoritmo desarrollado para la detección y extracción de registros se basa en parámetros como el tiempo, la intensidad de la señal laser y la forma de los registros, mientras que para completar los edificios se emplea información adicional de imágenes satélite. Aprovechando el conocimiento del material y dimensión de los registros, en disposición de los gestores municipales, el algoritmo propuesto filtra y clasifica los puntos de acuerdo a los valores de intensidad. De aquellos clasificados como registros se calcula el mínimo rectángulo que los contiene (Minimum Bounding Rectangle) y finalmente se ajusta la forma y se dibuja. Las imágenes de satélite son empleadas para obtener automáticamente la huella de los edificios. Posteriormente, se importan las esquinas visibles de los edificios obtenidas desde la nube de puntos y se realiza el ajuste comparándolas con las obtenidas desde satélite. Para llevar a cabo este ajuste se han evaluado dos métodos, el primero de ellos considera las diferencias entre las coordenadas XY, desplazándose el promedio. En el segundo, se aplica una transformación Helmert2D. MMS emplea sistemas de navegación global por satélite (Global Navigation Satellite Systems, GNSS) e inerciales (Inertial Measurement Unit, IMU) para georreferenciar la nube de puntos. La precisión de estos sistemas de posicionamiento depende del entorno de adquisición. Por ello, en este estudio se han seleccionado tres áreas con distintas características urbanas (altura de edificios, nivel de urbanización y áreas abiertas) de Abu Dhabi con el fin de analizar su influencia, tanto en la captura, como en la extracción de los elementos. En el caso de farolas, señales viales, árboles y aceras se ha realizado con software comercial, y para registros y edificios con los algoritmos propuestos. Las ratios de corrección y completitud, y la precisión geométrica se han calculado en las diferentes áreas urbanas. Los mejores resultados se han conseguido para las farolas, señales y bordillos, independientemente de la altura de los edificios. La peor ratio se obtuvo para los mismos elementos en áreas peri-urbanas, debido a la vegetación. Resultados opuestos se han conseguido en la detección de árboles. El coste económico y en tiempo de la metodología propuesta resulta inferior al de métodos tradicionales. Lo cual demuestra el gran potencial y eficiencia de la tecnología LiDAR móvil para la actualización cartografía de referenciaPostprint (published version

En Introduksjon til Kunstig Syn i Autonom Kjøring

Autonom kjøring er en av de fremtredende teknologiene i dagens samfunn. Et bredt spekter av applikasjoner bruker derfor denne teknologien for fordelene den gir. For eksempel vil en autonom kjørende robot frigjøre arbeidskraft og øke produktiviteten i bransjer som krever rask transport. For å oppnå disse fordelene krever det imidlertid utvikling av pålitelig og nøyaktig programvare og algoritmer som skal implementeres i disse autonome kjøresytemene. Ettersom dette feltet har vokst gjennom årene, har forskjellige selskaper implementert denne teknologien med stor suksess. Dermed gjør det økte fokuset på autonom kjøre teknologi dette til et aktuelt tema å forske på. Siden utvikling av et autonomt kjøresystem er et krevende tema, fokuserer dette prosjektet kun på hvordan kunstig syn kan brukes i autonome kjøresystemer. Først og fremst utvikles en kunstig syns basert programvare for autonom kjøring. Programvaren er først implementert på et lite forhåndslaget kjøretøy i bok størrelse. Dette systemet brukes deretter til å teste programvarens funksjonalitet. Autonome kjørefunksjoner som fungerer tilfredsstillende på det lille test kjøretøyet blir også testet på et større kjøretøy for å se om programvaren fungerer for andre systemer. Videre er den en utviklede programvaren begrenset til enkelte autonome kjørehandlinger. Dette inkluderer handlinger som å stoppe når en hindring eller et stoppskilt er oppdaget, kjøring på en enkel vei og parkering. Selv om dette bare er noen få autonome kjøre funksjoner, er de grunnleggende operasjoner som kan gjøre det autonome kjøresystemet allerede anvendelig for forskjellige brukstilfeller. Ulike kunstig syn metode for gjenstands deteksjon har blitt implementert for å oppdage ulike typer gjenstander som hindringer og skilt for å bestemme kjøretøyets miljø. Programvaren inkluderer også bruk av en linje deteksjonsmetode for å oppdage vei- og parkerings linjer som brukes til å sentrere og parkere kjøretøyet. Dessuten skapes et fuglebilde av den fysiske verden fra kamera bilder som skal brukes som et miljøkart for å planlegge den mest optimale rute i forskjellige scenarier. Til slutt blir disse implementeringene kombinert for å bygge kjørelogikken til kjøretøyet, noe som gjør det i stand til å utføre kjørehandlingene nevnt i forrige avsnitt. Ved bruk av den utviklede programvaren for kjøreoppgave, deteksjon av hindringer, viste resultatet at selv om de faktiske hindringene ble oppdaget, var det scenarier der blokkader ble oppdaget selv om det ikke var noen. På den annen side var den utviklede funksjonen med å stoppe når et stoppskilt blir oppdaget svært nøyaktig og pålitelig ettersom den utførte som forventet. Når det gjelder de resterende to implementerte handlingene, sentrering og parkering av kjøretøyet, slet systemet med å oppnå et lovende resultat. Til tross for det viste de fysiske valideringstestene uten bruk av kjøretøymodell positive resultater, men med mindre avvik fra ønsket resultat. Samlet sett har programvaren potensial for å bli anvendelig i mer krevende scenarier, men det er behov for videre utvikling for å fikse noen problemområder først.Autonomous driving is one of the rising technology in today's society. Thus, a wide range of applications uses this technology for the benefits it yields. For instance, an autonomous driving robot will free up the labor force and increase productivity in industries that require rapid transportation. However, to gain these benefits, it requires the development of reliable and accurate software and algorithms to be implemented in these autonomous driving systems. As this field has been growing over the years, different companies have implemented this technology with great success. Thus, the increased focus on autonomous driving technology makes this a relevant topic to perform research on. As developing an autonomous driving system is a demanding topic, this project focuses solely on how computer vision can be used in autonomous driving systems. First and foremost, a computer-vision based autonomous driving software is developed. The software is first implemented on a small premade book-size vehicle. This system is then used to test the software's functionality. Autonomous driving functions that perform satisfactorily on the small test vehicle are also tested on a larger vehicle to see if the software works for other systems. Furthermore, the developed software is limited to some autonomous driving actions. This includes actions such as stopping when a hindrance or a stop sign is detected, driving on a simple road, and parking. Although these are only a few autonomous driving actions, they are fundamental operations that can make the autonomous driving system already applicable to different use cases. Different computer vision methods for object detection have been implemented for detecting different types of objects such as hindrances and signs to determine the vehicle's environment. The software also includes the usage of a line detection method for detecting road and parking lines that are used for centering and parking the vehicle. Moreover, a bird-view of the physical world is created from the camera output to be used as an environment map to plan the most optimal path in different scenarios. Finally, these implementations are combined to build the driving logic of the vehicle, making it able to perform the driving actions mentioned in the previous paragraph. When utilizing the developed software for the driving task, hindrance detection, the result showed that although the actual hindrances were detected, there were scenarios where blockades were detected even though there were none. On the other hand, the developed function of stopping when a stop sign is detected was highly accurate and reliable as it performed as expected. With regard to the remaining two implemented actions, centering and parking the vehicle, the system struggled to achieve a promising result. Despite that, the physical validation tests without the use of a vehicle model showed positive outcomes although with minor deviation from the desired result. Overall, the software showed potential to be developed even further to be applicable in more demanding scenarios, however, the current issues must be addressed first

En Introduksjon til Kunstig Syn i Autonom Kjøring

Autonomous driving is one of the rising technology in today’s society. Thus, a wide range of applications uses this technology for the benefits it yields. For instance, an autonomous driving robot will free up the labor force and increase productivity in industries that require rapid transportation. However, to gain these benefits, it requires the development of reliable and accurate software and algorithms to be implemented in these autonomous driving systems. As this field has been growing over the years, different companies have implemented this technology with great success. Thus, the increased focus on autonomous driving technology makes this a relevant topic to perform research on. As developing an autonomous driving system is a demanding topic, this project focuses solely on how computer vision can be used in autonomous driving systems. First and foremost, a computer-vision based autonomous driving software is developed. The software is first imple- mented on a small premade book-size vehicle. This system is then used to test the software’s functionality. Autonomous driving functions that perform satisfactorily on the small test vehicle are also tested on a larger vehicle to see if the software works for other systems. Furthermore, the developed software is limited to some autonomous driving actions. This includes actions such as stopping when a hindrance or a stop sign is detected, driving on a simple road, and parking. Although these are only a few autonomous driving actions, they are fundamental operations that can make the autonomous driving system already applicable to different use cases. Different computer vision methods for object detection have been implemented for detecting different types of objects such as hindrances and signs to determine the vehicle’s environment. The software also includes the usage of a line detection method for detecting road and parking lines that are used for centering and parking the vehicle. Moreover, a bird-view of the physical world is created from the camera output to be used as an environment map to plan the most optimal path in different scenarios. Finally, these implementations are combined to build the driving logic of the vehicle, making it able to perform the driving actions mentioned in the previous paragraph. When utilizing the developed software for the driving task, hindrance detection, the result showed that although the actual hindrances were detected, there were scenarios where block- ades were detected even though there were none. On the other hand, the developed function of stopping when a stop sign is detected was highly accurate and reliable as it performed as expected. With regard to the remaining two implemented actions, centering and parking the vehicle, the system struggled to achieve a promising result. Despite that, the physical validation tests without the use of a vehicle model showed positive outcomes although with minor deviation from the desired result. Overall, the software showed potential to be developed even further to be applicable in more demanding scenarios, however, the current issues must be addressed first

En Introduksjon til Kunstig Syn i Autonom Kjøring

Autonom kjøring er en av de fremtredende teknologiene i dagens samfunn. Et bredt spekter av applikasjoner bruker derfor denne teknologien for fordelene den gir. For eksempel vil en autonom kjørende robot frigjøre arbeidskraft og øke produktiviteten i bransjer som krever rask transport. For å oppnå disse fordelene krever det imidlertid utvikling av pålitelig og nøyaktig programvare og algoritmer som skal implementeres i disse autonome kjøresytemene. Ettersom dette feltet har vokst gjennom årene, har forskjellige selskaper implementert denne teknologien med stor suksess. Dermed gjør det økte fokuset på autonom kjøre teknologi dette til et aktuelt tema å forske på. Siden utvikling av et autonomt kjøresystem er et krevende tema, fokuserer dette prosjektet kun på hvordan kunstig syn kan brukes i autonome kjøresystemer. Først og fremst utvikles en kunstig syns basert programvare for autonom kjøring. Programvaren er først implementert på et lite forhåndslaget kjøretøy i bok størrelse. Dette systemet brukes deretter til å teste programvarens funksjonalitet. Autonome kjørefunksjoner som fungerer tilfredsstillende på det lille test kjøretøyet blir også testet på et større kjøretøy for å se om programvaren fungerer for andre systemer. Videre er den en utviklede programvaren begrenset til enkelte autonome kjørehandlinger. Dette inkluderer handlinger som å stoppe når en hindring eller et stoppskilt er oppdaget, kjøring på en enkel vei og parkering. Selv om dette bare er noen få autonome kjøre funksjoner, er de grunnleggende operasjoner som kan gjøre det autonome kjøresystemet allerede anvendelig for forskjellige brukstilfeller. Ulike kunstig syn metode for gjenstands deteksjon har blitt implementert for å oppdage ulike typer gjenstander som hindringer og skilt for å bestemme kjøretøyets miljø. Programvaren inkluderer også bruk av en linje deteksjonsmetode for å oppdage vei- og parkerings linjer som brukes til å sentrere og parkere kjøretøyet. Dessuten skapes et fuglebilde av den fysiske verden fra kamera bilder som skal brukes som et miljøkart for å planlegge den mest optimale rute i forskjellige scenarier. Til slutt blir disse implementeringene kombinert for å bygge kjørelogikken til kjøretøyet, noe som gjør det i stand til å utføre kjørehandlingene nevnt i forrige avsnitt. Ved bruk av den utviklede programvaren for kjøreoppgave, deteksjon av hindringer, viste resultatet at selv om de faktiske hindringene ble oppdaget, var det scenarier der blokkader ble oppdaget selv om det ikke var noen. På den annen side var den utviklede funksjonen med å stoppe når et stoppskilt blir oppdaget svært nøyaktig og pålitelig ettersom den utførte som forventet. Når det gjelder de resterende to implementerte handlingene, sentrering og parkering av kjøretøyet, slet systemet med å oppnå et lovende resultat. Til tross for det viste de fysiske valideringstestene uten bruk av kjøretøymodell positive resultater, men med mindre avvik fra ønsket resultat. Samlet sett har programvaren potensial for å bli anvendelig i mer krevende scenarier, men det er behov for videre utvikling for å fikse noen problemområder først.Autonomous driving is one of the rising technology in today’s society. Thus, a wide range of applications uses this technology for the benefits it yields. For instance, an autonomous driving robot will free up the labor force and increase productivity in industries that require rapid transportation. However, to gain these benefits, it requires the development of reliable and accurate software and algorithms to be implemented in these autonomous driving systems. As this field has been growing over the years, different companies have implemented this technology with great success. Thus, the increased focus on autonomous driving technology makes this a relevant topic to perform research on. As developing an autonomous driving system is a demanding topic, this project focuses solely on how computer vision can be used in autonomous driving systems. First and foremost, a computer-vision based autonomous driving software is developed. The software is first imple- mented on a small premade book-size vehicle. This system is then used to test the software’s functionality. Autonomous driving functions that perform satisfactorily on the small test vehicle are also tested on a larger vehicle to see if the software works for other systems. Furthermore, the developed software is limited to some autonomous driving actions. This includes actions such as stopping when a hindrance or a stop sign is detected, driving on a simple road, and parking. Although these are only a few autonomous driving actions, they are fundamental operations that can make the autonomous driving system already applicable to different use cases. Different computer vision methods for object detection have been implemented for detecting different types of objects such as hindrances and signs to determine the vehicle’s environment. The software also includes the usage of a line detection method for detecting road and parking lines that are used for centering and parking the vehicle. Moreover, a bird-view of the physical world is created from the camera output to be used as an environment map to plan the most optimal path in different scenarios. Finally, these implementations are combined to build the driving logic of the vehicle, making it able to perform the driving actions mentioned in the previous paragraph. When utilizing the developed software for the driving task, hindrance detection, the result showed that although the actual hindrances were detected, there were scenarios where block- ades were detected even though there were none. On the other hand, the developed function of stopping when a stop sign is detected was highly accurate and reliable as it performed as expected. With regard to the remaining two implemented actions, centering and parking the vehicle, the system struggled to achieve a promising result. Despite that, the physical validation tests without the use of a vehicle model showed positive outcomes although with minor deviation from the desired result. Overall, the software showed potential to be developed even further to be applicable in more demanding scenarios, however, the current issues must be addressed first

Intelligent Computational Transportation

Transportation is commonplace around our world. Numerous researchers dedicate great efforts to vast transportation research topics. The purpose of this dissertation is to investigate and address a couple of transportation problems with respect to geographic discretization, pavement surface automatic examination, and traffic ow simulation, using advanced computational technologies. Many applications require a discretized 2D geographic map such that local information can be accessed efficiently. For example, map matching, which aligns a sequence of observed positions to a real-world road network, needs to find all the nearby road segments to the individual positions. To this end, the map is discretized by cells and each cell retains a list of road segments coincident with this cell. An efficient method is proposed to form such lists for the cells without costly overlapping tests. Furthermore, the method can be easily extended to 3D scenarios for fast triangle mesh voxelization. Pavement surface distress conditions are critical inputs for quantifying roadway infrastructure serviceability. Existing computer-aided automatic examination techniques are mainly based on 2D image analysis or 3D georeferenced data set. The disadvantage of information losses or extremely high costs impedes their effectiveness iv and applicability. In this study, a cost-effective Kinect-based approach is proposed for 3D pavement surface reconstruction and cracking recognition. Various cracking measurements such as alligator cracking, traverse cracking, longitudinal cracking, etc., are identified and recognized for their severity examinations based on associated geometrical features. Smart transportation is one of the core components in modern urbanization processes. Under this context, the Connected Autonomous Vehicle (CAV) system presents a promising solution towards the enhanced traffic safety and mobility through state-of-the-art wireless communications and autonomous driving techniques. Due to the different nature between the CAVs and the conventional Human- Driven-Vehicles (HDVs), it is believed that CAV-enabled transportation systems will revolutionize the existing understanding of network-wide traffic operations and re-establish traffic ow theory. This study presents a new continuum dynamics model for the future CAV-enabled traffic system, realized by encapsulating mutually-coupled vehicle interactions using virtual internal and external forces. A Smoothed Particle Hydrodynamics (SPH)-based numerical simulation and an interactive traffic visualization framework are also developed