Lane Detection System based on Hough Transform with Retinex Algorithm

Nowadays, automotive system becomes a great innovation in the world and lane detection system is important to control automobile vehicles. This paper has developed an efficient lane detection system to deal with different types of lighting conditions. Six types of edge detection techniques: canny, sobel, prewitt, Roberts, Laplacian of Gaussian (LOG) and zero-cross methods are analyzed. Line detection based on canny operator is developed. Moreover, Retinex algorithm is employed to normalize input images for all types of illumination. And Hough Transform with Retinex algorithm is developed to solve lighting problem. The proposed method is compared to Hough Transform with Otsu’s threshold method. The experimental results show that the proposed method can reduce computation time and improve accuracy for lane detection system

LaneMapper: A City-scale Lane Map Generator for Autonomous Driving

Autonomous vehicles require lane maps to help navigate from a start to a goal position in a safe, comfortable and quick manner. A lane map represents a set of features inherent to the road, such as lanes, stop signs, traffic lights, and intersections. We present a novel approach to detect multiple lane boundaries and traffic signs to create a 3D city-scale map of the driving environment. We detect, recognize and track lane boundaries with multimodal sensory and prior inputs, such as camera, LiDAR, and GPS/IMU, to assist autonomous driving. We detect and classify traffic signs from the image considering high reflectivity of LiDAR points and further register the locations of traffic signs and lane boundaries together in the world coordinate frame. We have also made our code base open-source for the research community to tweak or use our algorithm for their purposes

LaneMapper: A City-scale Lane Map Generator for Autonomous Driving

Autonomous vehicles require lane maps to help navigate from a start to a goal position in a safe, comfortable and quick manner. A lane map represents a set of features inherent to the road, such as lanes, stop signs, traffic lights, and intersections. We present a novel approach to detect multiple lane boundaries and traffic signs to create a 3D city-scale map of the driving environment. We detect, recognize and track lane boundaries with multimodal sensory and prior inputs, such as camera, LiDAR, and GPS/IMU, to assist autonomous driving. We detect and classify traffic signs from the image considering high reflectivity of LiDAR points and further register the locations of traffic signs and lane boundaries together in the world coordinate frame. We have also made our code base open-source for the research community to tweak or use our algorithm for their purposes

Computer Vision System-On-Chip Designs for Intelligent Vehicles

Intelligent vehicle technologies are growing rapidly that can enhance road safety, improve transport efficiency, and aid driver operations through sensors and intelligence. Advanced driver assistance system (ADAS) is a common platform of intelligent vehicle technologies. Many sensors like LiDAR, radar, cameras have been deployed on intelligent vehicles. Among these sensors, optical cameras are most widely used due to their low costs and easy installation. However, most computer vision algorithms are complicated and computationally slow, making them difficult to be deployed on power constraint systems. This dissertation investigates several mainstream ADAS applications, and proposes corresponding efficient digital circuits implementations for these applications. This dissertation presents three ways of software / hardware algorithm division for three ADAS applications: lane detection, traffic sign classification, and traffic light detection. Using FPGA to offload critical parts of the algorithm, the entire computer vision system is able to run in real time while maintaining a low power consumption and a high detection rate. Catching up with the advent of deep learning in the field of computer vision, we also present two deep learning based hardware implementations on application specific integrated circuits (ASIC) to achieve even lower power consumption and higher accuracy. The real time lane detection system is implemented on Xilinx Zynq platform, which has a dual core ARM processor and FPGA fabric. The Xilinx Zynq platform integrates the software programmability of an ARM processor with the hardware programmability of an FPGA. For the lane detection task, the FPGA handles the majority of the task: region-of-interest extraction, edge detection, image binarization, and hough transform. After then, the ARM processor takes in hough transform results and highlights lanes using the hough peaks algorithm. The entire system is able to process 1080P video stream at a constant speed of 69.4 frames per second, realizing real time capability. An efficient system-on-chip (SOC) design which classifies up to 48 traffic signs in real time is presented in this dissertation. The traditional histogram of oriented gradients (HoG) and support vector machine (SVM) are proven to be very effective on traffic sign classification with an average accuracy rate of 93.77%. For traffic sign classification, the biggest challenge comes from the low execution efficiency of the HoG on embedded processors. By dividing the HoG algorithm into three fully pipelined stages, as well as leveraging extra on-chip memory to store intermediate results, we successfully achieved a throughput of 115.7 frames per second at 1080P resolution. The proposed generic HoG hardware implementation could also be used as an individual IP core by other computer vision systems. A real time traffic signal detection system is implemented to present an efficient hardware implementation of the traditional grass-fire blob detection. The traditional grass-fire blob detection method iterates the input image multiple times to calculate connected blobs. In digital circuits, five extra on-chip block memories are utilized to save intermediate results. By using additional memories, all connected blob information could be obtained through one-pass image traverse. The proposed hardware friendly blob detection can run at 72.4 frames per second with 1080P video input. Applying HoG + SVM as feature extractor and classifier, 92.11% recall rate and 99.29% precision rate are obtained on red lights, and 94.44% recall rate and 98.27% precision rate on green lights. Nowadays, convolutional neural network (CNN) is revolutionizing computer vision due to learnable layer by layer feature extraction. However, when coming into inference, CNNs are usually slow to train and slow to execute. In this dissertation, we studied the implementation of principal component analysis based network (PCANet), which strikes a balance between algorithm robustness and computational complexity. Compared to a regular CNN, the PCANet only needs one iteration training, and typically at most has a few tens convolutions on a single layer. Compared to hand-crafted features extraction methods, the PCANet algorithm well reflects the variance in the training dataset and can better adapt to difficult conditions. The PCANet algorithm achieves accuracy rates of 96.8% and 93.1% on road marking detection and traffic light detection, respectively. Implementing in Synopsys 32nm process technology, the proposed chip can classify 724,743 32-by-32 image candidates in one second, with only 0.5 watt power consumption. In this dissertation, binary neural network (BNN) is adopted as a potential detector for intelligent vehicles. The BNN constrains all activations and weights to be +1 or -1. Compared to a CNN with the same network configuration, the BNN achieves 50 times better resource usage with only 1% - 2% accuracy loss. Taking car detection and pedestrian detection as examples, the BNN achieves an average accuracy rate of over 95%. Furthermore, a BNN accelerator implemented in Synopsys 32nm process technology is presented in our work. The elastic architecture of the BNN accelerator makes it able to process any number of convolutional layers with high throughput. The BNN accelerator only consumes 0.6 watt and doesn\u27t rely on external memory for storage

Vehicular Instrumentation and Data Processing for the Study of Driver Intent

The primary goal of this thesis is to provide processed experimental data needed to determine whether driver intentionality and driving-related actions can be predicted from quantitative and qualitative analysis of driver behaviour. Towards this end, an instrumented experimental vehicle capable of recording several synchronized streams of data from the surroundings of the vehicle, the driver gaze with head pose and the vehicle state in a naturalistic driving environment was designed and developed. Several driving data sequences in both urban and rural environments were recorded with the instrumented vehicle. These sequences were automatically annotated for relevant artifacts such as lanes, vehicles and safely driveable areas within road lanes. A framework and associated algorithms required for cross-calibrating the gaze tracking system with the world coordinate system mounted on the outdoor stereo system was also designed and implemented, allowing the mapping of the driver gaze with the surrounding environment. This instrumentation is currently being used for the study of driver intent, geared towards the development of driver maneuver prediction models

Підвищення ефективності телевізійної системи керування автомобіля

Дисертаційна робота присвячена підвищенню ефективності телевізійної системи керування автомобілем. Актуальність теми. В даний час комп'ютеризація в нашому суспільстві розвивається швидкими темпами і відіграє величезну роль в житті людини. За допомогою комп'ютерних технологій автоматизується широке коло процесів, які в недалекому минулому покладалися на людину. А з використанням оптико-електронних пристроїв вдається вирішувати задачі, які неможливо вирішити іншими шляхами. Рішення проблеми розпізнавання і обробки зображень і, як наслідок, запобігання аварійної ситуації на дорозі є досить важливим аспектом безпеки і контролю дорожньої ситуації. Системи активної безпеки дозволяють коригувати помилкові дії водія, уникати екстрених ситуацій, підвищити безпеку дорожнього руху. «Електронні асистенти», в основному, використовують ультразвукові датчики. Вони дозволяють тільки визначити наявність перешкоду і відстань до нього. А введення оптичних сенсорів дає цілий спектр корисної інформації, від наявності перешкоди до його розмірів, відстані, траєкторії руху. Актуальною, затребуваною і цікавою темою цього проекту є аналіз і розробка власного універсального приладу «консультування» водія (ADAS) в питанні активної безпеки. Мета і задачі дослідження. Метою дисертаційної роботи є досягнення максимального узгодження оптичної системи телевізійних систем керування автомобілем, покращення роздільної здатності і якості отримуваного зображення за рахунок вибору параметрів об’єктива, підвищення ефективності та адаптація під всі види транспортних засобів. Для досягнення поставленої мети в дисертаційній роботі поставлені наступні задачі: 1. Обґрунтувати вимоги до характеристик телевізійних систем керування автомобілем. 2. Досягти критерію узгодження просторової роздільної здатності об’єктива і ПЗЗ приймача для телевізійних оптичних систем та впливу кута нахилу системи відносно дороги на кінцеве зображення. 3. Вдосконалити математичну модель обробки вихідного сигналу методами DSP обробки. 4. Розробити стартап-проект оптико-електронного детектора дорожньої розмітки. Об’єкт дослідження. Процес реєстрації оптичного випромінювання приймачем випромінювання та його подальша обробка. Предмет дослідження. Телевізійні системи керування автомобілем. Наукова новизна. Вперше розроблена швидка методика детектування та локалізації дорожньої розмітки за допомогою алгоритмів згортки та кореляції сумісно із атермальним об’єктивом. Розроблено програмне забезпечення для автоматичної обробки зображень для локалізації розмітки. Проведено аналіз розрахованих методик на реальних даних. Методи дослідження. У дисертаційній роботі для розв’язання даних поставлених задач використовується: 1. Аналітичні методи, засновані на застосуванні апарату геометричної оптики, теорії аберацій оптичних систем, проектування та контролю телевізійних систем, алгоритми розпізнавання дорожньої розмітки. 2. Використання програмного пакета «MatLab» та «Python» для моделювання різноманітних дорожніх ситуацій та розробки алгоритму розпізнавання розмітки, досягнення вдосконалення критеріїв. 3. Комп’ютерне моделювання і оптичний розрахунок різних телевізійних оптичних систем в програмі «PODIL» з абераційним і енергетичним аналізом якості отримуваного зображення. 4. Використання різних методик оцінки роздільної здатності та якості отримуваного зображення телевізійних систем з ПЗЗ приймачем. Магістерська дисертація складається з чотирьох розділів. У першому розділі проаналізовано основні переваги і недоліки існуючих систем керування автомобілем та методів цифрової обробки зображень. У другому розділі приведено загальну фізико-математичну модель телевізійної системи керування автомобілем та розроблено атермальний об’єктив. Третій розділ присвячено огляду, аналізу та вибору методу цифрової обробки зображень для задачі детекції дорожньої розмітки. Четвертий розділ присвячено розробці стартап-проекту «Телевізійна система керування автомобілем» і аналізу перспектив входження розробки на ринок з маркетологічної точки зору.The dissertation is devoted to increase of efficiency of the television control system of the car. Actuality of theme. Nowadays, computerization in our society is developing at a rapid pace and plays a huge role in human life. Computer technology automates a wide range of processes that have relied on humans in the recent past. And with the use of optoelectronic devices it is possible to solve problems that cannot be solved in other ways. Addressing image recognition and processing and, as a consequence, preventing an accident on the road is a very important aspect of road safety and control. Active safety systems allow you to correct driver misconduct, avoid emergencies, and improve road safety. "Electronic assistants" mainly use ultrasonic sensors. They only allow you to determine the presence of an obstacle and the distance to it. And the introduction of optical sensors gives a whole spectrum of useful information, from the presence of obstacles to its size, distance, trajectory. A topical, sought-after and interesting topic of this project is the analysis and development of our own Active Driver Advice (ADAS) device in active safety. The purpose and objectives of the study. The aim of the dissertation is to achieve the maximum harmonization of the optical system of television control systems of the car, to improve the resolution and quality of the image obtained by selecting lens parameters, improving efficiency and adaptation to all types of vehicles. In order to achieve this goal in the dissertation the following tasks are set: 1. To substantiate the requirements for the characteristics of television control systems of the car. 2. Achieve the criterion of spatial resolution of the lens and the CCD of the television optical systems and the effect of the inclination of the system relative to the road on the final image. 3. Improve mathematical model of output signal processing by DSP processing methods. 4. To develop a startup project of opto-electronic road marking detector. Object of study. The process of registration of optical radiation by the radiation receiver and its subsequent processing. Subject of study. Car TV systems. Research methods. In the dissertation the following tasks are used to solve the data of the given tasks: 1. Analytical methods based on the application of geometric optics apparatus, theory of aberrations of optical systems, design and control of television systems, algorithms for recognition of road marking. 2. Using MatLab and Python software to simulate various road situations and to develop a markup recognition algorithm to achieve criteria improvement. 3. Computer simulation and optical calculation of various television optical systems in the PODIL program with aberration and energy analysis of the image quality. 4. The use of different methods of estimating the resolution and quality of the resulting image of television systems with CCD receiver. The master's thesis consists of four sections. The first section analyzes the main advantages and disadvantages of existing car control systems and digital imaging techniques. The second section presents a general physics and mathematics model of a car television system and a thermal lens. The third section deals with the review, analysis and selection of the digital image processing method for the task of detecting road marking. The fourth section is devoted to the development of a startup project "TV Car Control System" and to analyze the prospects for market entry from a marketing point of view

Vision-based ego-lane analysis system : dataset and algorithms

A detecção e análise da faixa de trânsito são tarefas importantes e desafiadoras em sistemas avançados de assistência ao motorista e direção autônoma. Essas tarefas são necessárias para auxiliar veículos autônomos e semi-autônomos a operarem com segurança. A queda no custo dos sensores de visão e os avanços em hardware embarcado impulsionaram as pesquisas relacionadas a faixa de trânsito –detecção, estimativa, rastreamento, etc. – nas últimas duas décadas. O interesse nesse tópico aumentou ainda mais com a demanda por sistemas avançados de assistência ao motorista (ADAS) e carros autônomos. Embora amplamente estudado de forma independente, ainda há necessidade de estudos que propõem uma solução combinada para os vários problemas relacionados a faixa do veículo, tal como aviso de saída de faixa (LDW), detecção de troca de faixa, classificação do tipo de linhas de divisão de fluxo (LMT), detecção e classificação de inscrições no pavimento, e detecção da presença de faixas ajdacentes. Esse trabalho propõe um sistema de análise da faixa do veículo (ELAS) em tempo real capaz de estimar a posição da faixa do veículo, classificar as linhas de divisão de fluxo e inscrições na faixa, realizar aviso de saída de faixa e detectar eventos de troca de faixa. O sistema proposto, baseado em visão, funciona em uma sequência temporal de imagens. Características das marcações de faixa são extraídas tanto na perspectiva original quanto em images mapeadas para a vista aérea, que então são combinadas para aumentar a robustez. A estimativa final da faixa é modelada como uma spline usando uma combinação de métodos (linhas de Hough, filtro de Kalman e filtro de partículas). Baseado na faixa estimada, todos os outros eventos são detectados. Além disso, o sistema proposto foi integrado para experimentação em um sistema para carros autônomos que está sendo desenvolvido pelo Laboratório de Computação de Alto Desempenho (LCAD) da Universidade Federal do Espírito Santo (UFES). Para validar os algorítmos propostos e cobrir a falta de base de dados para essas tarefas na literatura, uma nova base dados com mais de 20 cenas diferentes (com mais de 15.000 imagens) e considerando uma variedade de cenários (estrada urbana, rodovias, tráfego, sombras, etc.) foi criada. Essa base de dados foi manualmente anotada e disponilizada publicamente para possibilitar a avaliação de diversos eventos que são de interesse para a comunidade de pesquisa (i.e. estimativa, mudança e centralização da faixa; inscrições no pavimento; cruzamentos; tipos de linhas de divisão de fluxo; faixas de pedestre e faixas adjacentes). Além disso, o sistema também foi validado qualitativamente com base na integração com o veículo autônomo. O sistema alcançou altas taxas de detecção em todos os eventos do mundo real e provou estar pronto para aplicações em tempo real.Lane detection and analysis are important and challenging tasks in advanced driver assistance systems and autonomous driving. These tasks are required in order to help autonomous and semi-autonomous vehicles to operate safely. Decreasing costs of vision sensors and advances in embedded hardware boosted lane related research – detection, estimation, tracking, etc. – in the past two decades. The interest in this topic has increased even more with the demand for advanced driver assistance systems (ADAS) and self-driving cars. Although extensively studied independently, there is still need for studies that propose a combined solution for the multiple problems related to the ego-lane, such as lane departure warning (LDW), lane change detection, lane marking type (LMT) classification, road markings detection and classification, and detection of adjacent lanes presence. This work proposes a real-time Ego-Lane Analysis System (ELAS) capable of estimating ego-lane position, classifying LMTs and road markings, performing LDW and detecting lane change events. The proposed vision-based system works on a temporal sequence of images. Lane marking features are extracted in perspective and Inverse Perspective Mapping (IPM) images that are combined to increase robustness. The final estimated lane is modeled as a spline using a combination of methods (Hough lines, Kalman filter and Particle filter). Based on the estimated lane, all other events are detected. Moreover, the proposed system was integrated for experimentation into an autonomous car that is being developed by the High Performance Computing Laboratory of the Universidade Federal do Espírito Santo. To validate the proposed algorithms and cover the lack of lane datasets in the literature, a new dataset with more than 20 different scenes (in more than 15,000 frames) and considering a variety of scenarios (urban road, highways, traffic, shadows, etc.) was created. The dataset was manually annotated and made publicly available to enable evaluation of several events that are of interest for the research community (i.e. lane estimation, change, and centering; road markings; intersections; LMTs; crosswalks and adjacent lanes). Furthermore, the system was also validated qualitatively based on the integration with the autonomous vehicle. ELAS achieved high detection rates in all real-world events and proved to be ready for real-time applications.FAPE

Advances in Automated Driving Systems

Electrification, automation of vehicle control, digitalization and new mobility are the mega-trends in automotive engineering, and they are strongly connected. While many demonstrations for highly automated vehicles have been made worldwide, many challenges remain in bringing automated vehicles to the market for private and commercial use. The main challenges are as follows: reliable machine perception; accepted standards for vehicle-type approval and homologation; verification and validation of the functional safety, especially at SAE level 3+ systems; legal and ethical implications; acceptance of vehicle automation by occupants and society; interaction between automated and human-controlled vehicles in mixed traffic; human–machine interaction and usability; manipulation, misuse and cyber-security; the system costs of hard- and software and development efforts. This Special Issue was prepared in the years 2021 and 2022 and includes 15 papers with original research related to recent advances in the aforementioned challenges. The topics of this Special Issue cover: Machine perception for SAE L3+ driving automation; Trajectory planning and decision-making in complex traffic situations; X-by-Wire system components; Verification and validation of SAE L3+ systems; Misuse, manipulation and cybersecurity; Human–machine interactions, driver monitoring and driver-intention recognition; Road infrastructure measures for the introduction of SAE L3+ systems; Solutions for interactions between human- and machine-controlled vehicles in mixed traffic

Driver lane change intention inference using machine learning methods.

Lane changing manoeuvre on highway is a highly interactive task for human drivers. The intelligent vehicles and the advanced driver assistance systems (ADAS) need to have proper awareness of the traffic context as well as the driver. The ADAS also need to understand the driver potential intent correctly since it shares the control authority with the human driver. This study provides a research on the driver intention inference, particular focus on the lane change manoeuvre on highways. This report is organised in a paper basis, where each chapter corresponding to a publication, which is submitted or to be submitted. Part Ⅰ introduce the motivation and general methodology framework for this thesis. Part Ⅱ includes the literature survey and the state-of-art of driver intention inference. Part Ⅲ contains the techniques for traffic context perception that focus on the lane detection. A literature review on lane detection techniques and its integration with parallel driving framework is proposed. Next, a novel integrated lane detection system is designed. Part Ⅳ contains two parts, which provides the driver behaviour monitoring system for normal driving and secondary tasks detection. The first part is based on the conventional feature selection methods while the second part introduces an end-to-end deep learning framework. The design and analysis of driver lane change intention inference system for the lane change manoeuvre is proposed in Part Ⅴ. Finally, discussions and conclusions are made in Part Ⅵ. A major contribution of this project is to propose novel algorithms which accurately model the driver intention inference process. Lane change intention will be recognised based on machine learning (ML) methods due to its good reasoning and generalizing characteristics. Sensors in the vehicle are used to capture context traffic information, vehicle dynamics, and driver behaviours information. Machine learning and image processing are the techniques to recognise human driver behaviour.PhD in Transpor