Object-level fusion for surround environment perception in automated driving applications

Driver assistance systems have increasingly relied on more sensors for new functions. As advanced driver assistance system continue to improve towards automated driving, new methods are required for processing the data in an efficient and economical manner from the sensors for such complex systems. The detection of dynamic objects is one of the most important aspects required by advanced driver assistance systems and automated driving. In this thesis, an environment model approach for the detection of dynamic objects is presented in order to realize an effective method for sensor data fusion. A scalable high-level fusion architecture is developed for fusing object data from several sensors in a single system, where processing occurs in three levels: sensor, fusion and application. A complete and consistent object model which includes the object’s dynamic state, existence probability and classification is defined as a sensor-independent and generic interface for sensor data fusion across all three processing levels. Novel algorithms are developed for object data association and fusion at the fusion-level of the architecture. An asynchronous sensor-to-global fusion strategy is applied in order to process sensor data immediately within the high-level fusion architecture, giving driver assistance systems the most up-to-date information about the vehicle’s environment. Track-to-track fusion algorithms are uniquely applied for dynamic state fusion, where the information matrix fusion algorithm produces results comparable to a low-level central Kalman filter approach. The existence probability of an object is fused using a novel approach based on the Dempster-Shafer evidence theory, where the individual sensor’s existence estimation performance is considered during the fusion process. A similar novel approach with the Dempster-Shafer evidence theory is also applied to the fusion of an object’s classification. The developed high-level sensor data fusion architecture and its algorithms are evaluated using a prototype vehicle equipped with 12 sensors for surround environment perception. A thorough evaluation of the complete object model is performed on a closed test track using vehicles equipped with hardware for generating an accurate ground truth. Existence and classification performance is evaluated using labeled data sets from real traffic scenarios. The evaluation demonstrates the accuracy and effectiveness of the proposed sensor data fusion approach. The work presented in this thesis has additionally been extensively used in several research projects as the dynamic object detection platform for automated driving applications on highways in real traffic

Robust Vehicle Detection and Distance Estimation Under Challenging Lighting Conditions

Avoiding high computational costs and calibration issues involved in stereo-vision-based algorithms, this paper proposes real-time monocular-vision-based techniques for simultaneous vehicle detection and inter-vehicle distance estimation, in which the performance and robustness of the system remain competitive, even for highly challenging benchmark datasets. This paper develops a collision warning system by detecting vehicles ahead and, by identifying safety distances to assist a distracted driver, prior to occurrence of an imminent crash. We introduce adaptive global Haar-like features for vehicle detection, tail-light segmentation, virtual symmetry detection, intervehicle distance estimation, as well as an efficient single-sensor multifeature fusion technique to enhance the accuracy and robustness of our algorithm. The proposed algorithm is able to detect vehicles ahead at both day or night and also for short- and long-range distances. Experimental results under various weather and lighting conditions (including sunny, rainy, foggy, or snowy) show that the proposed algorithm outperforms state-of-the-art algorithms

Temporospatial Context-Aware Vehicular Crash Risk Prediction

With the demand for more vehicles increasing, road safety is becoming a growing concern. Traffic collisions take many lives and cost billions of dollars in losses. This explains the growing interest of governments, academic institutions and companies in road safety. The vastness and availability of road accident data has provided new opportunities for gaining a better understanding of accident risk factors and for developing more effective accident prediction and prevention regimes. Much of the empirical research on road safety and accident analysis utilizes statistical models which capture limited aspects of crashes. On the other hand, data mining has recently gained interest as a reliable approach for investigating road-accident data and for providing predictive insights. While some risk factors contribute more frequently in the occurrence of a road accident, the importance of driver behavior, temporospatial factors, and real-time traffic dynamics have been underestimated. This study proposes a framework for predicting crash risk based on historical accident data. The proposed framework incorporates machine learning and data analytics techniques to identify driving patterns and other risk factors associated with potential vehicle crashes. These techniques include clustering, association rule mining, information fusion, and Bayesian networks. Swarm intelligence based association rule mining is employed to uncover the underlying relationships and dependencies in collision databases. Data segmentation methods are employed to eliminate the effect of dependent variables. Extracted rules can be used along with real-time mobility to predict crashes and their severity in real-time. The national collision database of Canada (NCDB) is used in this research to generate association rules with crash risk oriented subsequents, and to compare the performance of the swarm intelligence based approach with that of other association rule miners. Many industry-demanding datasets, including road-accident datasets, are deficient in descriptive factors. This is a significant barrier for uncovering meaningful risk factor relationships. To resolve this issue, this study proposes a knwoledgebase approximation framework to enhance the crash risk analysis by integrating pieces of evidence discovered from disparate datasets capturing different aspects of mobility. Dempster-Shafer theory is utilized as a key element of this knowledgebase approximation. This method can integrate association rules with acceptable accuracy under certain circumstances that are discussed in this thesis. The proposed framework is tested on the lymphography dataset and the road-accident database of the Great Britain. The derived insights are then used as the basis for constructing a Bayesian network that can estimate crash likelihood and risk levels so as to warn drivers and prevent accidents in real-time. This Bayesian network approach offers a way to implement a naturalistic driving analysis process for predicting traffic collision risk based on the findings from the data-driven model. A traffic incident detection and localization method is also proposed as a component of the risk analysis model. Detecting and localizing traffic incidents enables timely response to accidents and facilitates effective and efficient traffic flow management. The results obtained from the experimental work conducted on this component is indicative of the capability of our Dempster-Shafer data-fusion-based incident detection method in overcoming the challenges arising from erroneous and noisy sensor readings

Holistic Temporal Situation Interpretation for Traffic Participant Prediction

For a profound understanding of traffic situations including a prediction of traf- fic participants’ future motion, behaviors and routes it is crucial to incorporate all available environmental observations. The presence of sensor noise and depen- dency uncertainties, the variety of available sensor data, the complexity of large traffic scenes and the large number of different estimation tasks with diverging requirements require a general method that gives a robust foundation for the de- velopment of estimation applications. In this work, a general description language, called Object-Oriented Factor Graph Modeling Language (OOFGML), is proposed, that unifies formulation of esti- mation tasks from the application-oriented problem description via the choice of variable and probability distribution representation through to the inference method definition in implementation. The different language properties are dis- cussed theoretically using abstract examples. The derivation of explicit application examples is shown for the automated driv- ing domain. A domain-specific ontology is defined which forms the basis for four exemplary applications covering the broad spectrum of estimation tasks in this domain: Basic temporal filtering, ego vehicle localization using advanced interpretations of perceived objects, road layout perception utilizing inter-object dependencies and finally highly integrated route, behavior and motion estima- tion to predict traffic participant’s future actions. All applications are evaluated as proof of concept and provide an example of how their class of estimation tasks can be represented using the proposed language. The language serves as a com- mon basis and opens a new field for further research towards holistic solutions for automated driving

Obstacle Detection Based on Fusion Between Stereovision and 2D Laser Scanner

International audienceObstacle detection is an essential task for mobile robots. This subject has been investigated for many years by researchers and a lot of obstacle detection systems have been proposed so far. Yet designing an accurate and totally robust and reliable system remains a challenging task, above all in outdoor environments. Thus, the purpose of this chapter is to present new techniques and tools to design an accurate, robust and reliable obstacle detection system in outdoor environments based on a minimal number of sensors. So far, experiments and assessments of already developed systems show that using a single sensor is not enough to meet the requirements: at least two complementary sensors are needed. In this chapter a stereovision sensor and a 2D laser scanner are considered

Context Exploitation in Data Fusion

Complex and dynamic environments constitute a challenge for existing tracking algorithms. For this reason, modern solutions are trying to utilize any available information which could help to constrain, improve or explain the measurements. So called Context Information (CI) is understood as information that surrounds an element of interest, whose knowledge may help understanding the (estimated) situation and also in reacting to that situation. However, context discovery and exploitation are still largely unexplored research topics. Until now, the context has been extensively exploited as a parameter in system and measurement models which led to the development of numerous approaches for the linear or non-linear constrained estimation and target tracking. More specifically, the spatial or static context is the most common source of the ambient information, i.e. features, utilized for recursive enhancement of the state variables either in the prediction or the measurement update of the filters. In the case of multiple model estimators, context can not only be related to the state but also to a certain mode of the filter. Common practice for multiple model scenarios is to represent states and context as a joint distribution of Gaussian mixtures. These approaches are commonly referred as the join tracking and classification. Alternatively, the usefulness of context was also demonstrated in aiding the measurement data association. Process of formulating a hypothesis, which assigns a particular measurement to the track, is traditionally governed by the empirical knowledge of the noise characteristics of sensors and operating environment, i.e. probability of detection, false alarm, clutter noise, which can be further enhanced by conditioning on context. We believe that interactions between the environment and the object could be classified into actions, activities and intents, and formed into structured graphs with contextual links translated into arcs. By learning the environment model we will be able to make prediction on the target\u2019s future actions based on its past observation. Probability of target future action could be utilized in the fusion process to adjust tracker confidence on measurements. By incorporating contextual knowledge of the environment, in the form of a likelihood function, in the filter measurement update step, we have been able to reduce uncertainties of the tracking solution and improve the consistency of the track. The promising results demonstrate that the fusion of CI brings a significant performance improvement in comparison to the regular tracking approaches

Environnement logiciel pour l assistance à l autonomie à domicile (gestion de la dynamique et de l incertitude pour la fourniture sémantique en temps réel de services d assistance)

L hétérogénéité des environnements ainsi que la diversité des profils et des besoins des patients représentent des contraintes majeures qui remettent en question l utilisation à grande échelle des systèmes d assistance à l autonomie à domicile (AAL). En effet, afin de répondre à l évolution de l état des patients et de leurs besoins humains, les environnements AAL sont en évolution continue par l introduction ou la disparition de capteurs, de dispositifs d interaction et de services d assistance. Par conséquent, une plateforme générique et dynamique capable de s adapter à différents environnements et d intégrer de nouveaux capteurs, dispositifs d interaction et services d assistance est requise. La mise en œuvre d un tel aspect dynamique peut produire une situation d incertitude dérivée des problèmes techniques liés à la fiabilité des capteurs ou à des problèmes de réseau. Par conséquent, la notion d incertitude doit être introduite dans la représentation de contexte et la prise de décision afin de faire face à ce problème. Au cours de cette thèse, j ai développé une plateforme dynamique et extensible capable de s adapter à différents environnements et aux besoins des patients. Ceci a été réalisé sur la base de l approche Plug&Play sémantique que j ai proposé. Afin de traiter le problème d incertitude de l information lié à des problèmes techniques, j ai proposé une approche de mesure d incertitude en utilisant les caractéristiques intrinsèques des capteurs et leurs comportements fonctionnels. J ai aussi fourni un modèle de représentation sémantique et de raisonnement avec incertitude associé avec la théorie de Dempster-Shafer (DST) pour la prise de décisionThe heterogeneity of the environments as well as the diversity of patients needs and profiles are major constraints that challenge the spread of ambient assistive living (AAL) systems. AAL environments are usually evolving by the introduction or the disappearance of sensors, devices and assistive services to respond to the evolution of patients conditions and human needs. Therefore, a generic framework that is able to adapt to such dynamic environments and to integrate new sensors, devices and assistive services at runtime is required. Implementing such a dynamic aspect may produce an uncertainty derived from technical problems related to sensors reliability or network problems. Therefore, a notion of uncertain should be introduced in context representation and decision making in order to deal with this problem. During this thesis, I have developed a dynamic and extendible framework able to adapt to different environments and patients needs. This was achieved based on my proposed approach of semantic Plug&Play mechanism. In order to handle the problem of uncertain information related to technical problems, I have proposed an approach for uncertainty measurement based on intrinsic characteristics of the sensors and their functional behaviors, then I have provided a model of semantic representation and reasoning under uncertainty coupled with the Dempster-Shafer Theory of evidence (DST) for decision makingEVRY-INT (912282302) / SudocSudocFranceF

Exploitation des données cartographiques pour la perception de véhicules intelligents

This thesis is situated in the domains of robotics and data fusion, and concerns geographic information systems. We study the utility of adding digital maps, which model the urban environment in which the vehicle evolves, as a virtual sensor improving the perception results. Indeed, the maps contain a phenomenal quantity of information about the environment : its geometry, topology and additional contextual information. In this work, we extract road surface geometry and building models in order to deduce the context and the characteristics of each detected object. Our method is based on an extension of occupancy grids : the evidential perception grids. It permits to model explicitly the uncertainty related to the map and sensor data. By this means, the approach presents also the advantage of representing homogeneously the data originating from various sources : lidar, camera or maps. The maps are handled on equal terms with the physical sensors. This approach allows us to add geographic information without imputing unduly importance to it, which is essential in presence of errors. In our approach, the information fusion result, stored in a perception grid, is used to predict the stateof environment on the next instant. The fact of estimating the characteristics of dynamic elements does not satisfy the hypothesis of static world. Therefore, it is necessary to adjust the level of certainty attributed to these pieces of information. We do so by applying the temporal discounting. Due to the fact that existing methods are not well suited for this application, we propose a family of discoun toperators that take into account the type of handled information. The studied algorithms have been validated through tests on real data. We have thus developed the prototypes in Matlab and the C++ software based on Pacpus framework. Thanks to them, we present the results of experiments performed in real conditions.La plupart des logiciels contrôlant les véhicules intelligents traite de la compréhension de la scène. De nombreuses méthodes existent actuellement pour percevoir les obstacles de façon automatique. La majorité d’entre elles emploie ainsi les capteurs extéroceptifs comme des caméras ou des lidars. Cette thèse porte sur les domaines de la robotique et de la fusion d’information et s’intéresse aux systèmes d’information géographique. Nous étudions ainsi l’utilité d’ajouter des cartes numériques, qui cartographient le milieu urbain dans lequel évolue le véhicule, en tant que capteur virtuel améliorant les résultats de perception. Les cartes contiennent en effet une quantité phénoménale d’information sur l’environnement : sa géométrie, sa topologie ainsi que d’autres informations contextuelles. Dans nos travaux, nous avons extrait la géométrie des routes et des modèles de bâtiments afin de déduire le contexte et les caractéristiques de chaque objet détecté. Notre méthode se base sur une extension de grilles d’occupations : les grilles de perception crédibilistes. Elle permet de modéliser explicitement les incertitudes liées aux données de cartes et de capteurs. Elle présente également l’avantage de représenter de façon uniforme les données provenant de différentes sources : lidar, caméra ou cartes. Les cartes sont traitées de la même façon que les capteurs physiques. Cette démarche permet d’ajouter les informations géographiques sans pour autant leur donner trop d’importance, ce qui est essentiel en présence d’erreurs. Dans notre approche, le résultat de la fusion d’information contenu dans une grille de perception est utilisé pour prédire l’état de l’environnement à l’instant suivant. Le fait d’estimer les caractéristiques des éléments dynamiques ne satisfait donc plus l’hypothèse du monde statique. Par conséquent, il est nécessaire d’ajuster le niveau de certitude attribué à ces informations. Nous y parvenons en appliquant l’affaiblissement temporel. Étant donné que les méthodes existantes n’étaient pas adaptées à cette application, nous proposons une famille d’opérateurs d’affaiblissement prenant en compte le type d’information traitée. Les algorithmes étudiés ont été validés par des tests sur des données réelles. Nous avons donc développé des prototypes en Matlab et des logiciels en C++ basés sur la plate-forme Pacpus. Grâce à eux nous présentons les résultats des expériences effectués en conditions réelles

Technologies for safe and resilient earthmoving operations: A systematic literature review

Resilience engineering relates to the ability of a system to anticipate, prepare, and respond to predicted and unpredicted disruptions. It necessitates the use of monitoring and object detection technologies to ensure system safety in excavation systems. Given the increased investment and speed of improvement in technologies, it is necessary to review the types of technology available and how they contribute to excavation system safety. A systematic literature review was conducted which identified and classified the existing monitoring and object detection technologies, and introduced essential enablers for reliable and effective monitoring and object detection systems including: 1) the application of multisensory and data fusion approaches, and 2) system-level application of technologies. This study also identified the developed functionalities for accident anticipation, prevention and response to safety hazards during excavation, as well as those that facilitate learning in the system. The existing research gaps and future direction of research have been discussed