Explainable and Advisable Learning for Self-driving Vehicles

Deep neural perception and control networks are likely to be a key component of self-driving vehicles. These models need to be explainable - they should provide easy-to-interpret rationales for their behavior - so that passengers, insurance companies, law enforcement, developers, etc., can understand what triggered a particular behavior. Explanations may be triggered by the neural controller, namely introspective explanations, or informed by the neural controller's output, namely rationalizations. Our work has focused on the challenge of generating introspective explanations of deep models for self-driving vehicles. In Chapter 3, we begin by exploring the use of visual explanations. These explanations take the form of real-time highlighted regions of an image that causally influence the network's output (steering control). In the first stage, we use a visual attention model to train a convolution network end-to-end from images to steering angle. The attention model highlights image regions that potentially influence the network's output. Some of these are true influences, but some are spurious. We then apply a causal filtering step to determine which input regions actually influence the output. This produces more succinct visual explanations and more accurately exposes the network's behavior. In Chapter 4, we add an attention-based video-to-text model to produce textual explanations of model actions, e.g. "the car slows down because the road is wet". The attention maps of controller and explanation model are aligned so that explanations are grounded in the parts of the scene that mattered to the controller. We explore two approaches to attention alignment, strong- and weak-alignment. These explainable systems represent an externalization of tacit knowledge. The network's opaque reasoning is simplified to a situation-specific dependence on a visible object in the image. This makes them brittle and potentially unsafe in situations that do not match training data. In Chapter 5, we propose to address this issue by augmenting training data with natural language advice from a human. Advice includes guidance about what to do and where to attend. We present the first step toward advice-giving, where we train an end-to-end vehicle controller that accepts advice. The controller adapts the way it attends to the scene (visual attention) and the control (steering and speed). Further, in Chapter 6, we propose a new approach that learns vehicle control with the help of long-term (global) human advice. Specifically, our system learns to summarize its visual observations in natural language, predict an appropriate action response (e.g. "I see a pedestrian crossing, so I stop"), and predict the controls, accordingly

System Design and Analysis for Creating a 3D Virtual Street Scene for Autonomous Vehicles using Geometric Proxies from a Single Video Camera

Self-driving vehicles use a variety of sensors to understand the environment they are in. In order to do so, they must accurately measure the distances and positions of the objects around them. A common representation of the environment around the vehicle is a 3D point cloud, or a set of 3D data points which represent the positions of objects in the real world relative to the car. However, while accurate and useful, these point clouds require large amounts of memory compared to other representations such as lightweight polygonal meshes. In addition, 3D point clouds can be difficult for a human to visually understand as the data points do not always form a naturally coherent object. This paper introduces a system to lower the memory consumption needed for the graphical representation of a virtual street environment. At this time, the proposed system takes in as input a single front-facing video. The system uses the video to retrieve still images of a scene which are then segmented to distinguish the different relevant objects, such as cars and stop signs. The system generates a corresponding virtual street scene and these key objects are visualized in the virtual world as low poly, or low resolution, models of the respective objects. This virtual 3D street environment is created to allow a remote operator to visualize the world that the car is traveling through. At this time, the virtual street includes geometric proxies for parallel parked cars in the form of lightweight polygonal meshes. These meshes are predefined, taking up less memory than a point cloud, which can be costly to transmit from the remote vehicle and potentially difficult for a remote human operator to understand. This paper contributes a design and analysis of an initial system for generating and placing these geometric proxies of parked cars in a virtual street environment from one input video. We discuss the limitations and measure the error for this system as well as reflect on future improvements

Determination of parking space and its concurrent usage over time using semantically segmented mobile mapping data

Public space is a scarce good in cities. There are many concurrent usages, which makes an adequate allocation of space both difficult and highly attractive. A lot of space is allocated by parking cars - even if the parking spaces are not occupied by cars all the time. In this work, we analyze space demand and usage by parking cars, in order to evaluate, when this space could be used for other purposes. The analysis is based on 3D point clouds acquired at several times during a day. We propose a processing pipeline to extract car bounding boxes from a given 3D point cloud. For the car extraction we utilize a label transfer technique for transfers from semantically segmented 2D RGB images to 3D point cloud data. This semantically segmented 3D data allows us to identify car instances. Subsequently, we aggregate and analyze information about parking cars. We present an exemplary analysis of the urban area where we extracted 15.000 cars at five different points in time. Based on this aggregated we present analytical results for time dependent parking behavior, parking space availability and utilization

Dawn of autonomous vehicles: review and challenges ahead

This paper reviews the state of the art on autonomous vehicles as of 2017, including their impact at socio-economic, energy, safety, congestion and land-use levels. This impact study focuses on the issues that are common denominators and are bound to arise independently of regional factors, such as (but not restricted to) change to vehicle ownership patterns and driver behaviour, opportunities for energy and emissions savings, potential for accident reduction and lower insurance costs, and requalification of urban areas previously assigned to parking. The challenges that lie ahead for carmakers, law and policy makers are also explored, with an emphasis on how these challenges affect the urban infrastructure and issues they create for municipal planners and decision makers. The paper concludes with strengths, weaknesses, opportunities, and threats analysis that integrates and relates all these aspects.info:eu-repo/semantics/publishedVersio

Software architectural design for safety in Automated Parking System

The automotive industry has seen a revolution brought about by self-driving cars. However, one of the main challenges facing autonomous driving systems is ensuring safety in the absence of a supervising driver and verifying safe vehicle behaviour under various circumstances. Autonomous Driving Systems (ADS), due to their complexity, cannot be solved straightforwardly without proper structure. Thus, they need a well-defined architecture to guide their development with requirements that involve modularity, scalability, and maintainability among other properties. To help overcome some of the challenges, this master thesis defines and implements in a simulated environment an automated parking system that complies with industrial and safety standards. The work has been divided into four parts. Firstly, the safety rules for the development of an autonomous function have been analysed. Secondly, the use cases and system requirements have been defined following the needs of the automated parking system. Thirdly, the system has been implemented in the simulation environment with a structure based on a widely adopted automotive standard. The final result is the software architecture of an autonomous vehicle with automated parking functionality. This concept has been validated within the virtual environment together with the integration of the AUTOSAR runtime environment, which the communication between components and mode switching functionality in the CARLA simulation environment. The result of this project shows the benefit of integrating architecture and simulation, thus easing the development and testing of future autonomous systems

Improving Parking Availability Maps using Information from Nearby Roads

Parking search traffic causes increased travel times and air pollution in many cities. Real-time parking availability maps are expected to help drivers to find a parking space faster and thus to reduce parking search traffic. A possibility to create such maps is the aggregation of parking availability information from crowdsourcing solutions like probe vehicles and mobile phone applications. Since these sources cannot sense the whole city at the same time, estimation methods are necessary to fill uncovered areas. This paper investigates the estimation of parking availability based on spatial methods using sensor data from San Francisco. First, spatial similarities in parking availability are evaluated for different aspects like time of day and number of parking spaces depending on the distance to reveal the parking characteristics. Then, interpolation methods are examined to estimate parking availability in unobserved road segments. Results show that relevant similarities mainly exist for short distances of less than hundred meters. Their similarity values are lower than the temporal similarity even for multiple hours of time gap. Nevertheless, spatial information is useful to interpolate parking availability. Investigated interpolation methods show significantly better results than random guess. Inverse distance weighting method outperforms a simple averaging by up to 5%.DFG/GRK/193

ATG-PVD: Ticketing Parking Violations on A Drone

In this paper, we introduce a novel suspect-and-investigate framework, which can be easily embedded in a drone for automated parking violation detection (PVD). Our proposed framework consists of: 1) SwiftFlow, an efficient and accurate convolutional neural network (CNN) for unsupervised optical flow estimation; 2) Flow-RCNN, a flow-guided CNN for car detection and classification; and 3) an illegally parked car (IPC) candidate investigation module developed based on visual SLAM. The proposed framework was successfully embedded in a drone from ATG Robotics. The experimental results demonstrate that, firstly, our proposed SwiftFlow outperforms all other state-of-the-art unsupervised optical flow estimation approaches in terms of both speed and accuracy; secondly, IPC candidates can be effectively and efficiently detected by our proposed Flow-RCNN, with a better performance than our baseline network, Faster-RCNN; finally, the actual IPCs can be successfully verified by our investigation module after drone re-localization.Comment: 17 pages, 11 figures and 3 tables. This paper is accepted by ECCV Workshops 202

Evaluating TCP Performance of Routing Protocols for Traffic Exchange in Street-parked Vehicles based Fog Computing Infrastructure

As most vehicles remain parked 95% of its time, this suggests that leveraging the use of On-board Units (OBUs) in parked vehicles would provide communication and computation services to other mobile and fixed nodes for de- livery of services such as multimedia streaming, data storage and data processing. The nearby vehicles can form an infrastructure using IEEE 802.11p communication interface, facilitating communication, computation and storage services to the end users. We refer to this as a Vehicular Fog Computing (VFC) infrastructure. In this study, using NS-2 simulator, we investigate how six routing protocols consisting of two proactive routing protocols, Destination Sequence Destination Vector (DSDV) and Fisheye State Routing (FSR); two reactive routing protocols, Ad Hoc On-Demand Distance Vector (AODV) and Dynamic Source Routing (DSR); and two geographic routing protocols, Distance Routing Effect Algorithm for Mobility (DREAM) and Location Aided Routing (LAR) perform when forwarding TCP traffic among the parked vehicles that form a VFC infrastructure in an urban street parking scenario. In order to reflect an urban street parking scenario, we consider a traffic mobility traces that are generated using SUMO in our simulation. To the best of our knowledge, this work is the first effort to understand how vehicle density, vehicle speed and parking duration can influence TCP in an urban street parking scenario when packet forwarding decision is made using proactive, reactive and geographic routing protocols. In our performance evaluation, positive results are observed on the influence of parking duration in parked vehicles as TCP performance in all routing protocols increases with longer parking duration. However, variable speed in parked vehicles and moving vehicles in an urban street parking scenario may not have significant influence on TCP performance, especially in case of reactive and proactive routing protocols. Further, our findings reveal that vehicle density in a VFC infrastructure can noticeably influence TCP performance. Towards the end of the paper, we delineate some important future research issues in order to improve routing performance in a street-parked vehicle based VFC infrastructure