Environment Modeling, Action Classification, and Control for Urban Automated Driving

This thesis discusses the design and implementation of WATonomous' Automated Driving Stack (ADS), which is capable of performing robo-taxi services in specific operational domains when deployed to WATonomous' research vehicle (Bolty). Three ADS modules are discussed in detail: (1) mapping, environment modeling, and behavioral planning, (2) action classification in video streams, and (3) trajectory planning and control. Additionally, the software architecture within which the ADS is developed and deployed, and the ADS data pipeline itself, are outlined. The thesis begins with preliminaries on WATonomous' Dockerized software architecture (coined watod) which runs and orchestrates the communication of the ADS modules. The watod ecosystem, due to its Dockerized and cloud-based design, enables rapid prototyping of new software modules, rapid onboarding of new team members, and parallel execution of many ADS development instances on the WATonomous server cluster's Virtual Machines (VMs). Cloud-based CARLA simulation development of the ADS and deployment to the Bolty research vehicle are also encapsulated in and facilitated by the watod ecosystem. The ADS can be developed in simulation and deployed to the physical research vehicle without modifications to the ADS modules due to the replication of the physical platform in the Carla ROS Bridge sensor configuration. The design of the ADS data pipeline is also presented, from raw sensor input to the Controlled Area Network Bus (CAN Bus) interface, as well as the human-computer interface. The first ADS module discussed is the mapping and environment modeling module. Environment modeling is the backbone of how autonomous agents understand the world, and therefore has significant implications for decision-making and verification. Motivated by the success of relational mapping tools such as Lanelet2, we present the Dynamic Relation Graph (DRG). The DRG is a novel method for extending prior relational maps to include online observations, creating a unified environment model which incorporates both prior and online data sources. Our prototype implementation models a finite set of heterogeneous features including road signage and pedestrian movement. However, the methodology behind the DRG can be expanded to a wider range of features in a fashion that does not increase the complexity of behavioral planning. Simulated stress tests indicate the DRG's effectiveness in decreasing decision-making complexity, and deployment to the WATonomous research vehicle demonstrates its practical utility. The prototype code is available at https://github.com/WATonomous/DRG. The second ADS module discussed is the action classification module. When applied in the context of Autonomous Vehicles (AVs), action classification algorithms can help enrich an AV's environment model and understanding of the world to improve behavioral planning decisions. Towards these improvements in AV decision-making, we propose a novel online action recognition system, coined the Road Action Detection Network (RAD-Net). RAD-Net formulates the problem of active agent detection and adapts ideas about actor-context relations from human activity recognition in a straightforward two-stage pipeline for action detection and classification. We show that our proposed scheme can outperform the baseline on the ICCV 2021 Road Challenge dataset. Furthermore, by integrating RAD-Net with the ADS' perception stack and the DRG, we demonstrate how a higher-order understanding of agent actions in the environment can improve decisions on a real AV system. The last ADS module discussed is trajectory planning and control. Trajectory planning and control have historically been separated into two modules in automated driving stacks. Trajectory planning focuses on higher-level tasks like avoiding obstacles and staying on the road surface, whereas the controller tries its best to follow an ever changing reference trajectory. We argue that this separation is (1) flawed due to the mismatch between planned trajectories and what the controller can feasibly execute, and (2) unnecessary due to the flexibility of the Model Predictive Control (MPC) paradigm. Instead, in this thesis, we present a unified MPC-based trajectory planning and control scheme that guarantees feasibility with respect to road boundaries, the static and dynamic environment, and enforces passenger comfort constraints. The scheme is evaluated rigorously in a variety of scenarios focused on proving the effectiveness of the OCP design and real-time solution methods. The prototype code is available at github.com/WATonomous/control

Real-Time Unified Trajectory Planning and Optimal Control for Urban Autonomous Driving Under Static and Dynamic Obstacle Constraints

Trajectory planning and control have historically been separated into two modules in automated driving stacks. Trajectory planning focuses on higher-level tasks like avoiding obstacles and staying on the road surface, whereas the controller tries its best to follow an ever changing reference trajectory. We argue that this separation is (1) flawed due to the mismatch between planned trajectories and what the controller can feasibly execute, and (2) unnecessary due to the flexibility of the model predictive control (MPC) paradigm. Instead, in this paper, we present a unified MPC-based trajectory planning and control scheme that guarantees feasibility with respect to road boundaries, the static and dynamic environment, and enforces passenger comfort constraints. The scheme is evaluated rigorously in a variety of scenarios focused on proving the effectiveness of the optimal control problem (OCP) design and real-time solution methods. The prototype code will be released at https://github.com/WATonomous/control

RALACs: Action Recognition in Autonomous Vehicles using Interaction Encoding and Optical Flow

When applied to autonomous vehicle (AV) settings, action recognition can enhance an environment model's situational awareness. This is especially prevalent in scenarios where traditional geometric descriptions and heuristics in AVs are insufficient. However, action recognition has traditionally been studied for humans, and its limited adaptability to noisy, un-clipped, un-pampered, raw RGB data has limited its application in other fields. To push for the advancement and adoption of action recognition into AVs, this work proposes a novel two-stage action recognition system, termed RALACs. RALACs formulates the problem of action recognition for road scenes, and bridges the gap between it and the established field of human action recognition. This work shows how attention layers can be useful for encoding the relations across agents, and stresses how such a scheme can be class-agnostic. Furthermore, to address the dynamic nature of agents on the road, RALACs constructs a novel approach to adapting Region of Interest (ROI) Alignment to agent tracks for downstream action classification. Finally, our scheme also considers the problem of active agent detection, and utilizes a novel application of fusing optical flow maps to discern relevant agents in a road scene. We show that our proposed scheme can outperform the baseline on the ICCV2021 Road Challenge dataset and by deploying it on a real vehicle platform, we provide preliminary insight to the usefulness of action recognition in decision making

Rehabilitation versus surgical reconstruction for non-acute anterior cruciate ligament injury (ACL SNNAP): a pragmatic randomised controlled trial

BackgroundAnterior cruciate ligament (ACL) rupture is a common debilitating injury that can cause instability of the knee. We aimed to investigate the best management strategy between reconstructive surgery and non-surgical treatment for patients with a non-acute ACL injury and persistent symptoms of instability.MethodsWe did a pragmatic, multicentre, superiority, randomised controlled trial in 29 secondary care National Health Service orthopaedic units in the UK. Patients with symptomatic knee problems (instability) consistent with an ACL injury were eligible. We excluded patients with meniscal pathology with characteristics that indicate immediate surgery. Patients were randomly assigned (1:1) by computer to either surgery (reconstruction) or rehabilitation (physiotherapy but with subsequent reconstruction permitted if instability persisted after treatment), stratified by site and baseline Knee Injury and Osteoarthritis Outcome Score—4 domain version (KOOS4). This management design represented normal practice. The primary outcome was KOOS4 at 18 months after randomisation. The principal analyses were intention-to-treat based, with KOOS4 results analysed using linear regression. This trial is registered with ISRCTN, ISRCTN10110685, and ClinicalTrials.gov, NCT02980367.FindingsBetween Feb 1, 2017, and April 12, 2020, we recruited 316 patients. 156 (49%) participants were randomly assigned to the surgical reconstruction group and 160 (51%) to the rehabilitation group. Mean KOOS4 at 18 months was 73·0 (SD 18·3) in the surgical group and 64·6 (21·6) in the rehabilitation group. The adjusted mean difference was 7·9 (95% CI 2·5–13·2; p=0·0053) in favour of surgical management. 65 (41%) of 160 patients allocated to rehabilitation underwent subsequent surgery according to protocol within 18 months. 43 (28%) of 156 patients allocated to surgery did not receive their allocated treatment. We found no differences between groups in the proportion of intervention-related complications.InterpretationSurgical reconstruction as a management strategy for patients with non-acute ACL injury with persistent symptoms of instability was clinically superior and more cost-effective in comparison with rehabilitation management

Effect of angiotensin-converting enzyme inhibitor and angiotensin receptor blocker initiation on organ support-free days in patients hospitalized with COVID-19

IMPORTANCE Overactivation of the renin-angiotensin system (RAS) may contribute to poor clinical outcomes in patients with COVID-19. Objective To determine whether angiotensin-converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) initiation improves outcomes in patients hospitalized for COVID-19. DESIGN, SETTING, AND PARTICIPANTS In an ongoing, adaptive platform randomized clinical trial, 721 critically ill and 58 non–critically ill hospitalized adults were randomized to receive an RAS inhibitor or control between March 16, 2021, and February 25, 2022, at 69 sites in 7 countries (final follow-up on June 1, 2022). INTERVENTIONS Patients were randomized to receive open-label initiation of an ACE inhibitor (n = 257), ARB (n = 248), ARB in combination with DMX-200 (a chemokine receptor-2 inhibitor; n = 10), or no RAS inhibitor (control; n = 264) for up to 10 days. MAIN OUTCOMES AND MEASURES The primary outcome was organ support–free days, a composite of hospital survival and days alive without cardiovascular or respiratory organ support through 21 days. The primary analysis was a bayesian cumulative logistic model. Odds ratios (ORs) greater than 1 represent improved outcomes. RESULTS On February 25, 2022, enrollment was discontinued due to safety concerns. Among 679 critically ill patients with available primary outcome data, the median age was 56 years and 239 participants (35.2%) were women. Median (IQR) organ support–free days among critically ill patients was 10 (–1 to 16) in the ACE inhibitor group (n = 231), 8 (–1 to 17) in the ARB group (n = 217), and 12 (0 to 17) in the control group (n = 231) (median adjusted odds ratios of 0.77 [95% bayesian credible interval, 0.58-1.06] for improvement for ACE inhibitor and 0.76 [95% credible interval, 0.56-1.05] for ARB compared with control). The posterior probabilities that ACE inhibitors and ARBs worsened organ support–free days compared with control were 94.9% and 95.4%, respectively. Hospital survival occurred in 166 of 231 critically ill participants (71.9%) in the ACE inhibitor group, 152 of 217 (70.0%) in the ARB group, and 182 of 231 (78.8%) in the control group (posterior probabilities that ACE inhibitor and ARB worsened hospital survival compared with control were 95.3% and 98.1%, respectively). CONCLUSIONS AND RELEVANCE In this trial, among critically ill adults with COVID-19, initiation of an ACE inhibitor or ARB did not improve, and likely worsened, clinical outcomes. TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT0273570

Effects of scorpion (Buthus-tamulus) venom on neuromuscular transmission in vitro

The effects of venom from the Indian red scorpion Buthus tamulus (BT) on neuromuscular transmission have been investigated by means of twitch tension and electrophysical recording techniques using isolated skeletal muscle preparations. On chick biventer cervicis preparations, BT (1-3 μg/ml) augmented the twitch responses to indirect, but not direct, muscle stimulation. Higher concentrations caused a transient augmentation followed by a large contracture and then a reduction in twitch height. BT at the concentrations tested caused little change in postjunctional sensitivity as assessed by responses to exogenous acetylcholine, carbachol and KCl. Tubocurarine abolished the prolonged contracture induced by BT (10 μg/ml) in the presence or absence of nerve stimulation. On mouse hemidiaphragm preparations, BT (3-10 μg/ml) increased the twitch responses to indirect stimulation but caused little change in directly stimulated preparations. On mouse triangularis sterni preparations, BT (3-10 μg/ml) increased quantal content of the evoked end-plate potentials (epps) by about 70%, without markedly affecting the time course and amplitude of miniature epps. BT also caused repetitive epps in response to single shock nerve stimulation. Extracellular recording of nerve terminal current waveforms in triangularis sterni preparations revealed that BT (10-30 μg/ml) slightly reduced the amplitude of the waveform. Subsequently, BT induced repetitive firing of nerve endings in response to single shock stimulation, and eventually markedly prolonged the time course of the nerve terminal waveform. The effects caused by BT were different from those caused by iberiotoxin, the blocker of Ca2+-activated K+ currents, isolated from BT. The effects were similar to those caused by ATX-II, a toxin that delays inactivation of Na+ channels. However, BT and ATX-II behaved differently in the presence of K+ channel blockers, 3,4-diaminopyridine (DAP) and tetraethylammonium (TEA). These results confirm that Buthus tamulus venom acts mainly prejunctionally to increase the release of acetylcholine. The effect of BT on the perineural waveforms suggests that some of its actions may be due to effects on Na+ channels at or near the nerve terminals; however, additional effects on K+ channels are likely