Learning Interpretable Models Through Multi-Objective Neural Architecture Search

Monumental advances in deep learning have led to unprecedented achievements across a multitude of domains. While the performance of deep neural networks is indubitable, the architectural design and interpretability of such models are nontrivial. Research has been introduced to automate the design of neural network architectures through neural architecture search (NAS). Recent progress has made these methods more pragmatic by exploiting distributed computation and novel optimization algorithms. However, there is little work in optimizing architectures for interpretability. To this end, we propose a multi-objective distributed NAS framework that optimizes for both task performance and introspection. We leverage the non-dominated sorting genetic algorithm (NSGA-II) and explainable AI (XAI) techniques to reward architectures that can be better comprehended by humans. The framework is evaluated on several image classification datasets. We demonstrate that jointly optimizing for introspection ability and task error leads to more disentangled architectures that perform within tolerable error.Comment: 14 pages main text, 5 pages references, 17 pages supplementa

A scalable method for parallelizing sampling-based motion planning algorithms

Abstract—This paper describes a scalable method for paral-lelizing sampling-based motion planning algorithms. It subdi-vides configuration space (C-space) into (possibly overlapping) regions and independently, in parallel, uses standard (sequen-tial) sampling-based planners to construct roadmaps in each region. Next, in parallel, regional roadmaps in adjacent regions are connected to form a global roadmap. By subdividing the space and restricting the locality of connection attempts, we reduce the work and inter-processor communication associated with nearest neighbor calculation, a critical bottleneck for scalability in existing parallel motion planning methods. We show that our method is general enough to handle a variety of planning schemes, including the widely used Probabilistic Roadmap (PRM) and Rapidly-exploring Random Trees (RRT) algorithms. We compare our approach to two other existing parallel algorithms and demonstrate that our approach achieves better and more scalable performance. Our approach achieves almost linear scalability on a 2400 core LINUX cluster and on a 153,216 core Cray XE6 petascale machine. I

ZeRO++: Extremely Efficient Collective Communication for Giant Model Training

Zero Redundancy Optimizer (ZeRO) has been used to train a wide range of large language models on massive GPUs clusters due to its ease of use, efficiency, and good scalability. However, when training on low-bandwidth clusters, or at scale which forces batch size per GPU to be small, ZeRO's effective throughput is limited because of high communication volume from gathering weights in forward pass, backward pass, and averaging gradients. This paper introduces three communication volume reduction techniques, which we collectively refer to as ZeRO++, targeting each of the communication collectives in ZeRO. First is block-quantization based all-gather. Second is data remapping that trades-off communication for more memory. Third is a novel all-to-all based quantized gradient averaging paradigm as replacement of reduce-scatter collective, which preserves accuracy despite communicating low precision data. Collectively, ZeRO++ reduces communication volume of ZeRO by 4x, enabling up to 2.16x better throughput at 384 GPU scale.Comment: 12 page

Parallelizing Training of Deep Generative Models on Massive Scientific Datasets

Training deep neural networks on large scientific data is a challenging task that requires enormous compute power, especially if no pre-trained models exist to initialize the process. We present a novel tournament method to train traditional as well as generative adversarial networks built on LBANN, a scalable deep learning framework optimized for HPC systems. LBANN combines multiple levels of parallelism and exploits some of the worlds largest supercomputers. We demonstrate our framework by creating a complex predictive model based on multi-variate data from high-energy-density physics containing hundreds of millions of images and hundreds of millions of scalar values derived from tens of millions of simulations of inertial confinement fusion. Our approach combines an HPC workflow and extends LBANN with optimized data ingestion and the new tournament-style training algorithm to produce a scalable neural network architecture using a CORAL-class supercomputer. Experimental results show that 64 trainers (1024 GPUs) achieve a speedup of 70.2 over a single trainer (16 GPUs) baseline, and an effective 109% parallel efficiency

Prospective observational cohort study on grading the severity of postoperative complications in global surgery research

Background The Clavien–Dindo classification is perhaps the most widely used approach for reporting postoperative complications in clinical trials. This system classifies complication severity by the treatment provided. However, it is unclear whether the Clavien–Dindo system can be used internationally in studies across differing healthcare systems in high- (HICs) and low- and middle-income countries (LMICs). Methods This was a secondary analysis of the International Surgical Outcomes Study (ISOS), a prospective observational cohort study of elective surgery in adults. Data collection occurred over a 7-day period. Severity of complications was graded using Clavien–Dindo and the simpler ISOS grading (mild, moderate or severe, based on guided investigator judgement). Severity grading was compared using the intraclass correlation coefficient (ICC). Data are presented as frequencies and ICC values (with 95 per cent c.i.). The analysis was stratified by income status of the country, comparing HICs with LMICs. Results A total of 44 814 patients were recruited from 474 hospitals in 27 countries (19 HICs and 8 LMICs). Some 7508 patients (16·8 per cent) experienced at least one postoperative complication, equivalent to 11 664 complications in total. Using the ISOS classification, 5504 of 11 664 complications (47·2 per cent) were graded as mild, 4244 (36·4 per cent) as moderate and 1916 (16·4 per cent) as severe. Using Clavien–Dindo, 6781 of 11 664 complications (58·1 per cent) were graded as I or II, 1740 (14·9 per cent) as III, 2408 (20·6 per cent) as IV and 735 (6·3 per cent) as V. Agreement between classification systems was poor overall (ICC 0·41, 95 per cent c.i. 0·20 to 0·55), and in LMICs (ICC 0·23, 0·05 to 0·38) and HICs (ICC 0·46, 0·25 to 0·59). Conclusion Caution is recommended when using a treatment approach to grade complications in global surgery studies, as this may introduce bias unintentionally

The surgical safety checklist and patient outcomes after surgery: a prospective observational cohort study, systematic review and meta-analysis

© 2017 British Journal of Anaesthesia Background: The surgical safety checklist is widely used to improve the quality of perioperative care. However, clinicians continue to debate the clinical effectiveness of this tool. Methods: Prospective analysis of data from the International Surgical Outcomes Study (ISOS), an international observational study of elective in-patient surgery, accompanied by a systematic review and meta-analysis of published literature. The exposure was surgical safety checklist use. The primary outcome was in-hospital mortality and the secondary outcome was postoperative complications. In the ISOS cohort, a multivariable multi-level generalized linear model was used to test associations. To further contextualise these findings, we included the results from the ISOS cohort in a meta-analysis. Results are reported as odds ratios (OR) with 95% confidence intervals. Results: We included 44 814 patients from 497 hospitals in 27 countries in the ISOS analysis. There were 40 245 (89.8%) patients exposed to the checklist, whilst 7508 (16.8%) sustained ≥1 postoperative complications and 207 (0.5%) died before hospital discharge. Checklist exposure was associated with reduced mortality [odds ratio (OR) 0.49 (0.32–0.77); P\u3c0.01], but no difference in complication rates [OR 1.02 (0.88–1.19); P=0.75]. In a systematic review, we screened 3732 records and identified 11 eligible studies of 453 292 patients including the ISOS cohort. Checklist exposure was associated with both reduced postoperative mortality [OR 0.75 (0.62–0.92); P\u3c0.01; I2=87%] and reduced complication rates [OR 0.73 (0.61–0.88); P\u3c0.01; I2=89%). Conclusions: Patients exposed to a surgical safety checklist experience better postoperative outcomes, but this could simply reflect wider quality of care in hospitals where checklist use is routine

Critical care admission following elective surgery was not associated with survival benefit:prospective analysis of data from 27 countries

Purpose: As global initiatives increase patient access to surgical treatments, there is a need to define optimal levels of perioperative care. Our aim was to describe the relationship between the provision and use of critical care resources and postoperative mortality. Methods: Planned analysis of data collected during an international 7-day cohort study of adults undergoing elective in-patient surgery. We used risk-adjusted mixed-effects logistic regression models to evaluate the association between admission to critical care immediately after surgery and in-hospital mortality. We evaluated hospital-level associations between mortality and critical care admission immediately after surgery, critical care admission to treat life-threatening complications, and hospital provision of critical care beds. We evaluated the effect of national income using interaction tests. Results: 44,814 patients from 474 hospitals in 27 countries were available for analysis. Death was more frequent amongst patients admitted directly to critical care after surgery (critical care: 103/4317 patients [2%], standard ward: 99/39,566 patients [0.3%]; adjusted OR 3.01 [2.10–5.21]; p < 0.001). This association may differ with national income (high income countries OR 2.50 vs. low and middle income countries OR 4.68; p = 0.07). At hospital level, there was no association between mortality and critical care admission directly after surgery (p = 0.26), critical care admission to treat complications (p = 0.33), or provision of critical care beds (p = 0.70). Findings of the hospital-level analyses were not affected by national income status. A sensitivity analysis including only high-risk patients yielded similar findings. Conclusions: We did not identify any survival benefit from critical care admission following surgery

Effects of hospital facilities on patient outcomes after cancer surgery: an international, prospective, observational study

© 2022 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 licenseBackground: Early death after cancer surgery is higher in low-income and middle-income countries (LMICs) compared with in high-income countries, yet the impact of facility characteristics on early postoperative outcomes is unknown. The aim of this study was to examine the association between hospital infrastructure, resource availability, and processes on early outcomes after cancer surgery worldwide. Methods: A multimethods analysis was performed as part of the GlobalSurg 3 study—a multicentre, international, prospective cohort study of patients who had surgery for breast, colorectal, or gastric cancer. The primary outcomes were 30-day mortality and 30-day major complication rates. Potentially beneficial hospital facilities were identified by variable selection to select those associated with 30-day mortality. Adjusted outcomes were determined using generalised estimating equations to account for patient characteristics and country-income group, with population stratification by hospital. Findings: Between April 1, 2018, and April 23, 2019, facility-level data were collected for 9685 patients across 238 hospitals in 66 countries (91 hospitals in 20 high-income countries; 57 hospitals in 19 upper-middle-income countries; and 90 hospitals in 27 low-income to lower-middle-income countries). The availability of five hospital facilities was inversely associated with mortality: ultrasound, CT scanner, critical care unit, opioid analgesia, and oncologist. After adjustment for case-mix and country income group, hospitals with three or fewer of these facilities (62 hospitals, 1294 patients) had higher mortality compared with those with four or five (adjusted odds ratio [OR] 3·85 [95% CI 2·58–5·75]; p<0·0001), with excess mortality predominantly explained by a limited capacity to rescue following the development of major complications (63·0% vs 82·7%; OR 0·35 [0·23–0·53]; p<0·0001). Across LMICs, improvements in hospital facilities would prevent one to three deaths for every 100 patients undergoing surgery for cancer. Interpretation: Hospitals with higher levels of infrastructure and resources have better outcomes after cancer surgery, independent of country income. Without urgent strengthening of hospital infrastructure and resources, the reductions in cancer-associated mortality associated with improved access will not be realised. Funding: National Institute for Health and Care Research