Evaluating Uncertainty Estimation Methods on 3D Semantic Segmentation of Point Clouds

Deep learning models are extensively used in various safety critical applications. Hence these models along with being accurate need to be highly reliable. One way of achieving this is by quantifying uncertainty. Bayesian methods for UQ have been extensively studied for Deep Learning models applied on images but have been less explored for 3D modalities such as point clouds often used for Robots and Autonomous Systems. In this work, we evaluate three uncertainty quantification methods namely Deep Ensembles, MC-Dropout and MC-DropConnect on the DarkNet21Seg 3D semantic segmentation model and comprehensively analyze the impact of various parameters such as number of models in ensembles or forward passes, and drop probability values, on task performance and uncertainty estimate quality. We find that Deep Ensembles outperforms other methods in both performance and uncertainty metrics. Deep ensembles outperform other methods by a margin of 2.4% in terms of mIOU, 1.3% in terms of accuracy, while providing reliable uncertainty for decision making.Comment: 12 pages, 19 figures, ICML 2020 Workshop on Uncertainty and Robustness in Deep Learnin

Single Application of Low-Dose, Hydroxyapatite-Bound BMP-2 or GDF-5 Induces Long-Term Bone Formation and Biomechanical Stabilization of a Bone Defect in a Senile Sheep Lumbar Osteopenia Model

Effects of hydroxyapatite (HA) particles with bone morphogenetic BMP-2 or GDF-5 were compared in sheep lumbar osteopenia; in vitro release in phosphate-buffered saline (PBS) or sheep serum was assessed by ELISA. Lumbar (L) vertebral bone defects (Ø 3.5 mm) were generated in aged, osteopenic female sheep ( n = 72; 9.00 ± 0.11 years; mean ± SEM). Treatment was: (a) HA particles (2.5 mg; L5); or (b) particles coated with BMP-2 (1 µg; 10 µg) or GDF-5 (5 µg; 50 µg; L4; all groups n = 6). Untouched vertebrae (L3) served as controls. Three and nine months post-therapy, bone formation was assessed by osteodensitometry, histomorphometry, and biomechanical testing. Cumulative 14-day BMP release was high in serum (76–100%), but max. 1.4% in PBS. In vivo induction of bone formation by HA particles with either growth factor was shown by: (i) significantly increased bone volume, trabecular and cortical thickness (overall increase HA + BMP vs. control close to the injection channel 71%, 110%, and 37%, respectively); (ii) partial significant effects for bone mineral density, bone formation, and compressive strength (increase 17%; 9 months; GDF-5). Treatment effects were not dose-dependent. Combined HA and BMPs (single low-dose) highly augment long-term bone formation and biomechanical stabilization in sheep lumbar osteopenia. Thus, carrier-bound BMP doses 20,000-fold to 1000-fold lower than previously applied appear suitable for spinal fusion/bone regeneration and improved treatment safety

Mutations in GDF5 Reveal a Key Residue Mediating BMP Inhibition by NOGGIN

Signaling output of bone morphogenetic proteins (BMPs) is determined by two sets of opposing interactions, one with heterotetrameric complexes of cell surface receptors, the other with secreted antagonists that act as ligand traps. We identified two mutations (N445K,T) in patients with multiple synostosis syndrome (SYM1) in the BMP–related ligand GDF5. Functional studies of both mutants in chicken micromass culture demonstrated a gain of function caused by a resistance to the BMP–inhibitor NOGGIN and an altered signaling effect. Residue N445, situated within overlapping receptor and antagonist interfaces, is highly conserved among the BMP family with the exception of BMP9 and BMP10, in which it is substituted with lysine. Like the mutant GDF5, both BMPs are insensitive to NOGGIN and show a high chondrogenic activity. Ectopic expression of BMP9 or the GDF5 mutants resulted in massive induction of cartilage in an in vivo chick model presumably by bypassing the feedback inhibition imposed by endogenous NOGGIN. Swapping residues at the mutation site alone was not sufficient to render Bmp9 NOG-sensitive; however, successive introduction of two additional substitutions imparted high to total sensitivity on customized variants of Bmp9. In conclusion, we show a new mechanism for abnormal joint development that interferes with a naturally occurring regulatory mechanism of BMP signaling

Black-Box Optimization of Object Detector Scales

Object detectors have improved considerably in the last years by using advanced CNN architectures. However, many detector hyper-parameters are generally manually tuned, or they are used with values set by the detector authors. Automatic Hyper-parameter optimization has not been explored in improving CNN-based object detectors hyper-parameters. In this work, we propose the use of Black-box optimization methods to tune the prior/default box scales in Faster R-CNN and SSD, using Bayesian Optimization, SMAC, and CMA-ES. We show that by tuning the input image size and prior box anchor scale on Faster R-CNN mAP increases by 2% on PASCAL VOC 2007, and by 3% with SSD. On the COCO dataset with SSD there are mAP improvement in the medium and large objects, but mAP decreases by 1% in small objects. We also perform a regression analysis to find the significant hyper-parameters to tune

Evaluating Uncertainty Estimation Methods on 3D Semantic Segmentation of Point Clouds

Deep learning models are extensively used in various safety critical applications. Hence these models along with being accurate need to be highly reliable. One way of achieving this is by quantifying uncertainty. Bayesian methods for UQ have been extensively studied for Deep Learning models applied on images but have been less explored for 3D modalities such as point clouds often used for Robots and Autonomous Systems. In this work, we evaluate three uncertainty quantification methods namely Deep Ensembles, MC-Dropout and MC-DropConnect on the DarkNet21Seg 3D semantic segmentation model and comprehensively analyze the impact of various parameters such as number of models in ensembles or forward passes, and drop probability values, on task performance and uncertainty estimate quality. We find that Deep Ensembles outperforms other methods in both performance and uncertainty metrics. Deep ensembles outperform other methods by a margin of 2.4% in terms of mIOU, 1.3% in terms of accuracy, while providing reliable uncertainty for decision making

Human Chondrocytes Respond Discordantly to the Protein Encoded by the Osteoarthritis Susceptibility Gene <i>GDF5</i>

<div><p>A genetic deficit mediated by SNP rs143383 that leads to reduced expression of <i>GDF5</i> is strongly associated with large-joint osteoarthritis. We speculated that this deficit could be attenuated by the application of exogenous GDF5 protein and as a first step we have assessed what effect such application has on primary osteoarthritis chondrocyte gene expression. Chondrocytes harvested from cartilage of osteoarthritic patients who had undergone joint replacement were cultured with wildtype recombinant mouse and human GDF5 protein. We also studied variants of GDF5, one that has a higher affinity for the receptor BMPR-IA and one that is insensitive to the GDF5 antagonist noggin. As a positive control, chondrocytes were treated with TGF-β1. Chondrocytes were cultured in monolayer and micromass and the expression of genes coding for catabolic and anabolic proteins of cartilage were measured by quantitative PCR. The expression of the GDF5 receptor genes and the presence of their protein products was confirmed and the ability of GDF5 signal to translocate to the nucleus was demonstrated by the activation of a luciferase reporter construct. The capacity of GDF5 to elicit an intracellular signal in chondrocytes was demonstrated by the phosphorylation of intracellular Smads. Chondrocytes cultured with TGF-β1 demonstrated a consistent down regulation of <i>MMP1</i>, <i>MMP13</i> and a consistent upregulation of <i>TIMP1</i> and <i>COL2A1</i> with both culture techniques. In contrast, chondrocytes cultured with wildtype GDF5, or its variants, did not show any consistent response, irrespective of the culture technique used. Our results show that osteoarthritis chondrocytes do not respond in a predictable manner to culture with exogenous GDF5. This may be a cause or a consequence of the osteoarthritis disease process and will need to be surmounted if treatment with exogenous GDF5 is to be advanced as a potential means to overcome the genetic deficit conferring osteoarthritis susceptibility at this gene.</p></div

Dose response analysis using a Smad responsive reporter assay.

<p>Y-axis represents the luciferase activity readings generated in SW1353 cells in response to exogenous growth factor. Cells were stimulated for 6, 12, 24 and 48 hours with wildtype mouse GDF5 (A), wildtype human GDF5 (B), human GDF5 variant A (C) and human GDF5 variant B (D). BMP2 and TGF-β1 stimulations were used as positive controls. Error bars represent the standard error of the mean. GDF5 10, 10 ng/ml; GDF 30, 30 ng/ml; GDF5 100, 100 ng/ml; GDF5 300, 300 ng/ml. *P<0.05, **P<0.01, ***P<0.001, ****P<0.0001, *****P<0.00001, two-tailed Student’s t-test.</p

Activation of Smad signalling in OA chondrocytes after growth factor stimulation.

<p>Chondrocytes were cultured in monolayer and stimulated with each of the four GDF5 proteins for 15 minutes, 30 minutes, 1-β1 stimulation for 1 hour was used as a positive control. Protein was extracted and subjected to western blot analysis using an antibody against Smad 1/5/8 (anti-Smad1/5/8), with anti-β-actin antibody used as a loading control. This data comes from one OA patient. Identical data was obtained for a second OA patient (data not shown).</p

The amino acid substitutions of GDF5 variants A and B.

<p>The amino acid sequences from 430–466 for mouse GDF5 and from 436–472 for human GDF5 proteins are shown with the change in amino acid sequences in variant A and B highlighted in red. mGDF5, mouse GDF5; hGDF5, human GDF5.</p

Gene expression changes in monolayer chondrocytes treated with GDF5 or TGF-β1 compared to untreated cells.

<p>Chondrocytes were stimulated for 6, 12, 24 and 48 hours with wildtype mouse GDF5 (A), wildtype human GDF5 (B), human GDF5 variant A (C), human GDF5 variant B (D) and TGF-β1 (E). Each cross represents a significant (P<0.05, two-tailed Student’s t-test) up/down regulation of gene expression relative to untreated cells in one patient. Twelve patients were studied for each of the GDF5 growth factor treatments and ten patients for the TGF-β1 treatments.</p