Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks

Uncertainty quantification (UQ) is important for reliability assessment and enhancement of machine learning models. In deep learning, uncertainties arise not only from data, but also from the training procedure that often injects substantial noises and biases. These hinder the attainment of statistical guarantees and, moreover, impose computational challenges on UQ due to the need for repeated network retraining. Building upon the recent neural tangent kernel theory, we create statistically guaranteed schemes to principally \emph{quantify}, and \emph{remove}, the procedural uncertainty of over-parameterized neural networks with very low computation effort. In particular, our approach, based on what we call a procedural-noise-correcting (PNC) predictor, removes the procedural uncertainty by using only \emph{one} auxiliary network that is trained on a suitably labeled data set, instead of many retrained networks employed in deep ensembles. Moreover, by combining our PNC predictor with suitable light-computation resampling methods, we build several approaches to construct asymptotically exact-coverage confidence intervals using as low as four trained networks without additional overheads

Quantifying Epistemic Uncertainty in Deep Learning

Uncertainty quantification is at the core of the reliability and robustness of machine learning. In this paper, we provide a theoretical framework to dissect the uncertainty, especially the epistemic component, in deep learning into procedural variability (from the training procedure) and data variability (from the training data), which is the first such attempt in the literature to our best knowledge. We then propose two approaches to estimate these uncertainties, one based on influence function and one on batching. We demonstrate how our approaches overcome the computational difficulties in applying classical statistical methods. Experimental evaluations on multiple problem settings corroborate our theory and illustrate how our framework and estimation can provide direct guidance on modeling and data collection effort to improve deep learning performance

A modified constitutive model for tensile deformation of 9%Cr steel under prior fatigue loading

Reliable constitutive models are necessary for the precise design and manufacture of complicated components. This study is devoted to developing a modified constitutive model to capture the effects of prior fatigue loading on subsequent tensile deformation of 9%Cr steel. In the proposed model, a strain hardening rule combined with a defined fatigue damage parameter was introduced to represent prior fatigue damage. The defined fatigue damage parameter based on the inelastic strain range of each cycle is capable of describing the evolution of tensile strength, recovery of martensite laths and decline of dislocation density, regardless of the variation in fatigue loading conditions. To validate the predictive capacity of the proposed model, experimental tensile results at different strain amplitudes, lifetime fractions and hold times of prior fatigue loading were compared with the predicted results. Good agreement between experimental and predicted results indicates that the proposed model is robust in describing the tensile behaviour under prior fatigue loading. Moreover, few determined material parameters are required, which makes the proposed model convenient for practical applications

Partition-A-Medical-Image: Extracting Multiple Representative Sub-regions for Few-shot Medical Image Segmentation

Few-shot Medical Image Segmentation (FSMIS) is a more promising solution for medical image segmentation tasks where high-quality annotations are naturally scarce. However, current mainstream methods primarily focus on extracting holistic representations from support images with large intra-class variations in appearance and background, and encounter difficulties in adapting to query images. In this work, we present an approach to extract multiple representative sub-regions from a given support medical image, enabling fine-grained selection over the generated image regions. Specifically, the foreground of the support image is decomposed into distinct regions, which are subsequently used to derive region-level representations via a designed Regional Prototypical Learning (RPL) module. We then introduce a novel Prototypical Representation Debiasing (PRD) module based on a two-way elimination mechanism which suppresses the disturbance of regional representations by a self-support, Multi-direction Self-debiasing (MS) block, and a support-query, Interactive Debiasing (ID) block. Finally, an Assembled Prediction (AP) module is devised to balance and integrate predictions of multiple prototypical representations learned using stacked PRD modules. Results obtained through extensive experiments on three publicly accessible medical imaging datasets demonstrate consistent improvements over the leading FSMIS methods. The source code is available at https://github.com/YazhouZhu19/PAMI

Microstructural damage mechanics-based model for creep fracture of 9%Cr steel under prior fatigue loading

Predicting the remnant creep fracture life precisely is crucial for ensuring safety of high temperature components. This study presents a microstructural damage mechanics-based model for creep fracture of 9%Cr steel under prior fatigue loading. Microstructure observation reveals that the decrease of dislocation density and the growth of martensite lath width occurred during prior fatigue process contribute to the degradation of creep strength. Particularly, coarsening of martensite lath width plays the dominated role. To take into account the effect of the prior fatigue loading, kinematic damage equations that represent the evolution of dislocation density and martensite lath are proposed in the developed model. With the proposed model, creep fracture life and creep failure strain at various lifetime factions, strain amplitudes and hold times of prior fatigue loading can be satisfactorily predicted, which manifests that the proposed model is robust in capturing the effects of various prior fatigue loadings. The proposed model is also shown to be able to accurately predict prolonged creep deformation of other similar steel after different prior fatigue loadings

Few-Shot Medical Image Segmentation via a Region-enhanced Prototypical Transformer

Automated segmentation of large volumes of medical images is often plagued by the limited availability of fully annotated data and the diversity of organ surface properties resulting from the use of different acquisition protocols for different patients. In this paper, we introduce a more promising few-shot learning-based method named Region-enhanced Prototypical Transformer (RPT) to mitigate the effects of large intra-class diversity/bias. First, a subdivision strategy is introduced to produce a collection of regional prototypes from the foreground of the support prototype. Second, a self-selection mechanism is proposed to incorporate into the Bias-alleviated Transformer (BaT) block to suppress or remove interferences present in the query prototype and regional support prototypes. By stacking BaT blocks, the proposed RPT can iteratively optimize the generated regional prototypes and finally produce rectified and more accurate global prototypes for Few-Shot Medical Image Segmentation (FSMS). Extensive experiments are conducted on three publicly available medical image datasets, and the obtained results show consistent improvements compared to state-of-the-art FSMS methods. The source code is available at: https://github.com/YazhouZhu19/RPT.Comment: Accepted by MICCA

Evaluation of the effect of various prior creep-fatigue interaction damages on subsequent tensile and creep properties of 9%Cr steel

The degradation of tensile and creep properties is inevitable during high temperature service operation. Hence this work aims to evaluate the effect of prior creep-fatigue interaction damages on remnant tensile and creep properties of 9%Cr steel. Prior creep-fatigue tests interrupted at different lifetime fractions and different tensile hold times are performed at 650 °C. Afterwards, subsequent tensile and creep tests are conducted at the same temperature. Results reveal that high lifetime fraction of prior creep-fatigue loading leads to obvious reduction of remnant tensile strength and creep resistance. However, the increase in tensile hold time hardly alters the remnant properties. Microstructure and fracture surface observations indicate that the deterioration of remnant tensile strength is mainly ascribed to the decline of dislocation density occurred during prior creep-fatigue process, whereas the growth of martensite lath plays the dominated role in the reduction of remnant creep resistance. Moreover, surface crack also accelerates the decline of creep resistance at high lifetime fraction. To quantify the prior creep-fatigue interaction damage, a fatigue damage indicator is proposed. Determined relationships between remnant tensile, creep properties and defined fatigue damage are obtained

Confidence-and-Refinement Adaptation Model for Cross-Domain Semantic Segmentation

With the rapid development of convolutional neural networks (CNNs), significant progress has been achieved in semantic segmentation. Despite the great success, such deep learning approaches require large scale real-world datasets with pixel-level annotations. However, considering that pixel-level labeling of semantics is extremely laborious, many researchers turn to utilize synthetic data with free annotations. But due to the clear domain gap, the segmentation model trained with the synthetic images tends to perform poorly on the real-world datasets. Unsupervised domain adaptation (UDA) for semantic segmentation recently gains an increasing research attention, which aims at alleviating the domain discrepancy. Existing methods in this scope either simply align features or the outputs across the source and target domains or have to deal with the complex image processing and post-processing problems. In this work, we propose a novel multi-level UDA model named Confidence-and-Refinement Adaptation Model (CRAM), which contains a confidence-aware entropy alignment (CEA) module and a style feature alignment (SFA) module. Through CEA, the adaptation is done locally via adversarial learning in the output space, making the segmentation model pay attention to the high-confident predictions. Furthermore, to enhance the model transfer in the shallow feature space, the SFA module is applied to minimize the appearance gap across domains. Experiments on two challenging UDA benchmarks ``GTA5-to-Cityscapes'' and ``SYNTHIA-to-Cityscapes'' demonstrate the effectiveness of CRAM. We achieve comparable performance with the existing state-of-the-art works with advantages in simplicity and convergence speed

Estimate-Then-Optimize Versus Integrated-Estimation-Optimization: A Stochastic Dominance Perspective

In data-driven stochastic optimization, model parameters of the underlying distribution need to be estimated from data in addition to the optimization task. Recent literature suggests the integration of the estimation and optimization processes, by selecting model parameters that lead to the best empirical objective performance. Such an integrated approach can be readily shown to outperform simple ``estimate then optimize" when the model is misspecified. In this paper, we argue that when the model class is rich enough to cover the ground truth, the performance ordering between the two approaches is reversed for nonlinear problems in a strong sense. Simple ``estimate then optimize" outperforms the integrated approach in terms of stochastic dominance of the asymptotic optimality gap, i,e, the mean, all other moments, and the entire asymptotic distribution of the optimality gap is always better. Analogous results also hold under constrained settings and when contextual features are available. We also provide experimental findings to support our theory