Advancing Model Pruning via Bi-level Optimization

The deployment constraints in practical applications necessitate the pruning of large-scale deep learning models, i.e., promoting their weight sparsity. As illustrated by the Lottery Ticket Hypothesis (LTH), pruning also has the potential of improving their generalization ability. At the core of LTH, iterative magnitude pruning (IMP) is the predominant pruning method to successfully find 'winning tickets'. Yet, the computation cost of IMP grows prohibitively as the targeted pruning ratio increases. To reduce the computation overhead, various efficient 'one-shot' pruning methods have been developed, but these schemes are usually unable to find winning tickets as good as IMP. This raises the question of how to close the gap between pruning accuracy and pruning efficiency? To tackle it, we pursue the algorithmic advancement of model pruning. Specifically, we formulate the pruning problem from a fresh and novel viewpoint, bi-level optimization (BLO). We show that the BLO interpretation provides a technically-grounded optimization base for an efficient implementation of the pruning-retraining learning paradigm used in IMP. We also show that the proposed bi-level optimization-oriented pruning method (termed BiP) is a special class of BLO problems with a bi-linear problem structure. By leveraging such bi-linearity, we theoretically show that BiP can be solved as easily as first-order optimization, thus inheriting the computation efficiency. Through extensive experiments on both structured and unstructured pruning with 5 model architectures and 4 data sets, we demonstrate that BiP can find better winning tickets than IMP in most cases, and is computationally as efficient as the one-shot pruning schemes, demonstrating 2-7 times speedup over IMP for the same level of model accuracy and sparsity.Comment: Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022

Estimating the Fractional Cycle Biases for GPS Triple-Frequency Precise Point Positioning with Ambiguity Resolution Based on IGS Ultra-Rapid Predicted Orbits

We investigate the estimation of the fractional cycle biases (FCBs) for GPS triple-frequency uncombined precise point positioning (PPP) with ambiguity resolution (AR) based on the IGS ultra-rapid predicted (IGU) orbits. The impact of the IGU orbit errors on the performance of GPS triple-frequency PPP AR is also assessed. The extra-wide-lane (EWL), wide-lane (WL) and narrow-lane (NL) FCBs are generated with the single difference (SD) between satellites model using the global reference stations based on the IGU orbits. For comparison purposes, the EWL, WL and NL FCBs based on the IGS final precise (IGF) orbits are estimated. Each of the EWL, WL and NL FCBs based on IGF and IGU orbits are converted to the uncombined FCBs to implement the static and kinematic triple-frequency PPP AR. Due to the short wavelengths of NL ambiguities, the IGU orbit errors significantly impact the precision and stability of NL FCBs. An average STD of 0.033 cycles is achieved for the NL FCBs based on IGF orbits, while the value of the NL FCBs based on IGU orbits is 0.133 cycles. In contrast, the EWL and WL FCBs generated based on IGU orbits have comparable precision and stability to those generated based on IGF orbits. The use of IGU orbits results in an increased time-to-first-fix (TTFF) and lower fixing rates compared to the use of IGF orbits. Average TTFFs of 23.3 min (static) and 31.1 min (kinematic) and fixing rates of 98.1% (static) and 97.4% (kinematic) are achieved for the triple-frequency PPP AR based on IGF orbits. The average TTFFs increase to 27.0 min (static) and 37.9 min (kinematic) with fixing rates of 97.0% (static) and 96.3% (kinematic) based on the IGU orbits. The convergence times and positioning accuracy of PPP and PPP AR based on IGU orbits are slightly worse than those based on IGF orbits. Additionally, limited by the number of satellites transmitting three frequency signals, the introduction of the third frequency, L5, has a marginal impact on the performance of PPP and PPP AR. The GPS triple-frequency PPP AR performance is expected to improve with the deployment of new-generation satellites capable of transmitting the L5 signal

Synergistic suppression of pre-perfusion of donor livers with recipient serum and cobra venom factor treatment on hyperacute rejection following liver xenotransplantation Supressão sinérgica da rejeição hiperaguda no xenotransplante hepático com uso da pre-perfusão dos fígados doadores tratados com soro do receptor e com fator veneno de cobra

PURPOSE: To investigate synergistic suppression of donor liver pre-perfusion with recipient serum (RS) and cobra venom factor (CVF) treatment on hyperacute rejection (HAR) following liver xenotransplantation. METHODS: Guinea-pigs (GP, n=24) and Sprague-Dawley rats (SD, n=24) were recruited. Before transplantation, serum was collected from SD rats and used for preparation of inactivated complements. GP and SD rats were randomly assigned into four groups (n=6), respectively: RS group, CVF group, RS+CVF group and control group. Orthotopic liver xenotransplantation was performed with modified two-cuff technique. The survival time and liver function of recipients, morphological and pathological changes in rat livers were investigated. RESULTS: There was no piebald like change in the recipient livers in all experiment groups. The survival time of recipients in all experiment groups was longer than that in control group (p<0.05). Moreover, the survival time in the RS+CVF group was markedly longer than that in the RS group (p<0.01) and CVF group (p<0.05). The serum ALT level in all experiment groups were lower than that in the control group (p<0.05). Furthermore, the ALT level in the RS+CVF group was significantly lower than that in the CVF group (p<0.05) and RS group (p<0.01). The histological damages were significantly improved when compared with the control group, and the histological damages in the RS+CVF group were milder than those in the remaining groups (p<0.05) CONCLUSION: Pre-perfusion of donor liver with recipient serum and cobra venom factor treatment can exert synergistic suppressive effects on the hyperacute rejection following liver xenotransplantation.<br>OBJETIVO: Investigar a supressão sinérgica da pré-perfusão do doador de fígado com soro do receptor (SR) e tratamento com fator veneno de cobra (FVC) na rejeição hiperaguda (RHA) após o xenotransplante de fígado. MÉTODOS: Foram utilizados Cobaias (GP, n=24) e ratos Sprague-Dawley (SD, n=24). Antes do transplante foram coletadas amostras de soro dos ratos SD e usados para a preparação dos complementos inativados. Cobaias GP e ratos SD foram randomicamente distribuídos em quatro grupos (n=6), respectivamente: grupo RS, grupo FVC, grupo SR+FVC e grupo controle. Xenotransplante ortotópico do fígado foi realizado com a técnica de dois cuffs modificados. Foram investigados o de tempo de sobrevida, a função hepática dos receptores e alterações morfopatológicas em fígados de ratos. RESULTADOS: Não houve alteração na coloração do parênquima dos fígados nos receptores. O tempo de sobrevida dos receptores em todos os grupos experimentais foi mais longo do que o grupo controle (p<0,05). Além disso, o tempo de sobrevida do grupo SR+ FVC foi marcadamente maior do que o grupo SR (p<0,01) e o grupo FVC (p<0,05). O nível sérico ALT foi menor em todos os grupos experimentais do que o grupo controle (p<0,05). O nível de ALT no grupo SR+ FVC foi significantemente menor do que no grupo FVC (p<0,05) e o grupo SR (p<0,01). As alterações histológicas foram significantemente melhoradas quando comparado com o grupo controle, e os danos histológicos no grupo SR+ FVC foram mais moderados do que nos grupos restantes (p<0,05). CONCLUSÃO: Pré-perfusão do fígado doador com soro do receptor e fator veneno de cobra pode exercer efeito supressor sinérgico da rejeição hiperaguda após xenotransplante de fígado

Endothelial cell heterogeneity and microglia regulons revealed by a pig cell landscape at single-cell level

Pigs are valuable large animal models for biomedical and genetic research, but insights into the tissue- and cell-type-specific transcriptome and heterogeneity remain limited. By leveraging single-cell RNA sequencing, we generate a multiple-organ single-cell transcriptomic map containing over 200,000 pig cells from 20 tissues/organs. We comprehensively characterize the heterogeneity of cells in tissues and identify 234 cell clusters, representing 58 major cell types. In-depth integrative analysis of endothelial cells reveals a high degree of heterogeneity. We identify several functionally distinct endothelial cell phenotypes, including an endothelial to mesenchymal transition subtype in adipose tissues. Intercellular communication analysis predicts tissue- and cell type-specific crosstalk between endothelial cells and other cell types through the VEGF, PDGF, TGF-beta, and BMP pathways. Regulon analysis of single-cell transcriptome of microglia in pig and 12 other species further identifies MEF2C as an evolutionally conserved regulon in the microglia. Our work describes the landscape of single-cell transcriptomes within diverse pig organs and identifies the heterogeneity of endothelial cells and evolutionally conserved regulon in microglia