Compressing Transformer-based self-supervised models for speech processing

Despite the success of Transformers in self- supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices. Several isolated attempts have been made to compress Transformers, but the settings and metrics are different across studies. Trade-off at various compression rates are also largely missing in prior work, making it difficult to compare compression techniques. In this work, we aim to provide context for the isolated results, studying several commonly used compression techniques, including weight pruning, head pruning, low-rank approximation, and knowledge distillation. We report trade- off at various compression rate, including wall-clock time, the number of parameters, and the number of multiply-accumulate operations. Our results show that compared to recent approaches, basic compression techniques are strong baselines. We further present several applications of our results, revealing properties of Transformers, such as the significance of diagonal attention heads. In addition, our results lead to a simple combination of compression techniques that improves trade-off over recent approaches. We hope the results would promote more diverse comparisons among model compression techniques and promote the use of model compression as a tool for analyzing models. Our code of compressing speech self-supervised model is available at https://github.com/nervjack2/Speech-SSL-Compression/.Comment: Submitted to IEEE Transactions on Audio, Speech and Language Processing (TASLP

Isolated tracheal injury after whiplash

AbstractWhiplash, a sudden acceleration–deceleration movement that can cause diverse symptoms such as neck pain, cervicogenic headache, restricted neck movement, tingling of the arms (central cord syndrome), and dizziness. However, laryngotracheal injuries after whiplash are extremely rare. We report the case of a 25-year-old Taiwanese female who presented to the emergency department with severe posterior midline neck pain after a rear-end motorcycle collision. Her C-spine X-ray showed no definite fracture; furthermore, her neck noncontrast-enhanced CT scan revealed paratracheal free air. She was discharged uneventfully after a 12-h observation period. Laryngotracheal injuries after whiplash, a hyperextension–hyperflexion movement, are potentially life-threatening and could lead to airway obstruction. Such injuries should not be overlooked. To the best of our knowledge, this is the first case report of isolated laryngotracheal injury after whiplash

Induction chemotherapy with dose-modified docetaxel, cisplatin, and 5-fluorouracil in Asian patients with borderline resectable or unresectable head and neck cancer

BackgroundSignificant ethnic differences in susceptibility to the effects of chemotherapy exist. Here, we retrospectively analyzed the safety and efficacy of induction chemotherapy (ICT) with dose-modified docetaxel, cisplatin, and 5-fluorouracil (TPF) in Asian patients with borderline resectable or unresectable head and neck squamous cell carcinoma (HNSCC).MethodsBased on the incidence of adverse events that occurred during daily practice, TPF90 (90% of the original TPF dosage; docetaxel 67.5 mg/m2 on Day 1, cisplatin 67.5 mg/m2 on Day 1, and 5-fluorouracil 675 mg/m2 on Days 1–5) was used for HNSCC patients who were scheduled to receive ICT TPF.ResultsBetween March 2011 and May 2014, 52 consecutive patients with borderline resectable or unresectable HNSCC were treated with ICT TPF90 followed by concurrent chemoradiotherapy. Forty-four patients (84.6%) received at least three cycles of ICT TPF90. The most commonly observed Grade 3–4 adverse events included neutropenia (35%), anemia (25%), stomatitis (35%), diarrhea (16%), and infections (13.5%). In an intention-to-treat analysis, the complete and partial response rates after ICT TPF90 were 13.5% and 59.6%, respectively. The complete and partial response rates following radiotherapy and salvage surgery were 42.3% and 25.0%, respectively. The estimated 3-year overall survival and progression-free survival rates were 41% [95% confidence interval (CI): 25–56%] and 23% (95% CI: 10–39%), respectively. The observed median overall survival and progression-free survival were 21.0 months (95% CI: 13.3–28.7 months) and 16.0 months (95% CI: 10.7–21.3 months), respectively.ConclusionTPF90 is a suitable option for Asian patients with borderline resectable or unresectable HNSCC who are scheduled for ICT

Reducing TRPC1 Expression through Liposome-Mediated siRNA Delivery Markedly Attenuates Hypoxia-Induced Pulmonary Arterial Hypertension in a Murine Model

We tested the hypothesis that Lipofectamine siRNA delivery to deplete transient receptor potential cation channel (TRPC) 1 protein expression can suppress hypoxia-induced pulmonary arterial hypertension (PAH) in mice. Adult male C57BL/6 mice were equally divided into group 1 (normal controls), group 2 (hypoxia), and group 3 (hypoxia + siRNA TRPC1). By day 28, right ventricular systolic pressure (RVSP), number of muscularized arteries, right ventricle (RV), and lung weights were increased in group 2 than in group 1 and reduced in group 3 compared with group 2. Pulmonary crowded score showed similar pattern, whereas number of alveolar sacs exhibited an opposite pattern compared to that of RVSP in all groups. Protein expressions of TRPCs, HIF-1α, Ku-70, apoptosis, and fibrosis and pulmonary mRNA expressions of inflammatory markers were similar pattern, whereas protein expressions of antifibrosis and VEGF were opposite to the pattern of RVSP. Cellular markers of pulmonary DNA damage, repair, and smooth muscle proliferation exhibited a pattern similar to that of RVSP. The mRNA expressions of proapoptotic and hypertrophy biomarkers displayed a similar pattern, whereas sarcomere length showed an opposite pattern compared to that of RVSP in all groups. Lipofectamine siRNA delivery effectively reduced TRPC1 expression, thereby attenuating PAH-associated RV and pulmonary arteriolar remodeling

Development of a Microfluidic-Based Optical Sensing Device for Label-Free Detection of Circulating Tumor Cells (CTCs) Through Their Lactic Acid Metabolism

This study reports a microfluidic-based optical sensing device for label-free detection of circulating tumor cells (CTCs), a rare cell species in blood circulation. Based on the metabolic features of cancer cells, live CTCs can be quantified indirectly through their lactic acid production. Compared with the conventional schemes for CTC detection, this label-free approach could prevent the biological bias due to the heterogeneity of the surface antigens on cancer cells. In this study, a microfluidic device was proposed to generate uniform water-in-oil cell-encapsulating micro-droplets, followed by the fluorescence-based optical detection of lactic acid produced within the micro-droplets. To test its feasibility to quantify cancer cells, experiments were carried out. Results showed that the detection signals were proportional to the number of cancer cells within the micro-droplets, whereas such signals were insensitive to the existence and number of leukocytes within. To further demonstrate its feasibility for cancer cell detection, the cancer cells with known cell number in a cell suspension was detected based on the method. Results revealed that there was no significant difference between the detected number and the real number of cancer cells. As a whole, the proposed method opens up a new route to detect live CTCs in a label-free manner

An Optically Induced Dielectrophoresis (ODEP)-Based Microfluidic System for the Isolation of High-Purity CD45^neg/EpCAM^neg Cells from the Blood Samples of Cancer Patients—Demonstration and Initial Exploration of the Clinical Significance of These Cells

Circulating tumour cells (CTCs) in blood circulation play an important role in cancer metastasis. CTCs are generally defined as the cells in circulating blood expressing the surface antigen EpCAM (epithelial cell adhesion molecule). Nevertheless, CTCs with a highly metastatic nature might undergo an epithelial-to-mesenchymal transition (EMT), after which their EpCAM expression is downregulated. In current CTC-related studies, however, these clinically important CTCs with high relevance to cancer metastasis could be missed due to the use of the conventional CTC isolation methodologies. To precisely explore the clinical significance of these cells (i.e., CD45neg/EpCAMneg cells), the high-purity isolation of these cells from blood samples is required. To achieve this isolation, the integration of fluorescence microscopic imaging and optically induced dielectrophoresis (ODEP)-based cell manipulation in a microfluidic system was proposed. In this study, an ODEP microfluidic system was developed. The optimal ODEP operating conditions and the performance of live CD45neg/EpCAMneg cell isolation were evaluated. The results demonstrated that the proposed system was capable of isolating live CD45neg/EpCAMneg cells with a purity as high as 100%, which is greater than the purity attainable using the existing techniques for similar tasks. As a demonstration case, the cancer-related gene expression of CD45neg/EpCAMneg cells isolated from the blood samples of healthy donors and cancer patients was successfully compared. The initial results indicate that the CD45neg/EpCAMneg nucleated cell population in the blood samples of cancer patients might contain cancer-related cells, particularly EMT-transformed CTCs, as suggested by the high detection rate of vimentin gene expression. Overall, this study presents an ODEP microfluidic system capable of simply and effectively isolating a specific, rare cell species from a cell mixture