SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

How to boost speech pre-training with textual data is an unsolved problem due to the fact that speech and text are very different modalities with distinct characteristics. In this paper, we propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities, including phoneme-unit and hidden-unit tokenizers, which can be trained using a small amount of paired speech-text data. Based on the trained tokenizers, we convert the unlabeled speech and text data into tokens of phoneme units or hidden units. The pre-training objective is designed to unify the speech and the text into the same discrete semantic space with a unified Transformer network. Leveraging only 10K text sentences, our SpeechLM gets a 16\% relative WER reduction over the best base model performance (from 6.8 to 5.7) on the public LibriSpeech ASR benchmark. Moreover, SpeechLM with fewer parameters even outperforms previous SOTA models on CoVoST-2 speech translation tasks. We also evaluate our SpeechLM on various spoken language processing tasks under the universal representation evaluation framework SUPERB, demonstrating significant improvements on content-related tasks. Our code and models are available at https://aka.ms/SpeechLM.Comment: 14 page

The First Release of the CSTAR Point Source Catalog from Dome A, Antarctica

In 2008 January the 24th Chinese expedition team successfully deployed the Chinese Small Telescope ARray (CSTAR) to DomeA, the highest point on the Antarctic plateau. CSTAR consists of four 14.5cm optical telescopes, each with a different filter (g, r, i and open) and has a 4.5degree x 4.5degree field of view (FOV). It operates robotically as part of the Plateau Observatory, PLATO, with each telescope taking an image every 30 seconds throughout the year whenever it is dark. During 2008, CSTAR #1 performed almost flawlessly, acquiring more than 0.3 million i-band images for a total integration time of 1728 hours during 158 days of observations. For each image taken under good sky conditions, more than 10,000 sources down to 16 mag could be detected. We performed aperture photometry on all the sources in the field to create the catalog described herein. Since CSTAR has a fixed pointing centered on the South Celestial Pole (Dec =-90 degree), all the sources within the FOV of CSTAR were monitored continuously for several months. The photometric catalog can be used for studying any variability in these sources, and for the discovery of transient sources such as supernovae, gamma-ray bursts and minor planets.Comment: 1 latex file and 9 figures The paper is accepted by PAS

Machine Learning Clustering Analysis Towards Educator’s Readiness to Adopt Augmented Reality as a Teaching Tool

The advanced digital revolution has shifted conventional teaching and learning into digital education. In consistency with digital education, Augmented Reality (AR) applications started to shine in the education industry for their ability to create conducive teaching and learning environments, especially in remote learning during the COVID-19 pandemic. Movement Control Order (MCO) implemented in the year 2020 has led to emergency remote teaching and learning without much preparation for all educators and learners. Throughout these few years, most educators got familiar with digital teaching tools and online teaching platforms. Hence, this study aims to explore educators’ readiness to adopt AR as a teaching tool in their teaching during the endemic period. A quantitative approach via questionnaire has been distributed to the Private Higher Education Institutions (PHEIs) in the states of Selangor and Kuala Lumpur. Machine learning using a clustering technique was used to find patterns between the demographics of educators towards the AR perception of educators. The results revealed that educators' perceptions of AR technology are influenced by their familiarity with it, their personal beliefs, and their attitudes toward technology. This study provides an insightful overview of the benefits of AR applications in education and the implications of the adoption of AR in Malaysian schools and educational institutions. It also highlights the importance of motivating educators and students to embrace AR as an enhancement learning tool, providing a valuable discussion for the government, learning institutions, and educators on the implementation of AR in Malaysia

The sky brightness and transparency in i-band at Dome A, Antarctica

The i-band observing conditions at Dome A on the Antarctic plateau have been investigated using data acquired during 2008 with the Chinese Small Telescope ARray. The sky brightness, variations in atmospheric transparency, cloud cover, and the presence of aurorae are obtained from these images. The median sky brightness of moonless clear nights is 20.5 mag arcsec^{-2} in the SDSS $i$ band at the South Celestial Pole (which includes a contribution of about 0.06 mag from diffuse Galactic light). The median over all Moon phases in the Antarctic winter is about 19.8 mag arcsec^{-2}. There were no thick clouds in 2008. We model contributions of the Sun and the Moon to the sky background to obtain the relationship between the sky brightness and transparency. Aurorae are identified by comparing the observed sky brightness to the sky brightness expected from this model. About 2% of the images are affected by relatively strong aurorae.Comment: There are 1 Latex file and 14 figures accepted by A

Toll-like receptor 2 and Toll-like receptor 4 exhibit distinct regulation of cancer cell stemness mediated by cell death-induced high-mobility group box 1

Background: High-mobility group box 1 (HMGB1), a common extracellular damage associated molecular pattern molecule, is overexpressed in several solid tumors including pancreatic carcinoma. We previously observed that radiotherapy induced dying cells secrete HMGB1 and accelerate pancreatic carcinoma progression through an unclear mechanism.Methods: Using the Millicell system as an in vitro co-culture model, we performed quantitative reverse transcriptase-polymerase chain reaction, western blot and sphere forming ability analyses to access the effect of dying-cell-derived HMGB1 on CD133(+) cancer cell stemness in vitro and in vivo. Interactions between HMGB1 and Toll-like receptor 2(TLR2)/TLR4 were studied by co-immunoprecipitation. Western blot and short-hairpin RNA-based knockdown assays were conducted to detect HMGB1 and TLR2/TLR4 signaling activity.Findings: Radiation-associated, dying-cell-derived HMGB1 maintained stemness and contributed to CD133(+) cancer stemcell self-renewal in vitro and in vivo. In overexpressing and silencing experiments, we demonstrated that the process was activated by TLR2 receptor, whereas TLR4 antagonized HMGB1-TLR2 signaling. Wnt/beta-catenin signaling supported the HMGB1-TLR2 mediated stemness of CD133(+) cancer cells.Interpretation: Our results show how irradiation-induced cell death might enhance the stemness of resident cancer cells, and indicate HMGB1-TLR2 signaling as a potential therapeutic target for preventing pancreatic cancer recurrence. (c) 2018 Haitao Zhu. Published by Elsevier B.V

Dedifferentiation process driven by radiotherapy-induced HMGB1/TLR2/YAP/HIF-1α signaling enhances pancreatic cancer stemness

Differentiated cancer cells reacquiring stem cell traits following radiotherapy may enrich cancer stem cells and accelerate tumor recurrence and metastasis. We are interested in the mechanistic role of dying cells-derived HMGB1 in CD133− pancreatic cancer cells dedifferentiation following radiotherapy. We firstly confirmed that X-ray irradiation induced differentiation of CD133− pancreatic cancer cells, from either sorted from patient samples or established cell lines, into cancer stem-like cells (iCSCs). Using an in vitro coculture model, X-ray irradiation induced dying cells to release HMGB1, which further promoted CD133− pancreatic cancer cells regaining stem cell traits, such as higher sphere forming ability and expressed higher level of stemness-related genes and proteins. Inhibiting the expression and activity of HMGB1 attenuated the dedifferentiation stimulating effect of irradiated, dying cells on C133− pancreatic cancer cells in vitro and in PDX models. Mechanistically, HMGB1 binding with TLR2 receptor functions in a paracrine manner to affect CD133− pancreatic cancer cells dedifferentiation via activating Hippo-YAP pathway and HIF-1α expression in oxygen independent manner in vitro and in vivo. We conclude that X-ray irradiation induces CD133− pancreatic cancer cell dedifferentiation into a CSC phenotype, and inhibiting HMGB1 may be a strategy to prevent CSC enrichment and further pancreatic carcinoma relapse.</p

Identification and characterization of circular RNAs in mammary gland tissue from sheep at peak lactation and during the nonlactating period

Circular RNAs are a class of noncoding RNA with a widespread occurrence in eukaryote tissues, and with some having been demonstrated to have clear biological function. In sheep, little is known about the role of circular RNAs in mammary gland tissue, and therefore an RNA sequencing approach was used to compare mammary gland tissue expression of circular RNAs in 9 Small Tail Han sheep at peak lactation, and subsequently when they were not lactating. These 9 sheep had their RNA pooled for analysis into 3 libraries from peak lactation and 3 from the nonlactating period. A total of 3,278 and 1,756 circular RNAs were identified in the peak lactation and nonlactating mammary gland tissues, respectively, and the expression and identity of 9 of them was confirmed using reverse transcriptase-polymerase chain reaction analysis and DNA sequencing. The type, chromosomal location and length of the circular RNAs identified were ascertained. Forty upregulated and one downregulated circular RNAs were characterized in the mammary gland tissue at peak lactation compared with the nonlactating mammary gland tissue. Gene ontology enrichment analysis revealed that the parental genes of these differentially expressed circular RNAs were related to molecular function, binding, protein binding, ATP binding, and ion binding. Five differentially expression circular RNAs were selected for further analysis to predict their target microRNAs, and some microRNAs reportedly associated with the development of the mammary gland were found in the constructed circular RNA–microRNA network. This study reveals the expression profiles and characterization of circular RNAs at 2 key stages of mammary gland activity, thereby providing an improved understanding of the roles of circular RNAs in the mammary gland of sheep