Connecting Speech Encoder and Large Language Model for ASR

The impressive capability and versatility of large language models (LLMs) have aroused increasing attention in automatic speech recognition (ASR), with several pioneering studies attempting to build integrated ASR models by connecting a speech encoder with an LLM. This paper presents a comparative study of three commonly used structures as connectors, including fully connected layers, multi-head cross-attention, and Q-Former. Speech encoders from the Whisper model series as well as LLMs from the Vicuna model series with different model sizes were studied. Experiments were performed on the commonly used LibriSpeech, Common Voice, and GigaSpeech datasets, where the LLMs with Q-Formers demonstrated consistent and considerable word error rate (WER) reductions over LLMs with other connector structures. Q-Former-based LLMs can generalise well to out-of-domain datasets, where 12% relative WER reductions over the Whisper baseline ASR model were achieved on the Eval2000 test set without using any in-domain training data from Switchboard. Moreover, a novel segment-level Q-Former is proposed to enable LLMs to recognise speech segments with a duration exceeding the limitation of the encoders, which results in 17% relative WER reductions over other connector structures on 90-second-long speech data

Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models

Audio-visual large language models (LLM) have drawn significant attention, yet the fine-grained combination of both input streams is rather under-explored, which is challenging but necessary for LLMs to understand general video inputs. To this end, a fine-grained audio-visual joint representation (FAVOR) learning framework for multimodal LLMs is proposed in this paper, which extends a text-based LLM to simultaneously perceive speech and audio events in the audio input stream and images or videos in the visual input stream, at the frame level. To fuse the audio and visual feature streams into joint representations and to align the joint space with the LLM input embedding space, we propose a causal Q-Former structure with a causal attention module to enhance the capture of causal relations of the audio-visual frames across time. An audio-visual evaluation benchmark (AVEB) is also proposed which comprises six representative single-modal tasks with five cross-modal tasks reflecting audio-visual co-reasoning abilities. While achieving competitive single-modal performance on audio, speech and image tasks in AVEB, FAVOR achieved over 20% accuracy improvements on the video question-answering task when fine-grained information or temporal causal reasoning is required. FAVOR, in addition, demonstrated remarkable video comprehension and reasoning abilities on tasks that are unprecedented by other multimodal LLMs. An interactive demo of FAVOR is available at https://github.com/BriansIDP/AudioVisualLLM.git, and the training code and model checkpoints will be released soon

Mutations of genes in synthesis of the carotenoid precursors of ABA lead to pre-harvest sprouting and photo-oxidation in rice

Pre-harvest sprouting (PHS) or vivipary in cereals is an important agronomic trait that results in significant economic loss. A considerable number of mutations that cause PHS have been identified in several species. However, relatively few viviparous mutants in rice (Oryza sativa L.) have been reported. To explore the mechanism of PHS in rice, we carried out an extensive genetic screening and identified 12 PHS mutants (phs). Based on their phenotypes, these phs mutants were classified into three groups. Here we characterize in detail one of these groups, which contains mutations in genes encoding major enzymes of the carotenoid biosynthesis pathway, including phytoene desaturase (OsPDS), ζ-carotene desaturase (OsZDS), carotenoid isomerase (OsCRTISO) and lycopene β-cyclase (β-OsLCY), which are essential for the biosynthesis of carotenoid precursors of ABA. As expected, the amount of ABA was reduced in all four phs mutants compared with that in the wild type. Chlorophyll fluorescence analysis revealed the occurrence of photoinhibition in the photosystem and decreased capacity for eliminating excess energy by thermal dissipation. The greatly increased activities of reactive oxygen species (ROS) scavenging enzymes, and reduced photosystem (PS) II core proteins CP43, CP47 and D1 in leaves of the Oscrtiso/phs3-1 mutant and OsLCY RNAi transgenic rice indicated that photo-oxidative damage occurred in PS II, consistent with the accumulation of ROS in these plants. These results suggest that the impairment of carotenoid biosynthesis causes photo-oxidation and ABA-deficiency phenotypes, of which the latter is a major factor controlling the PHS trait in rice

H9N2 Viruses Isolated From Mammals Replicated in Mice at Higher Levels Than Avian-Origin Viruses

H9N2 subtype influenza A virus (IAV) has more than 20 genotypes that are able to cross species barriers and expand from birds to mammals and humans. To better understand the impact of different H9N2 genotypes and their characteristics, five H9N2 viruses from different hosts including chickens, geese, pigs, mink, and humans representing the B69 88(Gs/14, Ck/15, and Mi/14), B35 (Sw/08) and G9 genotypes (Hu/04) were infected in chicken and mice. In mice, mammal-origin viruses replicated at higher levels in the lungs compared to avian viruses. The goose-virus replicated at the lowest levels indicating poor adaptation. Increased pro-inflammatory cytokines were positively correlated with viral loads in the lung. In chickens, all viruses were excreted from cloacal and/or oropharyngeal swabs. Interestingly, Mink-origin virus exhibited higher virulence and replication in mice and chickens. Our data indicate that mammal-origin H9N2 viruses are more adapted and virulent in mice than the avian-origin viruses

The Protective Efficacy of a SARS-CoV-2 Vaccine Candidate B.1.351V against Several Variant Challenges in K18-hACE2 Mice

The emergence of SARS-CoV-2 variants of concern (VOCs) with increased transmissibility and partial resistance to neutralization by antibodies has been observed globally. There is an urgent need for an effective vaccine to combat these variants. Our study demonstrated that the B.1.351 variant inactivated vaccine candidate (B.1.351V) generated strong binding and neutralizing antibody responses in BALB/c mice against the B.1.351 virus and other SARS-CoV-2 variants after two doses within 28 days. Immunized K18-hACE2 mice also exhibited elevated levels of live virus-neutralizing antibodies against various SARS-CoV-2 viruses. Following infection with these viruses, K18-hACE2 mice displayed a stable body weight, a high survival rate, minimal virus copies in lung tissue, and no lung damage compared to the control group. These findings indicate that B.1.351V offered protection against infection with multiple SARS-CoV-2 variants in mice, providing insights for the development of a vaccine targeting SARS-CoV-2 VOCs for human use

Recombinant proteins A29L, M1R, A35R, and B6R vaccination protects mice from mpox virus challenge

Since May 2022, mutant strains of mpox (formerly monkeypox) virus (MPXV) have been rapidly spreading among individuals who have not traveled to endemic areas in multiple locations, including Europe and the United States. Both intracellular and extracellular forms of mpox virus have multiple outer membrane proteins that can stimulate immune response. Here, we investigated the immunogenicity of MPXV structural proteins such as A29L, M1R, A35R, and B6R as a combination vaccine, and the protective effect against the 2022 mpox mutant strain was also evaluated in BALB/c mice. After mixed 15 μg QS-21 adjuvant, all four virus structural proteins were administered subcutaneously to mice. Antibody titers in mouse sera rose sharply after the initial boost, along with an increased capacity of immune cells to produce IFN-γ alongside an elevated level of cellular immunity mediated by Th1 cells. The vaccine-induced neutralizing antibodies significantly inhibited the replication of MPXV in mice and reduced the pathological damage of organs. This study demonstrates the feasibility of a multiple recombinant vaccine for MPXV variant strains

Evaluation and identification of powdery mildew-resistant genes in 137 wheat relatives

Powdery mildew is one of the most severe diseases affecting wheat yield and quality and is caused by Blumeria graminis f. sp. tritici (Bgt). Host resistance is the preferred strategy to prevent this disease. However, the narrow genetic basis of common wheat has increased the demand for diversified germplasm resources against powdery mildew. Wheat relatives, especially the secondary gene pool of common wheat, are important gene donors in the genetic improvement of common wheat because of its abundant genetic variation and close kinship with wheat. In this study, a series of 137 wheat relatives, including 53 Triticum monococcum L. (2n = 2x = 14, AA), 6 T. urartu Thumanjan ex Gandilyan (2n = 2x = 14, AA), 9 T. timopheevii Zhuk. (2n = 4x = 28, AAGG), 66 T. aestivum subsp. spelta (2n = 6x = 42, AABBDD), and 3 Aegilops speltoides (2n = 2x = 14, SS) were systematically evaluated for their powdery mildew resistance and composition of Pm genes. Out of 137 (60.58%) accessions, 83 were resistant to Bgt isolate E09 at the seedling stage, and 116 of 137 (84.67%) wheat relatives were resistant to the mixture of Bgt isolates at the adult stage. This indicates that these accessions show a high level of resistance to powdery mildew. Some 31 markers for 23 known Pm genes were used to test these 137 accessions, and, in the results, only Pm2, Pm4, Pm6, Pm58, and Pm68 were detected. Among them, three Pm4 alleles (Pm4a, Pm4b, and Pm4f) were identified in 4 T. subsp. spelta accessions. q-RT PCR further confirmed that Pm4 alleles played a role in disease resistance in these four accessions. The phylogenetic tree showed that the kinship of Pm4 was close to Pm24 and Sr62. This study not only provides reference information and valuable germplasm resources for breeding new wheat varieties with disease resistance but also lays a foundation for enriching the genetic basis of wheat resistance to powdery mildew

Chinese Antarctic Magnetometer Chain at the Cusp Latitude

A Chinese Antarctic Magnetometer (CAM) chain from Zhongshan Station (ZHS) to Dome-A (DMA) has been established since February 2009. A regular magnetometer is operated at ZHS, and four low power magnetometers are operated along the interior route from ZHS to DMA in the cusp latitude, extending over a distance of 1260 km. These stations fill an important void in the Antarctic magnetometer network. Furthermore, the CAM chain is magnetically conjugated with the Arctic region reaching from the Svalbard archipelago to Daneborg, on the east coast of Greenland. Conjugate measurements using the Arctic and Antarctic magnetometers provide excellent opportunities to investigate phenomena related to the coupling of the solar wind to the magnetosphere and ionosphere, such as magnetic impulse events, flux transfer events, traveling convection vortices and ultra-low frequency waves

The Ark: Sanctuary for ISFs

The thesis is aiming to decrease the potential flooding damages to the ISFs, and a balance between the limited resources and high population density to relieve the flooding issues through the renovation and reconstruction of local churches in the Baseco Compound. Flooding has become one of the most common and devastated issues in developing countries, and it is worsen by the housing crisis that lead to a formation of informal settlements with shoddy constructions and destabilized communities. What is an applicable approach to decrease the potential flooding damage to the ISFs? How to find a balance between the limited resources and high population density to relief the flooding issues architecturally