RepCodec: A Speech Representation Codec for Speech Tokenization

With recent rapid growth of large language models (LLMs), discrete speech tokenization has played an important role for injecting speech into LLMs. However, this discretization gives rise to a loss of information, consequently impairing overall performance. To improve the performance of these discrete speech tokens, we present RepCodec, a novel speech representation codec for semantic speech tokenization. In contrast to audio codecs which reconstruct the raw audio, RepCodec learns a vector quantization codebook through reconstructing speech representations from speech encoders like HuBERT or data2vec. Together, the speech encoder, the codec encoder and the vector quantization codebook form a pipeline for converting speech waveforms into semantic tokens. The extensive experiments illustrate that RepCodec, by virtue of its enhanced information retention capacity, significantly outperforms the widely used k-means clustering approach in both speech understanding and generation. Furthermore, this superiority extends across various speech encoders and languages, affirming the robustness of RepCodec. We believe our method can facilitate large language modeling research on speech processing

A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis

In human speech, the attitude of a speaker cannot be fully expressed only by the textual content. It has to come along with the intonation. Declarative questions are commonly used in daily Cantonese conversations, and they are usually uttered with rising intonation. Vanilla neural text-to-speech (TTS) systems are not capable of synthesizing rising intonation for these sentences due to the loss of semantic information. Though it has become more common to complement the systems with extra language models, their performance in modeling rising intonation is not well studied. In this paper, we propose to complement the Cantonese TTS model with a BERT-based statement/question classifier. We design different training strategies and compare their performance. We conduct our experiments on a Cantonese corpus named CanTTS. Empirical results show that the separate training approach obtains the best generalization performance and feasibility.Comment: Accepted by INTERSPEECH 202

CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning

Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech representation learning. The idea is to convert an utterance to a sequence of discrete codes, and perform code representation learning, where we predict the code representations based on a masked view of the original speech input. Unlike the prior self-distillation approaches of which the teacher and the student are of the same modality, our target model predicts representations from a different modality. CoBERT outperforms the most recent state-of-the-art performance on the ASR task and brings significant improvements on the SUPERB speech translation (ST) task.Comment: Submitted to ICASSP 202

Eigentrigraphemes for under-resourced languages

Abstract Grapheme-based modeling has an advantage over phone-based modeling in automatic speech recognition for under-resourced languages when a good dictionary is not available. Recently we proposed a new method for parameter estimation of context-dependent hidden Markov model (HMM) called eigentriphone modeling. Eigentriphone modeling outperforms conventional tied-state HMM by eliminating the quantization errors among the tied states. The eigentriphone modeling framework is very flexible and can be applied to any group of modeling unit provided that they may be represented by vectors of the same dimension. In this paper, we would like to port the eigentriphone modeling method from a phone-based system to a grapheme-based system; the new method will be called eigentrigrapheme modeling. Experiments on four official South African under-resourced languages (Afrikaans, South African English, Sesotho, siSwati) show that the new eigentrigrapheme modeling method reduces the word error rates of conventional tied-state trigrapheme modeling by an average of 4.08% relative

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Most existing vision-language pre-training (VLP) approaches adopt cross-modal masked language modeling (CMLM) to learn vision-language associations. However, we find that CMLM is insufficient for this purpose according to our observations: (1) Modality bias: a considerable amount of masked tokens in CMLM can be recovered with only the language information, ignoring the visual inputs. (2) Under-utilization of the unmasked tokens: CMLM primarily focuses on the masked token but it cannot simultaneously leverage other tokens to learn vision-language associations. To handle those limitations, we propose EPIC (lEveraging Per Image-Token Consistency for vision-language pre-training). In EPIC, for each image-sentence pair, we mask tokens that are salient to the image (i.e., Saliency-based Masking Strategy) and replace them with alternatives sampled from a language model (i.e., Inconsistent Token Generation Procedure), and then the model is required to determine for each token in the sentence whether they are consistent with the image (i.e., Image-Text Consistent Task). The proposed EPIC method is easily combined with pre-training methods. Extensive experiments show that the combination of the EPIC method and state-of-the-art pre-training approaches, including ViLT, ALBEF, METER, and X-VLM, leads to significant improvements on downstream tasks

A new type of slumping-induced soft-sediment deformation structure: the envelope structure

The sediments of the Cretaceous Gyeokpori Formation in south-western South Korea accumulated in a lake in which mainly siliciclastic rocks were deposited, with some interbedded volcaniclastics. The nearby volcanic activity resulted in unstable lake margins inducing a dominance of gravity-flow deposits. The high sedimentation rate facilitated soft-sediment deformation on the sloping margin. The deposition of numerous gravity-flow deposits resulted in a vertically heterolithic stratification. The slumps are composed of different lithologies, which is expressed in different types of deformation due to the difference in cohesion between sandy and mussy layers within the slumps. Coarser-grained (cohesionless) slumps tend to show more chaotic deformation of their lamination or layering. The difference in slumping behaviour of the cohesive and non-cohesive examples is explained and modelled. A unique soft-sediment deformation structure is recognized. This structure has not been described before, and we call it ‘envelope structure’. It consists of a conglomerate mass that has become entirely embedded in fine-grained sediment because slope failure took place and the fine-grained material slumped down with the conglomerate ‘at its back’. The cohesive laminated mudstone formed locally slump folds that embedded the non-cohesive overlying conglomerate unit, possibly partly due to the bulldozing effect of the latter. This structure presumably can develop when the density contrast with the underlying and overlying deposits is exceptionally high. The envelope structure should be regarded as a special – and rare – type of a slumping-induced deformation structure

GigaST: A 10,000-hour Pseudo Speech Translation Corpus

This paper introduces GigaST, a large-scale pseudo speech translation (ST) corpus. We create the corpus by translating the text in GigaSpeech, an English ASR corpus, into German and Chinese. The training set is translated by a strong machine translation system and the test set is translated by human. ST models trained with an addition of our corpus obtain new state-of-the-art results on the MuST-C English-German benchmark test set. We provide a detailed description of the translation process and verify its quality. We make the translated text data public and hope to facilitate research in speech translation. Additionally, we also release the training scripts on NeurST to make it easy to replicate our systems. GigaST dataset is available at https://st-benchmark.github.io/resources/GigaST.Comment: Submitted to Interspeech 2022. GigaST dataset is available at https://st-benchmark.github.io/resources/GigaS

From overtourism to undertourism... and back? The struggle to manage tourism regrowth in post-pandemic Amsterdam

Amsterdam is one of many cities that has struggled with problems of overtourism in recent years. These problems include nuisance, crowdedness, rising housing prices and economic dependence on tourism. City administrators were aware of these issues and took a variety of measures before the onset of the COVID-19 pandemic, such as placing restrictions on tourism rental (Airbnb) and setting up campaigns to tackle problem behaviour of tourists. Yet when COVID-19 halted the stream of tourists visiting Amsterdam, this created a unique opportunity to make more drastic changes, which the administration has addressed by proposing a new series of measures to proactively help contain tourism regrowth in the post-pandemic period. In this article we critically analyze these strategies to ascertain the extent to which they appear able to address the various issues of pre-pandemic overtourism to which they are aimed. Our analysis demonstrate that while trajectories already started before the pandemic are now being augmented and given more priority and some new strategies to further curb tourism growth are also being implemented, overall Amsterdam remains dominated by a growth-oriented approach to tourism planning