In this paper, we introduce our work of building a Streaming Multilingual
Speech Model (SM2), which can transcribe or translate multiple spoken languages
into texts of the target language. The backbone of SM2 is Transformer
Transducer, which has high streaming capability. Instead of human labeled
speech translation (ST) data, SM2 models are trained using weakly supervised
data generated by converting the transcriptions in speech recognition corpora
with a machine translation service. With 351 thousand hours of anonymized
speech training data from 25 languages, SM2 models achieve comparable or even
better ST quality than some recent popular large-scale non-streaming speech
models. More importantly, we show that SM2 has the truly zero-shot capability
when expanding to new target languages, yielding high quality ST results for
{source-speech, target-text} pairs that are not seen during training.Comment: submitted to ICASSP 202