Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models

Recent work in speech-to-speech translation (S2ST) has focused primarily on offline settings, where the full input utterance is available before any output is given. This, however, is not reasonable in many real-world scenarios. In latency-sensitive applications, rather than waiting for the full utterance, translations should be spoken as soon as the information in the input is present. In this work, we introduce a system for simultaneous S2ST targeting real-world use cases. Our system supports translation from 57 languages to English with tunable parameters for dynamically adjusting the latency of the output -- including four policies for determining when to speak an output sequence. We show that these policies achieve offline-level accuracy with minimal increases in latency over a Greedy (wait-$k$) baseline. We open-source our evaluation code and interactive test script to aid future SimulS2ST research and application development.Comment: To appear at INTERSPEECH 202

AdaptNet: Policy Adaptation for Physics-Based Character Control

Motivated by humans' ability to adapt skills in the learning of new ones, this paper presents AdaptNet, an approach for modifying the latent space of existing policies to allow new behaviors to be quickly learned from like tasks in comparison to learning from scratch. Building on top of a given reinforcement learning controller, AdaptNet uses a two-tier hierarchy that augments the original state embedding to support modest changes in a behavior and further modifies the policy network layers to make more substantive changes. The technique is shown to be effective for adapting existing physics-based controllers to a wide range of new styles for locomotion, new task targets, changes in character morphology and extensive changes in environment. Furthermore, it exhibits significant increase in learning efficiency, as indicated by greatly reduced training times when compared to training from scratch or using other approaches that modify existing policies. Code is available at https://motion-lab.github.io/AdaptNet.Comment: SIGGRAPH Asia 2023. Video: https://youtu.be/WxmJSCNFb28. Website: https://motion-lab.github.io/AdaptNet, https://pei-xu.github.io/AdaptNe

Motion capture-driven simulations that hit and react

Ph.D.Jessica Hodgin

Motion Filtering with Dynamics

This paper describes a technique for filtering and modifying human data using dynamic constraints and simulation. A simple control system calculates torques to track human motion data for a dynamic simulation of an articulated figure. The simulation generates smooth, physically plausible motion that maintains characteristics of the original. This technique is used to filter data for a variety of upper-body motions, animating two models with differing kinematic parameters and degrees of freedom. In addition, the system creates transitions from one motion sequence to another in the spirit of previous techniques. Further, the system imposes task and environmental constraints to generate believable behaviors and dynamic contacts

Solving Computer Animation Problems with Numeric Optimization

Optimization techniques have been used in computer animation to search for system parameters and control inputs in a variety of animated objects and characters. Researchers rely on numeric optimization to solve problems including behavior learning and morphology generation for characters as well as automatic tuning for weights and parameters in various models. However, even with the diverse group of published examples, the selection of an optimization technique for new problems can be difficult for inexperienced animators. The choice of appropriate methods and their proper implementation requires an understanding of the types of methods and their respective advantages and limitations. Toward this end, I describe a general approach for formulating an optimization problem to help organize the information pertinent to the selection process and provide a common vocabulary for discussing the issues related to this type of problem-solving. I provide a straightforward classification of optimization methods and discuss characteristics and trade-offs related to the algorithms. Then, I describe specific uses of the methods with results from recent works in computer animation. I detail solutions for two common optimization problems namely, inverse kinematics and control gain tuning, and make general recommendations about solving optimization problems in computer animation in closing

A Dynamics-based Comparison Metric for Motion Graphs

Motion graph approaches focus on the re-use of character animation contained in a motion-capture repository by connecting similar frames in the database with transitions. Because the output animation of a motion graph comes directly from the motion capture data except for the transitions, the quality of the motion depends largely on the transition points selected. In this paper, we investigate comparison metrics for choosing transition points, aiming at improving the visual quality of animations generated using motion graphs. Specifically, we focus on the weight assigned to each body part, which reflects the relative significance of the body part on the quality of the generated motion. We introduce a novel weighting scheme, based on an estimation of the character’s dynamics, which assigns weights for each body according to displaced mass and simplified friction terms. To assess the quality of the transitions selected by our proposed dynamic metric, we compare its results to previous methods, looking at both visual quality and quantitative analysis

Automatic splicing for hand and body animations

We propose a solution to a new problem in animation research: how to use human motion capture data to create character motion with detailed hand gesticulation without the need for the simultaneous capture of hands and the full body. Occlusion and a difference i