RADIO: Reference-Agnostic Dubbing Video Synthesis

One of the most challenging problems in audio-driven talking head generation is achieving high-fidelity detail while ensuring precise synchronization. Given only a single reference image, extracting meaningful identity attributes becomes even more challenging, often causing the network to mirror the facial and lip structures too closely. To address these issues, we introduce RADIO, a framework engineered to yield high-quality dubbed videos regardless of the pose or expression in reference images. The key is to modulate the decoder layers using latent space composed of audio and reference features. Additionally, we incorporate ViT blocks into the decoder to emphasize high-fidelity details, especially in the lip region. Our experimental results demonstrate that RADIO displays high synchronization without the loss of fidelity. Especially in harsh scenarios where the reference frame deviates significantly from the ground truth, our method outperforms state-of-the-art methods, highlighting its robustness. Pre-trained model and codes will be made public after the review.Comment: Under revie

Improving the Expressiveness of Deep Learning Frameworks with Recursion

Recursive neural networks have widely been used by researchers to handle applications with recursively or hierarchically structured data. However, embedded control flow deep learning frameworks such as TensorFlow, Theano, Caffe2, and MXNet fail to efficiently represent and execute such neural networks, due to lack of support for recursion. In this paper, we add recursion to the programming model of existing frameworks by complementing their design with recursive execution of dataflow graphs as well as additional APIs for recursive definitions. Unlike iterative implementations, which can only understand the topological index of each node in recursive data structures, our recursive implementation is able to exploit the recursive relationships between nodes for efficient execution based on parallel computation. We present an implementation on TensorFlow and evaluation results with various recursive neural network models, showing that our recursive implementation not only conveys the recursive nature of recursive neural networks better than other implementations, but also uses given resources more effectively to reduce training and inference time.Comment: Appeared in EuroSys 2018. 13 pages, 11 figure

Behavioral analysis of Pacific abalone, Haliotis discus hannai, reveals its feeding preference and attraction potential for brown alga, Sargassum horneri

The Pacific abalone, Haliotis discus hannai, is a highly valued and industrially important aquaculture species with growing demands of the expanding abalone aquaculture industry. To explore the feasibility of using the brown alga, Sargassum horneri, as a potential substitute for abalone feed, it is important to identify the feed preference and attractant effect of S. horneri on Pacific abalone. Our experiments indicated that the feeding-associated movement of abalone could be detected using a video tracking system under indirect illumination with dim red light. To further analyze the attraction potentials of various test materials, preference analysis was performed using Avicel-coated glass plates with ground powders of various seaweeds (e.g., S. horneri, Saccharina japonica, and Undaria pinnatifida) and commercial abalone feed, together with coffee waste. Heat map analysis indicated greater attraction by the kelp S. japonica than by S. horneri and commercial feed, which showed similar preference levels. Feeding preference based on the area of Avicel eaten by abalone showed a significant preference for U. pinnatifida over S. horneri (feeding area: 68.6 ± 20.1% vs. 37.5 ± 22.4%, p < 0.05). Additionally, the feeding area was significantly greater for plates with S. ja-ponica than for plates with S. horneri (44.0 ± 16.6% vs. 22.6 ± 15.4%, p < 0.05). There was no significant difference in feeding area between commercial feed and S. horneri (31.7 ± 11.6% vs. 31.6 ± 20.2%, p > 0.05). The methanol extracts attracted abalone in the following order: U. pinnatifida > S. horneri > S. japonica > commercial feed > coffee waste. To determine the attractive effects of the components of methanol extracts, mixtures of methanol extracts of commercial feed with increasing amounts of S. horneri were examined. The results showed a significant increase in feeding preference upon addition of S. horneri up to 50% and 75%, suggesting its potential for use as an appetite-enhancing feed additive. This study identified conditions that can be successfully used to monitor the movement of Pacific abalone; the results of preference analysis confirmed that abalone exhibited similar attraction and feeding preference for S. horneri, compared with commercial feed