Controllable Motion Diffusion Model

Generating realistic and controllable motions for virtual characters is a challenging task in computer animation, and its implications extend to games, simulations, and virtual reality. Recent studies have drawn inspiration from the success of diffusion models in image generation, demonstrating the potential for addressing this task. However, the majority of these studies have been limited to offline applications that target at sequence-level generation that generates all steps simultaneously. To enable real-time motion synthesis with diffusion models in response to time-varying control signals, we propose the framework of the Controllable Motion Diffusion Model (COMODO). Our framework begins with an auto-regressive motion diffusion model (A-MDM), which generates motion sequences step by step. In this way, simply using the standard DDPM algorithm without any additional complexity, our framework is able to generate high-fidelity motion sequences over extended periods with different types of control signals. Then, we propose our reinforcement learning-based controller and controlling strategies on top of the A-MDM model, so that our framework can steer the motion synthesis process across multiple tasks, including target reaching, joystick-based control, goal-oriented control, and trajectory following. The proposed framework enables the real-time generation of diverse motions that react adaptively to user commands on-the-fly, thereby enhancing the overall user experience. Besides, it is compatible with the inpainting-based editing methods and can predict much more diverse motions without additional fine-tuning of the basic motion generation models. We conduct comprehensive experiments to evaluate the effectiveness of our framework in performing various tasks and compare its performance against state-of-the-art methods

Cinematic Behavior Transfer via NeRF-based Differentiable Filming

In the evolving landscape of digital media and video production, the precise manipulation and reproduction of visual elements like camera movements and character actions are highly desired. Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections, neglecting 3D statuses. To address these issues, we first introduce a reverse filming behavior estimation technique. It optimizes camera trajectories by leveraging NeRF as a differentiable renderer and refining SMPL tracks. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment. The incorporation of 3D engine workflow enables superior rendering and control abilities, which also achieves a higher rating in the user study.Comment: Project Page: https://virtualfilmstudio.github.io/projects/cinetransfe

Unified Human-Scene Interaction via Prompted Chain-of-Contacts

Human-Scene Interaction (HSI) is a vital component of fields like embodied AI and virtual reality. Despite advancements in motion quality and physical plausibility, two pivotal factors, versatile interaction control and the development of a user-friendly interface, require further exploration before the practical application of HSI. This paper presents a unified HSI framework, UniHSI, which supports unified control of diverse interactions through language commands. This framework is built upon the definition of interaction as Chain of Contacts (CoC): steps of human joint-object part pairs, which is inspired by the strong correlation between interaction types and human-object contact regions. Based on the definition, UniHSI constitutes a Large Language Model (LLM) Planner to translate language prompts into task plans in the form of CoC, and a Unified Controller that turns CoC into uniform task execution. To facilitate training and evaluation, we collect a new dataset named ScenePlan that encompasses thousands of task plans generated by LLMs based on diverse scenarios. Comprehensive experiments demonstrate the effectiveness of our framework in versatile task execution and generalizability to real scanned scenes. The project page is at https://github.com/OpenRobotLab/UniHSI .Comment: A unified Human-Scene Interaction framework that supports versatile interactions through language commands.Project URL: https://xizaoqu.github.io/unihsi/ . Code: https://github.com/OpenRobotLab/UniHS

Multi-Job Intelligent Scheduling with Cross-Device Federated Learning

Recent years have witnessed a large amount of decentralized data in various (edge) devices of end-users, while the decentralized data aggregation remains complicated for machine learning jobs because of regulations and laws. As a practical approach to handling decentralized data, Federated Learning (FL) enables collaborative global machine learning model training without sharing sensitive raw data. The servers schedule devices to jobs within the training process of FL. In contrast, device scheduling with multiple jobs in FL remains a critical and open problem. In this paper, we propose a novel multi-job FL framework, which enables the training process of multiple jobs in parallel. The multi-job FL framework is composed of a system model and a scheduling method. The system model enables a parallel training process of multiple jobs, with a cost model based on the data fairness and the training time of diverse devices during the parallel training process. We propose a novel intelligent scheduling approach based on multiple scheduling methods, including an original reinforcement learning-based scheduling method and an original Bayesian optimization-based scheduling method, which corresponds to a small cost while scheduling devices to multiple jobs. We conduct extensive experimentation with diverse jobs and datasets. The experimental results reveal that our proposed approaches significantly outperform baseline approaches in terms of training time (up to 12.73 times faster) and accuracy (up to 46.4% higher).Comment: To appear in TPDS; 22 pages, 17 figures, 8 tables. arXiv admin note: substantial text overlap with arXiv:2112.0592

Flux density measurements for 32 pulsars in the 20 cm band

Flux density measurements provide fundamental observational parameters that describe a pulsar. In the current pulsar catalogue, 27% of radio pulsars have no flux density measurement in the 20 cm observing band. Here, we present the first measurements of the flux densities in this band for 32 pulsars observed using the Parkes radio telescope and provide updated pulse profiles for these pulsars. We have used both archival and new observations to make these measurements. Various schemes exist for measuring flux densities. We show how the flux densities measured vary between these methods and how the presence of radio-frequency-interference will bias flux density measurementsComment: Accepted by RA

FAK Is a Critical Regulator of Neuroblastoma Liver Metastasis

Neuroblastomas express increased levels of gastrin-releasing peptide receptor (GRP-R). However, the exact molecular mechanisms involved in GRP-R-mediated cell signaling in neuroblastoma growth and metastasis are unknown. Here, we report that focal adhesion kinase (FAK), as a critical downstream target of GRP-R, is an important regulator of neuroblastoma tumorigenicity. We found that FAK expression correlates with GRP-R expression in human neuroblastoma sections and cell lines. GRP-R overexpression in SK-N-SH cells increased FAK, integrin α3 and β1 expressions and cell migration. These cells demonstrated flatter cell morphology with broad lamellae, in which intense FAK expression was localized to the leading edges of lamellipodia. Interestingly, FAK activation was, in part, dependent on integrin α3 and β1 expression. Conversely, GRP-R silencing decreased FAK as well as Mycn levels in BE(2)-C cells, which displayed a denser cellular morphology. Importantly, rescue experiments in GRP-R silenced BE(2)-C cells showed FAK overexpression significantly enhanced cell viability and soft agar colony formation; similarly, FAK overexpression in SK-N-SH cells also resulted in increased cell growth. These effects were reversed in FAK silenced BE(2)-C cells in vitro as well as in vivo. Moreover, we evaluated the effect of FAK inhibition in vivo. FAK inhibitor (Y15) suppressed GRP-induced neuroblastoma growth and metastasis. Our results indicate that FAK is a critical downstream regulator of GRP-R, which mediates tumorigenesis and metastasis in neuroblastoma

Probing the Emission States of PSR J1107−5907

The emission from PSR J1107−5907 is erratic. Sometimes the radio pulse is undetectable, at other times the pulsed emission is weak, and for short durations the emission can be very bright. In order to improve our understanding of these state changes, we have identified archival data sets from the Parkes radio telescope in which the bright emission is present, and find that the emission never switches from the bright state to the weak state, but instead always transitions to the "off" state. Previous work had suggested the identification of the "off" state as an extreme manifestation of the weak state. However, the connection between the "off" and bright emission reported here suggests that the emission can be interpreted as undergoing only two emission states: a "bursting" state consisting of both bright pulses and nulls, and the weak emission state