ReliTalk: Relightable Talking Portrait Generation from a Single Video

Recent years have witnessed great progress in creating vivid audio-driven portraits from monocular videos. However, how to seamlessly adapt the created video avatars to other scenarios with different backgrounds and lighting conditions remains unsolved. On the other hand, existing relighting studies mostly rely on dynamically lighted or multi-view data, which are too expensive for creating video portraits. To bridge this gap, we propose ReliTalk, a novel framework for relightable audio-driven talking portrait generation from monocular videos. Our key insight is to decompose the portrait's reflectance from implicitly learned audio-driven facial normals and images. Specifically, we involve 3D facial priors derived from audio features to predict delicate normal maps through implicit functions. These initially predicted normals then take a crucial part in reflectance decomposition by dynamically estimating the lighting condition of the given video. Moreover, the stereoscopic face representation is refined using the identity-consistent loss under simulated multiple lighting conditions, addressing the ill-posed problem caused by limited views available from a single monocular video. Extensive experiments validate the superiority of our proposed framework on both real and synthetic datasets. Our code is released in https://github.com/arthur-qiu/ReliTalk

Construction of a prognostic model of lung adenocarcinoma based on machine learning

In order to more accurately predict the prognosis and survival of lung adenocarcinoma patients, this paper used the gene expression and clinical information data of lung adenocarcinoma patients in the open database of TCGA to jointly construct a prognosis model of lung adenocarcinoma. Three difference analysis methods and univariate cox regression analysis were used as the preliminary screening method. By comparing the variable selection ability of lasso regression and random survival forest, comparing the performance of cox proportional risk regression model and random survival forest model, and integrating clinical data, a model that can more accurately predict the prognosis of lung adenocarcinoma patients was constructed. After comparison and selection, lasso regression was used to select variables and cox proportional risk model was used as the prediction model. The consistency index of the model reached 0.712. The AUC for 1-year, 3-year and 5-year survival of lung adenocarcinoma patients in the validation set were 0.808, 0.816 and 0.754, respectively. After the fusion of clinical data, the 1-year, 3-year and 5-year survival prediction AUC in the validation set were 0.840, 0.836 and 0.865, respectively, indicating that the model had good predictive performance

Evolution of emergent monopoles into magnetic skyrmion strings

Topological defects are fundamental concepts in physics, but little is known about the transition between distinct types across different dimensionalities. In topological magnetism, as in field theory, the transition between 1D strings and 0D monopoles is a key process whose observation has remained elusive. Here, we introduce a novel mechanism that allows for the controlled stabilization of emergent monopoles and show that magnetic skyrmion strings can be folded into monopoles. Conversely, they act as seeds out of which the entire string structure can unfold, containing its complete information. In chiral magnets, this process can be observed by resonant elastic X-ray scattering when the objects are in proximity to a polarized ferromagnet, whereby a pure monopole lattice is emerging on the surface. Our experimental proof of the reversible evolution from monopole to string sheds new light on topological defects and establishes the emergent monopole lattice as a new 3D topological phase

Detection of streptavidin using liquid crystal based whispering gallery mode microbubble

Protein is a complex chemical substance essential for human survival. Traditional protein detection methods, such as colorimetry, electrochemical analysis, and enzyme-linked immunosorbent assays, have shown good specificity and accuracy for the protein detection. However, all these methods require specialized instruments, and the detection procedures are laborious and time-consuming. As a result, a rapid, sensitive, label-free protein detection method is urgently needed. Herein, we have developed an ultra-sensitive biosensor for the detection of low-concentration protein molecules, employing liquid crystal (LC)-amplified optofluidic resonator. Since the orientations of LCs highly depend on the surface biomolecular binding processes, LCs can be employed to realize the extremely sensitive detection of biomolecules. Immobilized protein molecules interfere with the orientation of LCs by reducing the vertical anchoring force from the alignment layer in which the spectral wavelength shift was monitored as a sensing parameter. A biosensing platform based on an LC-amplified optofluidic whispering gallery mode (WGM) resonator was designed and studied accordingly. Due to the simultaneous interaction of the WGM and the LCs in the optofluidic resonator, the changes caused by the injection of protein molecules will be amplified, resulting in a shift in the resonance wavelength. Total wavelength shifts scale proportionally to the molecular concentrations of the protein within a certain range. The detection limit for streptavidin (SA) can reach as low as the femtometer level, which is significantly higher than the detection limit in the classic polarized optical microscope (POM) method visible with the naked eye. In addition to SA, the LC-based optofluidic resonator can also be applied to detect a variety of protein molecules. Our study demonstrates that LC-amplified optofluidic resonator can provide a novel solution for ultrasensitive real-time characterization of biosensing and biomolecular interactions

Model-enhanced Vector Index

Embedding-based retrieval methods construct vector indices to search for document representations that are most similar to the query representations. They are widely used in document retrieval due to low latency and decent recall performance. Recent research indicates that deep retrieval solutions offer better model quality, but are hindered by unacceptable serving latency and the inability to support document updates. In this paper, we aim to enhance the vector index with end-to-end deep generative models, leveraging the differentiable advantages of deep retrieval models while maintaining desirable serving efficiency. We propose Model-enhanced Vector Index (MEVI), a differentiable model-enhanced index empowered by a twin-tower representation model. MEVI leverages a Residual Quantization (RQ) codebook to bridge the sequence-to-sequence deep retrieval and embedding-based models. To substantially reduce the inference time, instead of decoding the unique document ids in long sequential steps, we first generate some semantic virtual cluster ids of candidate documents in a small number of steps, and then leverage the well-adapted embedding vectors to further perform a fine-grained search for the relevant documents in the candidate virtual clusters. We empirically show that our model achieves better performance on the commonly used academic benchmarks MSMARCO Passage and Natural Questions, with comparable serving latency to dense retrieval solutions

Detection of DNA hybridization using liquid crystal based whispering gallery mode microbubble

DNA detection based on DNA hybridization has wide applications in various fields, such as clinical diagnostics, food safety, and environmental monitoring, etc. At present, common DNA detection approaches include fluorescence-based microarray technique, electrochemical method and surface plasmon resonance (SPR), etc. However, these methods require additional precise equipments and relatively complicated detection process. In this work, we developed a biosensor platform based on a liquid crystal (LC)-amplified optofluidic whispering gallery mode (WGM) resonator to achieve ultra-sensitive, label-free, and quick-response DNA hybridization detection. Liquid crystal is a material with high sensitivity, rapid response, and low cost. It exhibits significant directional and positional ordering, and it is sensitive and responsive to external stimuli. LC molecules exhibit a uniform orientation on the surface of the resonator, when the surface is covered with an appropriate amount of ssDNA. Once complementary DNA and ssDNA hybridize on the surface, the homotropic orientation of the LCs will be destroyed. Due to the simultaneous interaction of the WGM and the LCs in the optofluidic resonator, changes caused by the DNA hybridization can be amplified, resulting in a shift in the resonance wavelength. In this experiment, we used the spectral wavelength shift as a sensing parameter to achieve the detection of target DNA, and a lower detection limit compared to traditional DNA detection methods was obtained. At the same time, this biosensor platform also shows good selectivity. Our research results suggest that the LC based WGM optical microcavity sensing platform can provide an ultra-sensitive, label free solution for DNA detection

Label-free protein quantitation using liquid crystal-enhanced optofluidic biosensor

Protein detection plays an important role in the medical research. Liquid crystals (LCs), as a class of sensitive materials, exhibit a promising ability in the biosensing field. Herein, we exploited an ultrasensitive biosensor for protein detection, employing the whispering-gallery-mode (WGM) from the LC-amplified optofluidic micro-resonator. The biomolecules can trigger both light-matter interactions and the orientation transitions of LC molecules. The WGM spectral wavelength shift was recorded as the sensing indicator, and a detection limit of 15 fM for bovine serum albumin (BSA) was achieved. Our LC-amplified optofluidic biosensor provides a new solution for the ultrasensitive, real-time, and stable biological detection

PyPose: A Library for Robot Learning with Physics-based Optimization

Deep learning has had remarkable success in robotic perception, but its data-centric nature suffers when it comes to generalizing to ever-changing environments. By contrast, physics-based optimization generalizes better, but it does not perform as well in complicated tasks due to the lack of high-level semantic information and the reliance on manual parametric tuning. To take advantage of these two complementary worlds, we present PyPose: a robotics-oriented, PyTorch-based library that combines deep perceptual models with physics-based optimization techniques. Our design goal for PyPose is to make it user-friendly, efficient, and interpretable with a tidy and well-organized architecture. Using an imperative style interface, it can be easily integrated into real-world robotic applications. Besides, it supports parallel computing of any order gradients of Lie groups and Lie algebras and $2^{\text{nd}}$-order optimizers, such as trust region methods. Experiments show that PyPose achieves 3-20$\times$ speedup in computation compared to state-of-the-art libraries. To boost future research, we provide concrete examples across several fields of robotics, including SLAM, inertial navigation, planning, and control

PyPose v0.6: The Imperative Programming Interface for Robotics

PyPose is an open-source library for robot learning. It combines a learning-based approach with physics-based optimization, which enables seamless end-to-end robot learning. It has been used in many tasks due to its meticulously designed application programming interface (API) and efficient implementation. From its initial launch in early 2022, PyPose has experienced significant enhancements, incorporating a wide variety of new features into its platform. To satisfy the growing demand for understanding and utilizing the library and reduce the learning curve of new users, we present the fundamental design principle of the imperative programming interface, and showcase the flexible usage of diverse functionalities and modules using an extremely simple Dubins car example. We also demonstrate that the PyPose can be easily used to navigate a real quadruped robot with a few lines of code