A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation

Virtual humans have gained considerable attention in numerous industries, e.g., entertainment and e-commerce. As a core technology, synthesizing photorealistic face frames from target speech and facial identity has been actively studied with generative adversarial networks. Despite remarkable results of modern talking-face generation models, they often entail high computational burdens, which limit their efficient deployment. This study aims to develop a lightweight model for speech-driven talking-face synthesis. We build a compact generator by removing the residual blocks and reducing the channel width from Wav2Lip, a popular talking-face generator. We also present a knowledge distillation scheme to stably yet effectively train the small-capacity generator without adversarial learning. We reduce the number of parameters and MACs by 28$\times$ while retaining the performance of the original model. Moreover, to alleviate a severe performance drop when converting the whole generator to INT8 precision, we adopt a selective quantization method that uses FP16 for the quantization-sensitive layers and INT8 for the other layers. Using this mixed precision, we achieve up to a 19$\times$ speedup on edge GPUs without noticeably compromising the generation quality.Comment: MLSys Workshop on On-Device Intelligence, 2023; Demo: https://huggingface.co/spaces/nota-ai/compressed_wav2li

Atomic-Scale Scanning of Domain Network in the Ferroelectric HfO2 Thin Film

Ferroelectric HfO2-based thin films have attracted much interest in the utilization of ferroelectricity at the nanoscale for next-generation electronic devices. However, the structural origin and stabilization mechanism of the ferroelectric phase are not understood because the film is typically nanocrystalline with active yet stochastic ferroelectric domains. Here, electron microscopy is used to map the in-plane domain network structures of epitaxially grown ferroelectric Y:HfO2 films in atomic resolution. The ferroelectricity is confirmed in free-standing Y:HfO2 films, allowing for investigating the structural origin for their ferroelectricity by 4D-STEM, high-resolution STEM, and iDPC-STEM. At the grain boundaries of <111>-oriented Pca21 orthorhombic grains, a high-symmetry mixed-(R3m, Pnm21) phase is induced, exhibiting enhanced polarization due to in-plane compressive strain. Nanoscale Pca21 orthorhombic grains and their grain boundaries with mixed-(R3m, Pnm21) phases of higher symmetry cooperatively determine the ferroelectricity of the Y:HfO2 film. It is also found that such ferroelectric domain networks emerge when the film thickness is beyond a finite value. Furthermore, in-plane mapping of oxygen positions overlaid on ferroelectric domains discloses that polarization is suppressed at vertical domain walls, while it is active when domains are aligned horizontally with subangstrom domain walls. In addition, randomly distributed 180° charged domain walls are confined by spacer layers

Machine Learning-Based Anomaly Detection on Seawater Temperature Data with Oversampling

This study deals with a method for anomaly detection in seawater temperature data using machine learning methods with oversampling techniques. Data were acquired from 2017 to 2023 using a Conductivity–Temperature–Depth (CTD) system in the Pacific Ocean, Indian Ocean, and Sea of Korea. The seawater temperature data consist of 1414 profiles including 1218 normal and 196 abnormal profiles. This dataset has an imbalance problem in which the amount of abnormal data is insufficient compared to that of normal data. Therefore, we generated abnormal data with oversampling techniques using duplication, uniform random variable, Synthetic Minority Oversampling Technique (SMOTE), and autoencoder (AE) techniques for the balance of data class, and trained Interquartile Range (IQR)-based, one-class support vector machine (OCSVM), and Multi-Layer Perceptron (MLP) models with a balanced dataset for anomaly detection. In the experimental results, the F1 score of the MLP showed the best performance at 0.882 in the combination of learning data, consisting of 30% of the minor data generated by SMOTE. This result is a 71.4%-point improvement over the F1 score of the IQR-based model, which is the baseline of this study, and is 1.3%-point better than the best-performing model among the models without oversampling data

Fabrication of Micro-Patterned Chip with Controlled Thickness for High-Throughput Cryogenic Electron Microscopy

© 2022 JoVE Journal of Visualized Experiments.A major limitation for the efficient and high-throughput structure analysis of biomolecules using cryogenic electron microscopy (cryo-EM) is the difficulty of preparing cryo-EM samples with controlled ice thickness at the nanoscale. The silicon (Si)-based chip, which has a regular array of micro-holes with graphene oxide (GO) window patterned on a thickness-controlled silicon nitride (SixNy) film, has been developed by applying microelectromechanical system (MEMS) techniques. UV photolithography, chemical vapor deposition, wet and dry etching of the thin film, and drop-casting of 2D nanosheet materials were used for mass-production of the micro-patterned chips with GO windows. The depth of the micro-holes is regulated to control the ice thickness on-demand, depending on the size of the specimen for cryo-EM analysis. The favorable affinity of GO toward biomolecules concentrates the biomolecules of interest within the micro-hole during cryo-EM sample preparation. The micro-patterned chip with GO windows enables high-throughput cryo-EM imaging of various biological molecules, as well as inorganic nanomaterials.11Nsciescopu

Coalescence dynamics of platinum group metal nanoparticles revealed by liquid-phase transmission electron microscopy

Summary: Coalescence, one of the major pathways observed in the growth of nanoparticles, affects the structural diversity of the synthesized nanoparticles in terms of sizes, shapes, and grain boundaries. As coalescence events occur transiently during the growth of nanoparticles and are associated with the interaction between nanoparticles, mechanistic understanding is challenging. The ideal platform to study coalescence events may require real-time tracking of nanoparticle growth trajectories with quantitative analysis for coalescence events. Herein, we track nanoparticle growth trajectories using liquid-cell transmission electron microscopy (LTEM) to investigate the role of coalescence in nanoparticle formation and their morphologies. By evaluating multiple coalescence events for different platinum group metals, we reveal that the surface energy and ligand binding energy determines the rate of the reshaping process and the resulting final morphology of coalesced nanoparticles. The coalescence mechanism, based on direct LTEM observation explains the structures of noble metal nanoparticles that emerge in colloidal synthesis

Multiple-length scale investigation of Pt/C degradation by identical-location transmission electron microscopy

Pt-based electrocatalysts on the cathode side of proton exchange membrane fuel cells (PEMFCs) generally undergo severe degradation, which contributes to the short life span of PEMFCs. Thus, it is crucial to understand the structural degradation of Pt-based electrocatalysts. Here, various degradation mechanisms of individual Pt nanoparticles supported on Vulcan carbon during load-cycle accelerated stress tests were investigated and quantified by identical-location transmission electron microscopy (IL-TEM). The atomic-scale IL-STEM imaging revealed the formation of Pt single atoms on the carbon support, which resulted from the dissolution of nanoparticles, and the following pathway change in the oxygen reduction reaction (ORR) was analyzed by rotating ring-disk electrode tests. Our study provides new insight for understanding the relationship between the decline in the ORR activity and the formation of Pt atomic species resulting from the electrochemical degradation of Pt/C.11Nsciescopuskc

Synthesis of metal cation doped nanoparticles for single atom alloy catalysts using spontaneous cation exchange

Ion exchange is a chemical reaction in which the ionic components of a solid parent material are exchanged with other ions. It often happens in metal chalcogenide and ionic metal oxide crystals, which are composed of cation and anion constituents. Here, we discovered that the cations in a solution spontaneously exchange with the constituent atoms in metal nanoparticles, forming cation-doped metal nanoparticles. Owing to charge-charge repulsion, the cations are atomically dispersed without aggregating in the nanoparticles that act as single-atom catalysts. The prepared PdRu and PdCe catalysts exhibited remarkable activity for methanol oxidation reaction and hydrogen evolution reaction, in which the Ru and Ce cations serve as active sites. This cation exchange reaction provides a specific tool to synthesize cation single atom catalysts in mild conditions that cannot be obtained via other synthetic methods and will spawn applications like photodynamic therapy, chemical sensing, and devices.11Nsciescopu

Complex ligand adsorption on 3D atomic surfaces of synthesized nanoparticles investigated by machine-learning accelerated ab initio calculation

Nanoparticle surfaces are passivated by surface-bound ligands, and their adsorption on synthesized nanoparticles is complicated because of the intricate and low-symmetry surface structures. Thus, it is challenging to precisely investigate ligand adsorption on synthesized nanoparticles. Here, we applied machine-learning-accelerated ab initio calculation to experimentally resolved 3D atomic structures of Pt nanoparticles to analyze the complex adsorption behavior of polyvinylpyrrolidone (PVP) ligands on synthesized nanoparticles. Different angular configurations of large-sized ligands are thoroughly investigated to understand the adsorption behavior on various surface-exposed atoms with intrinsic low-symmetry. It is revealed that the ligand binding energy (E-ads) of the large-sized ligand shows a weak positive relationship with the generalized coordination number((CN)) . This is because the strong positive relationship of short-range direct bonding (E-bind) is attenuated by the negative relationship of long-range van der Waals interaction (E-vdW). In addition, it is demonstrated that the PVP ligands prefer to adsorb where the long-range vdW interaction with the surrounding surface structure is maximized. Our results highlight the significant contribution of vdW interactions and the importance of the local geometry of surface atoms to the adsorption behavior of large-sized ligands on synthesized nanoparticle surfaces.11Nscopu