SeamlessNeRF: Stitching Part NeRFs with Gradient Propagation

Neural Radiance Fields (NeRFs) have emerged as promising digital mediums of 3D objects and scenes, sparking a surge in research to extend the editing capabilities in this domain. The task of seamless editing and merging of multiple NeRFs, resembling the ``Poisson blending'' in 2D image editing, remains a critical operation that is under-explored by existing work. To fill this gap, we propose SeamlessNeRF, a novel approach for seamless appearance blending of multiple NeRFs. In specific, we aim to optimize the appearance of a target radiance field in order to harmonize its merge with a source field. We propose a well-tailored optimization procedure for blending, which is constrained by 1) pinning the radiance color in the intersecting boundary area between the source and target fields and 2) maintaining the original gradient of the target. Extensive experiments validate that our approach can effectively propagate the source appearance from the boundary area to the entire target field through the gradients. To the best of our knowledge, SeamlessNeRF is the first work that introduces gradient-guided appearance editing to radiance fields, offering solutions for seamless stitching of 3D objects represented in NeRFs.Comment: To appear in SIGGRAPH Asia 2023. Project website is accessible at https://sites.google.com/view/seamlessner

RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes

Radiance fields have gradually become a main representation of media. Although its appearance editing has been studied, how to achieve view-consistent recoloring in an efficient manner is still under explored. We present RecolorNeRF, a novel user-friendly color editing approach for the neural radiance fields. Our key idea is to decompose the scene into a set of pure-colored layers, forming a palette. By this means, color manipulation can be conducted by altering the color components of the palette directly. To support efficient palette-based editing, the color of each layer needs to be as representative as possible. In the end, the problem is formulated as an optimization problem, where the layers and their blending weights are jointly optimized with the NeRF itself. Extensive experiments show that our jointly-optimized layer decomposition can be used against multiple backbones and produce photo-realistic recolored novel-view renderings. We demonstrate that RecolorNeRF outperforms baseline methods both quantitatively and qualitatively for color editing even in complex real-world scenes.Comment: To appear in ACM Multimedia 2023. Project website is accessible at https://sites.google.com/view/recolorner

ME-PCN: Point Completion Conditioned on Mask Emptiness

Point completion refers to completing the missing geometries of an object from incomplete observations. Main-stream methods predict the missing shapes by decoding a global feature learned from the input point cloud, which often leads to deficient results in preserving topology consistency and surface details. In this work, we present ME-PCN, a point completion network that leverages `emptiness' in 3D shape space. Given a single depth scan, previous methods often encode the occupied partial shapes while ignoring the empty regions (e.g. holes) in depth maps. In contrast, we argue that these `emptiness' clues indicate shape boundaries that can be used to improve topology representation and detail granularity on surfaces. Specifically, our ME-PCN encodes both the occupied point cloud and the neighboring `empty points'. It estimates coarse-grained but complete and reasonable surface points in the first stage, followed by a refinement stage to produce fine-grained surface details. Comprehensive experiments verify that our ME-PCN presents better qualitative and quantitative performance against the state-of-the-art. Besides, we further prove that our `emptiness' design is lightweight and easy to embed in existing methods, which shows consistent effectiveness in improving the CD and EMD scores.Comment: Accepted to ICCV 2021; typos correcte

Diffusion Guided Domain Adaptation of Image Generators

Can a text-to-image diffusion model be used as a training objective for adapting a GAN generator to another domain? In this paper, we show that the classifier-free guidance can be leveraged as a critic and enable generators to distill knowledge from large-scale text-to-image diffusion models. Generators can be efficiently shifted into new domains indicated by text prompts without access to groundtruth samples from target domains. We demonstrate the effectiveness and controllability of our method through extensive experiments. Although not trained to minimize CLIP loss, our model achieves equally high CLIP scores and significantly lower FID than prior work on short prompts, and outperforms the baseline qualitatively and quantitatively on long and complicated prompts. To our best knowledge, the proposed method is the first attempt at incorporating large-scale pre-trained diffusion models and distillation sampling for text-driven image generator domain adaptation and gives a quality previously beyond possible. Moreover, we extend our work to 3D-aware style-based generators and DreamBooth guidance.Comment: Project website: https://styleganfusion.github.io

How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs

This work focuses on the potential of Vision LLMs (VLLMs) in visual reasoning. Different from prior studies, we shift our focus from evaluating standard performance to introducing a comprehensive safety evaluation suite, covering both out-of-distribution (OOD) generalization and adversarial robustness. For the OOD evaluation, we present two novel VQA datasets, each with one variant, designed to test model performance under challenging conditions. In exploring adversarial robustness, we propose a straightforward attack strategy for misleading VLLMs to produce visual-unrelated responses. Moreover, we assess the efficacy of two jailbreaking strategies, targeting either the vision or language component of VLLMs. Our evaluation of 21 diverse models, ranging from open-source VLLMs to GPT-4V, yields interesting observations: 1) Current VLLMs struggle with OOD texts but not images, unless the visual information is limited; and 2) These VLLMs can be easily misled by deceiving vision encoders only, and their vision-language training often compromise safety protocols. We release this safety evaluation suite at https://github.com/UCSC-VLAA/vllm-safety-benchmark.Comment: H.T., C.C., and Z.W. contribute equally. Work done during H.T. and Z.W.'s internship at UCSC, and C.C. and Y.Z.'s internship at UN

Resistance to receptor-blocking therapies primes tumors as targets for HER3-homing nanobiologics

Resistance to anti-tumor therapeutics is an important clinical problem. Tumor-targeted therapies currently used in the clinic are derived from antibodies or small molecules that mitigate growth factor activity. These have improved therapeutic efficacy and safety compared to traditional treatment modalities but resistance arises in the majority of clinical cases. Targeting such resistance could improve tumor abatement and patient survival. A growing number of such tumors are characterized by prominent expression of the human epidermal growth factor receptor 3 (HER3) on the cell surface. This study presents a “Trojan-Horse” approach to combating these tumors by using a receptor-targeted biocarrier that exploits the HER3 cell surface protein as a portal to sneak therapeutics into tumor cells by mimicking an essential ligand. The biocarrier used here combines several functions within a single fusion protein for mediating targeted cell penetration and non-covalent self-assembly with therapeutic cargo, forming HER3-homing nanobiologics. Importantly, we demonstrate here that these nanobiologics are therapeutically effective in several scenarios of resistance to clinically approved targeted inhibitors of the human EGF receptor family. We also show that such inhibitors heighten efficacy of our nanobiologics on naïve tumors by augmenting HER3 expression. This approach takes advantage of a current clinical problem (i.e. resistance to growth factor inhibition) and uses it to make tumors more susceptible to HER3 nanobiologic treatment. Moreover, we demonstrate a novel approach in addressing drug resistance by taking inhibitors against which resistance arises and re-introducing these as adjuvants, sensitizing tumors to the HER3 nanobiologics described here

Resistance to receptor-blocking therapies primes tumors as targets for HER3-homing nanobiologics

Resistance to anti-tumor therapeutics is an important clinical problem. Tumor-targeted therapies currently used in the clinic are derived from antibodies or small molecules that mitigate growth factor activity. These have improved therapeutic efficacy and safety compared to traditional treatment modalities but resistance arises in the majority of clinical cases. Targeting such resistance could improve tumor abatement and patient survival. A growing number of such tumors are characterized by prominent expression of the human epidermal growth factor receptor 3 (HER3) on the cell surface. This study presents a “Trojan-Horse” approach to combating these tumors by using a receptor-targeted biocarrier that exploits the HER3 cell surface protein as a portal to sneak therapeutics into tumor cells by mimicking an essential ligand. The biocarrier used here combines several functions within a single fusion protein for mediating targeted cell penetration and non-covalent self-assembly with therapeutic cargo, forming HER3-homing nanobiologics. Importantly, we demonstrate here that these nanobiologics are therapeutically effective in several scenarios of resistance to clinically approved targeted inhibitors of the human EGF receptor family. We also show that such inhibitors heighten efficacy of our nanobiologics on naïve tumors by augmenting HER3 expression. This approach takes advantage of a current clinical problem (i.e. resistance to growth factor inhibition) and uses it to make tumors more susceptible to HER3 nanobiologic treatment. Moreover, we demonstrate a novel approach in addressing drug resistance by taking inhibitors against which resistance arises and re-introducing these as adjuvants, sensitizing tumors to the HER3 nanobiologics described here

Sidewall profile reconstruction of microstructures with high aspect ratio based on near-infrared light scanning interferometry

Sidewall profile reconstruction of microstructures with the high aspect ratio is a problem urgently to be solved in MEMS field. In this paper, a measuring method based on near-infrared light scanning interferometry (NILSI) is presented according to the transmission principle of semiconductor materials in the infrared light region. The NILSI is extended from the white light to near-infrared light and from surface profile reconstruction to sidewall profile reconstruction. The NILSI system is constituted by a near-infrared light source, an interference microscope, infrared CCD, piezoelectric ceramics (PZT) with high accuracy and the data acquisition system. The test sample is taken from GaAs microstructures with high aspect ratio and made by two different height steps for measuring with different typical testing equipment. Near-infrared light vertical scanning interference (NILVSI) is improved to compensate optical path difference (OPD) and the large surface roughness. The sidewall profile of the sample is obtained and compared with that of scanning electron microscopy (SEM) and white light scanning interferometry (WLSI). Test results demonstrate that the steps have 2.115 μm and 0.762 μm relative heights and 1.34 % and 2.14% relative errors respectively. There is a good agreement with the results of SEM and WLSI. The system can reconstruct the sidewall profile of microstructures with high aspect ratio