48 research outputs found
SeamlessNeRF: Stitching Part NeRFs with Gradient Propagation
Neural Radiance Fields (NeRFs) have emerged as promising digital mediums of
3D objects and scenes, sparking a surge in research to extend the editing
capabilities in this domain. The task of seamless editing and merging of
multiple NeRFs, resembling the ``Poisson blending'' in 2D image editing,
remains a critical operation that is under-explored by existing work. To fill
this gap, we propose SeamlessNeRF, a novel approach for seamless appearance
blending of multiple NeRFs. In specific, we aim to optimize the appearance of a
target radiance field in order to harmonize its merge with a source field. We
propose a well-tailored optimization procedure for blending, which is
constrained by 1) pinning the radiance color in the intersecting boundary area
between the source and target fields and 2) maintaining the original gradient
of the target. Extensive experiments validate that our approach can effectively
propagate the source appearance from the boundary area to the entire target
field through the gradients. To the best of our knowledge, SeamlessNeRF is the
first work that introduces gradient-guided appearance editing to radiance
fields, offering solutions for seamless stitching of 3D objects represented in
NeRFs.Comment: To appear in SIGGRAPH Asia 2023. Project website is accessible at
https://sites.google.com/view/seamlessner
RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes
Radiance fields have gradually become a main representation of media.
Although its appearance editing has been studied, how to achieve
view-consistent recoloring in an efficient manner is still under explored. We
present RecolorNeRF, a novel user-friendly color editing approach for the
neural radiance fields. Our key idea is to decompose the scene into a set of
pure-colored layers, forming a palette. By this means, color manipulation can
be conducted by altering the color components of the palette directly. To
support efficient palette-based editing, the color of each layer needs to be as
representative as possible. In the end, the problem is formulated as an
optimization problem, where the layers and their blending weights are jointly
optimized with the NeRF itself. Extensive experiments show that our
jointly-optimized layer decomposition can be used against multiple backbones
and produce photo-realistic recolored novel-view renderings. We demonstrate
that RecolorNeRF outperforms baseline methods both quantitatively and
qualitatively for color editing even in complex real-world scenes.Comment: To appear in ACM Multimedia 2023. Project website is accessible at
https://sites.google.com/view/recolorner
ME-PCN: Point Completion Conditioned on Mask Emptiness
Point completion refers to completing the missing geometries of an object
from incomplete observations. Main-stream methods predict the missing shapes by
decoding a global feature learned from the input point cloud, which often leads
to deficient results in preserving topology consistency and surface details. In
this work, we present ME-PCN, a point completion network that leverages
`emptiness' in 3D shape space. Given a single depth scan, previous methods
often encode the occupied partial shapes while ignoring the empty regions (e.g.
holes) in depth maps. In contrast, we argue that these `emptiness' clues
indicate shape boundaries that can be used to improve topology representation
and detail granularity on surfaces. Specifically, our ME-PCN encodes both the
occupied point cloud and the neighboring `empty points'. It estimates
coarse-grained but complete and reasonable surface points in the first stage,
followed by a refinement stage to produce fine-grained surface details.
Comprehensive experiments verify that our ME-PCN presents better qualitative
and quantitative performance against the state-of-the-art. Besides, we further
prove that our `emptiness' design is lightweight and easy to embed in existing
methods, which shows consistent effectiveness in improving the CD and EMD
scores.Comment: Accepted to ICCV 2021; typos correcte
Diffusion Guided Domain Adaptation of Image Generators
Can a text-to-image diffusion model be used as a training objective for
adapting a GAN generator to another domain? In this paper, we show that the
classifier-free guidance can be leveraged as a critic and enable generators to
distill knowledge from large-scale text-to-image diffusion models. Generators
can be efficiently shifted into new domains indicated by text prompts without
access to groundtruth samples from target domains. We demonstrate the
effectiveness and controllability of our method through extensive experiments.
Although not trained to minimize CLIP loss, our model achieves equally high
CLIP scores and significantly lower FID than prior work on short prompts, and
outperforms the baseline qualitatively and quantitatively on long and
complicated prompts. To our best knowledge, the proposed method is the first
attempt at incorporating large-scale pre-trained diffusion models and
distillation sampling for text-driven image generator domain adaptation and
gives a quality previously beyond possible. Moreover, we extend our work to
3D-aware style-based generators and DreamBooth guidance.Comment: Project website: https://styleganfusion.github.io
How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs
This work focuses on the potential of Vision LLMs (VLLMs) in visual
reasoning. Different from prior studies, we shift our focus from evaluating
standard performance to introducing a comprehensive safety evaluation suite,
covering both out-of-distribution (OOD) generalization and adversarial
robustness. For the OOD evaluation, we present two novel VQA datasets, each
with one variant, designed to test model performance under challenging
conditions. In exploring adversarial robustness, we propose a straightforward
attack strategy for misleading VLLMs to produce visual-unrelated responses.
Moreover, we assess the efficacy of two jailbreaking strategies, targeting
either the vision or language component of VLLMs. Our evaluation of 21 diverse
models, ranging from open-source VLLMs to GPT-4V, yields interesting
observations: 1) Current VLLMs struggle with OOD texts but not images, unless
the visual information is limited; and 2) These VLLMs can be easily misled by
deceiving vision encoders only, and their vision-language training often
compromise safety protocols. We release this safety evaluation suite at
https://github.com/UCSC-VLAA/vllm-safety-benchmark.Comment: H.T., C.C., and Z.W. contribute equally. Work done during H.T. and
Z.W.'s internship at UCSC, and C.C. and Y.Z.'s internship at UN
Resistance to receptor-blocking therapies primes tumors as targets for HER3-homing nanobiologics
Resistance to anti-tumor therapeutics is an important clinical problem. Tumor-targeted therapies currently used in the clinic are derived from antibodies or small molecules that mitigate growth factor activity. These have improved therapeutic efficacy and safety compared to traditional treatment modalities but resistance arises in the majority of clinical cases. Targeting such resistance could improve tumor abatement and patient survival. A growing number of such tumors are characterized by prominent expression of the human epidermal growth factor receptor 3 (HER3) on the cell surface. This study presents a āTrojan-Horseā approach to combating these tumors by using a receptor-targeted biocarrier that exploits the HER3 cell surface protein as a portal to sneak therapeutics into tumor cells by mimicking an essential ligand. The biocarrier used here combines several functions within a single fusion protein for mediating targeted cell penetration and non-covalent self-assembly with therapeutic cargo, forming HER3-homing nanobiologics. Importantly, we demonstrate here that these nanobiologics are therapeutically effective in several scenarios of resistance to clinically approved targeted inhibitors of the human EGF receptor family. We also show that such inhibitors heighten efficacy of our nanobiologics on naĆÆve tumors by augmenting HER3 expression. This approach takes advantage of a current clinical problem (i.e. resistance to growth factor inhibition) and uses it to make tumors more susceptible to HER3 nanobiologic treatment. Moreover, we demonstrate a novel approach in addressing drug resistance by taking inhibitors against which resistance arises and re-introducing these as adjuvants, sensitizing tumors to the HER3 nanobiologics described here
Resistance to receptor-blocking therapies primes tumors as targets for HER3-homing nanobiologics
Resistance to anti-tumor therapeutics is an important clinical problem. Tumor-targeted therapies currently used in the clinic are derived from antibodies or small molecules that mitigate growth factor activity. These have improved therapeutic efficacy and safety compared to traditional treatment modalities but resistance arises in the majority of clinical cases. Targeting such resistance could improve tumor abatement and patient survival. A growing number of such tumors are characterized by prominent expression of the human epidermal growth factor receptor 3 (HER3) on the cell surface. This study presents a āTrojan-Horseā approach to combating these tumors by using a receptor-targeted biocarrier that exploits the HER3 cell surface protein as a portal to sneak therapeutics into tumor cells by mimicking an essential ligand. The biocarrier used here combines several functions within a single fusion protein for mediating targeted cell penetration and non-covalent self-assembly with therapeutic cargo, forming HER3-homing nanobiologics. Importantly, we demonstrate here that these nanobiologics are therapeutically effective in several scenarios of resistance to clinically approved targeted inhibitors of the human EGF receptor family. We also show that such inhibitors heighten efficacy of our nanobiologics on naĆÆve tumors by augmenting HER3 expression. This approach takes advantage of a current clinical problem (i.e. resistance to growth factor inhibition) and uses it to make tumors more susceptible to HER3 nanobiologic treatment. Moreover, we demonstrate a novel approach in addressing drug resistance by taking inhibitors against which resistance arises and re-introducing these as adjuvants, sensitizing tumors to the HER3 nanobiologics described here
Sidewall profile reconstruction of microstructures with high aspect ratio based on near-infrared light scanning interferometry
Sidewall profile reconstruction of microstructures with the high aspect ratio is a problem urgently to be solved in MEMS field. In this paper, a measuring method based on near-infrared light scanning interferometry (NILSI) is presented according to the transmission principle of semiconductor materials in the infrared light region. The NILSI is extended from the white light to near-infrared light and from surface profile reconstruction to sidewall profile reconstruction. The NILSI system is constituted by a near-infrared light source, an interference microscope, infrared CCD, piezoelectric ceramics (PZT) with high accuracy and the data acquisition system. The test sample is taken from GaAs microstructures with high aspect ratio and made by two different height steps for measuring with different typical testing equipment. Near-infrared light vertical scanning interference (NILVSI) is improved to compensate optical path difference (OPD) and the large surface roughness. The sidewall profile of the sample is obtained and compared with that of scanning electron microscopy (SEM) and white light scanning interferometry (WLSI). Test results demonstrate that the steps have 2.115 Ī¼m and 0.762 Ī¼m relative heights and 1.34 % and 2.14% relative errors respectively. There is a good agreement with the results of SEM and WLSI. The system can reconstruct the sidewall profile of microstructures with high aspect ratio