Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence

We present a novel architecture for dense correspondence. The current state-of-the-art are Transformer-based approaches that focus on either feature descriptors or cost volume aggregation. However, they generally aggregate one or the other but not both, though joint aggregation would boost each other by providing information that one has but other lacks, i.e., structural or semantic information of an image, or pixel-wise matching similarity. In this work, we propose a novel Transformer-based network that interleaves both forms of aggregations in a way that exploits their complementary information. Specifically, we design a self-attention layer that leverages the descriptor to disambiguate the noisy cost volume and that also utilizes the cost volume to aggregate features in a manner that promotes accurate matching. A subsequent cross-attention layer performs further aggregation conditioned on the descriptors of both images and aided by the aggregated outputs of earlier layers. We further boost the performance with hierarchical processing, in which coarser level aggregations guide those at finer levels. We evaluate the effectiveness of the proposed method on dense matching tasks and achieve state-of-the-art performance on all the major benchmarks. Extensive ablation studies are also provided to validate our design choices.Comment: v2 includes supplementary material, while v1 does no

Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation

This paper presents a novel cost aggregation network, called Volumetric Aggregation with Transformers (VAT), for few-shot segmentation. The use of transformers can benefit correlation map aggregation through self-attention over a global receptive field. However, the tokenization of a correlation map for transformer processing can be detrimental, because the discontinuity at token boundaries reduces the local context available near the token edges and decreases inductive bias. To address this problem, we propose a 4D Convolutional Swin Transformer, where a high-dimensional Swin Transformer is preceded by a series of small-kernel convolutions that impart local context to all pixels and introduce convolutional inductive bias. We additionally boost aggregation performance by applying transformers within a pyramidal structure, where aggregation at a coarser level guides aggregation at a finer level. Noise in the transformer output is then filtered in the subsequent decoder with the help of the query's appearance embedding. With this model, a new state-of-the-art is set for all the standard benchmarks in few-shot segmentation. It is shown that VAT attains state-of-the-art performance for semantic correspondence as well, where cost aggregation also plays a central role.Comment: Code and trained models are available at https://seokju-cho.github.io/VAT/ . This is ECCV'22 camera-ready version, which is revised from arXiv:2112.1168

MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation

We present a novel method for exemplar-based image translation, called matching interleaved diffusion models (MIDMs). Most existing methods for this task were formulated as GAN-based matching-then-generation framework. However, in this framework, matching errors induced by the difficulty of semantic matching across cross-domain, e.g., sketch and photo, can be easily propagated to the generation step, which in turn leads to degenerated results. Motivated by the recent success of diffusion models overcoming the shortcomings of GANs, we incorporate the diffusion models to overcome these limitations. Specifically, we formulate a diffusion-based matching-and-generation framework that interleaves cross-domain matching and diffusion steps in the latent space by iteratively feeding the intermediate warp into the noising process and denoising it to generate a translated image. In addition, to improve the reliability of the diffusion process, we design a confidence-aware process using cycle-consistency to consider only confident regions during translation. Experimental results show that our MIDMs generate more plausible images than state-of-the-art methods

D\"aRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation

Neural radiance fields (NeRF) shows powerful performance in novel view synthesis and 3D geometry reconstruction, but it suffers from critical performance degradation when the number of known viewpoints is drastically reduced. Existing works attempt to overcome this problem by employing external priors, but their success is limited to certain types of scenes or datasets. Employing monocular depth estimation (MDE) networks, pretrained on large-scale RGB-D datasets, with powerful generalization capability would be a key to solving this problem: however, using MDE in conjunction with NeRF comes with a new set of challenges due to various ambiguity problems exhibited by monocular depths. In this light, we propose a novel framework, dubbed D\"aRF, that achieves robust NeRF reconstruction with a handful of real-world images by combining the strengths of NeRF and monocular depth estimation through online complementary training. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints to enhance its robustness and coherence. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scale-shift fitting and geometry distillation, which adapts the MDE network to produce depths aligned accurately with NeRF geometry. Experiments show our framework achieves state-of-the-art results both quantitatively and qualitatively, demonstrating consistent and reliable performance in both indoor and outdoor real-world datasets. Project page is available at https://ku-cvlab.github.io/DaRF/.Comment: Project Page: https://ku-cvlab.github.io/DaRF

LANIT: Language-Driven Image-to-Image Translation for Unlabeled Data

Existing techniques for image-to-image translation commonly have suffered from two critical problems: heavy reliance on per-sample domain annotation and/or inability of handling multiple attributes per image. Recent truly-unsupervised methods adopt clustering approaches to easily provide per-sample one-hot domain labels. However, they cannot account for the real-world setting: one sample may have multiple attributes. In addition, the semantics of the clusters are not easily coupled to the human understanding. To overcome these, we present a LANguage-driven Image-to-image Translation model, dubbed LANIT. We leverage easy-to-obtain candidate attributes given in texts for a dataset: the similarity between images and attributes indicates per-sample domain labels. This formulation naturally enables multi-hot label so that users can specify the target domain with a set of attributes in language. To account for the case that the initial prompts are inaccurate, we also present prompt learning. We further present domain regularization loss that enforces translated images be mapped to the corresponding domain. Experiments on several standard benchmarks demonstrate that LANIT achieves comparable or superior performance to existing models.Comment: Accepted to CVPR 2023. Project Page: https://ku-cvlab.github.io/LANIT

DiffFace: Diffusion-based Face Swapping with Facial Guidance

In this paper, we propose a diffusion-based face swapping framework for the first time, called DiffFace, composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. In specific, in the training process, the ID conditional DDPM is trained to generate face images with the desired identity. In the sampling process, we use the off-the-shelf facial expert models to make the model transfer source identity while preserving target attributes faithfully. During this process, to preserve the background of the target image and obtain the desired face swapping result, we additionally propose a target-preserving blending strategy. It helps our model to keep the attributes of the target face from noise while transferring the source facial identity. In addition, without any re-training, our model can flexibly apply additional facial guidance and adaptively control the ID-attributes trade-off to achieve the desired results. To the best of our knowledge, this is the first approach that applies the diffusion model in face swapping task. Compared with previous GAN-based approaches, by taking advantage of the diffusion model for the face swapping task, DiffFace achieves better benefits such as training stability, high fidelity, diversity of the samples, and controllability. Extensive experiments show that our DiffFace is comparable or superior to the state-of-the-art methods on several standard face swapping benchmarks.Comment: Project Page: https://hxngiee.github.io/DiffFac

Piezo-Transmissive Structure Using a Multi-layered Heterogeneous Film for Optical Transmittance Modulation

As the damage caused by the recent climate crisis increases, efforts are being made to develop low-power and high-efficiency technologies to reduce pollution for energy production worldwide. Among them, research on the mechano-responsive optical transmittance modulation technology is being actively conducted as it can be applied to various application fields for reducing energy consumption: low-power sensors and smart windows. The piezo-transmittance structure, which is one of the optical transmittance modulation structures, has fewer constraints on the installation environment; therefore, many applications have been proposed. However, it is still challenging to fabricate a piezo-transmittance structure with a large-area production, high throughput, and good tunability because of complex curing and dissolution processes. Herein, we present an efficient fabrication method for a multi-layered piezo-transmittance structure using a large-area abrasive mold and thermal imprinting process. The piezo-transmittance performance (e.g., sensitivity and relative change of transmittance) shows temperature/humidity-independent characteristics and can be designed by tuning design parameters such as the number of layers, abrasive grade, and film material. Also, the surrogate model of the performance obtained from the Monte Carlo simulation and prediction model can offer tunability for various applications. Finally, we demonstrated two energy-efficient applications: the smart window integrated with a hydraulic pump showed high thermal efficiency in indoor environment control, and the telemetry system was demonstrated to measure pressure remotely

Piezo-Transmissive Structure Using a Multi-layered Heterogeneous Film for Optical Transmittance Modulation

As the damage caused by the recent climate crisis increases, efforts are being made to develop low-power and high-efficiency technologies to reduce pollution for energy production worldwide. Among them, research on the mechano-responsive optical transmittance modulation technology is being actively conducted as it can be applied to various application fields for reducing energy consumption: low-power sensors and smart windows. The piezo-transmittance structure, which is one of the optical transmittance modulation structures, has fewer constraints on the installation environment; therefore, many applications have been proposed. However, it is still challenging to fabricate a piezo-transmittance structure with a large-area production, high throughput, and good tunability because of complex curing and dissolution processes. Herein, we present an efficient fabrication method for a multi-layered piezo-transmittance structure using a large-area abrasive mold and thermal imprinting process. The piezo-transmittance performance (e.g., sensitivity and relative change of transmittance) shows temperature/humidity-independent characteristics and can be designed by tuning design parameters such as the number of layers, abrasive grade, and film material. Also, the surrogate model of the performance obtained from the Monte Carlo simulation and prediction model can offer tunability for various applications. Finally, we demonstrated two energy-efficient applications: the smart window integrated with a hydraulic pump showed high thermal efficiency in indoor environment control, and the telemetry system was demonstrated to measure pressure remotely

Piezo-Transmissive Structure Using a Multi-layered Heterogeneous Film for Optical Transmittance Modulation

As the damage caused by the recent climate crisis increases, efforts are being made to develop low-power and high-efficiency technologies to reduce pollution for energy production worldwide. Among them, research on the mechano-responsive optical transmittance modulation technology is being actively conducted as it can be applied to various application fields for reducing energy consumption: low-power sensors and smart windows. The piezo-transmittance structure, which is one of the optical transmittance modulation structures, has fewer constraints on the installation environment; therefore, many applications have been proposed. However, it is still challenging to fabricate a piezo-transmittance structure with a large-area production, high throughput, and good tunability because of complex curing and dissolution processes. Herein, we present an efficient fabrication method for a multi-layered piezo-transmittance structure using a large-area abrasive mold and thermal imprinting process. The piezo-transmittance performance (e.g., sensitivity and relative change of transmittance) shows temperature/humidity-independent characteristics and can be designed by tuning design parameters such as the number of layers, abrasive grade, and film material. Also, the surrogate model of the performance obtained from the Monte Carlo simulation and prediction model can offer tunability for various applications. Finally, we demonstrated two energy-efficient applications: the smart window integrated with a hydraulic pump showed high thermal efficiency in indoor environment control, and the telemetry system was demonstrated to measure pressure remotely