560 research outputs found
Deep Reflectance Maps
Undoing the image formation process and therefore decomposing appearance into
its intrinsic properties is a challenging task due to the under-constraint
nature of this inverse problem. While significant progress has been made on
inferring shape, materials and illumination from images only, progress in an
unconstrained setting is still limited. We propose a convolutional neural
architecture to estimate reflectance maps of specular materials in natural
lighting conditions. We achieve this in an end-to-end learning formulation that
directly predicts a reflectance map from the image itself. We show how to
improve estimates by facilitating additional supervision in an indirect scheme
that first predicts surface orientation and afterwards predicts the reflectance
map by a learning-based sparse data interpolation.
In order to analyze performance on this difficult task, we propose a new
challenge of Specular MAterials on SHapes with complex IllumiNation (SMASHINg)
using both synthetic and real images. Furthermore, we show the application of
our method to a range of image-based editing tasks on real images.Comment: project page: http://homes.esat.kuleuven.be/~krematas/DRM
Deep Learning Methods for Calibrated Photometric Stereo and Beyond
Photometric stereo recovers the surface normals of an object from multiple
images with varying shading cues, i.e., modeling the relationship between
surface orientation and intensity at each pixel. Photometric stereo prevails in
superior per-pixel resolution and fine reconstruction details. However, it is a
complicated problem because of the non-linear relationship caused by
non-Lambertian surface reflectance. Recently, various deep learning methods
have shown a powerful ability in the context of photometric stereo against
non-Lambertian surfaces. This paper provides a comprehensive review of existing
deep learning-based calibrated photometric stereo methods. We first analyze
these methods from different perspectives, including input processing,
supervision, and network architecture. We summarize the performance of deep
learning photometric stereo models on the most widely-used benchmark data set.
This demonstrates the advanced performance of deep learning-based photometric
stereo methods. Finally, we give suggestions and propose future research trends
based on the limitations of existing models.Comment: 19 pages, 11 figures, 4 table
Metappearance: Meta-Learning for Visual Appearance Reproduction
There currently are two main approaches to reproducing visual appearance
using Machine Learning (ML): The first is training models that generalize over
different instances of a problem, e.g., different images from a dataset. Such
models learn priors over the data corpus and use this knowledge to provide fast
inference with little input, often as a one-shot operation. However, this
generality comes at the cost of fidelity, as such methods often struggle to
achieve the final quality required. The second approach does not train a model
that generalizes across the data, but overfits to a single instance of a
problem, e.g., a flash image of a material. This produces detailed and
high-quality results, but requires time-consuming training and is, as mere
non-linear function fitting, unable to exploit previous experience. Techniques
such as fine-tuning or auto-decoders combine both approaches but are sequential
and rely on per-exemplar optimization. We suggest to combine both techniques
end-to-end using meta-learning: We over-fit onto a single problem instance in
an inner loop, while also learning how to do so efficiently in an outer-loop
that builds intuition over many optimization runs. We demonstrate this concept
to be versatile and efficient, applying it to RGB textures, Bi-directional
Reflectance Distribution Functions (BRDFs), or Spatially-varying BRDFs
(svBRDFs)
Research relative to angular distribution of snow reflectance/snow cover characterization and microwave emission
Remote sensing has been applied in recent years to monitoring snow cover properties for applications in hydrologic and energy balance modeling. In addition, snow cover has been recently shown to exert a considerable local influence on weather variables. Of particular importance is the potential of sensors to provide data on the physical properties of snow with high spatial and temporal resolution. Visible and near-infrared measurements of upwelling radiance can be used to infer near-surface properties through the calculation of albedo. Microwave signals usually come from deeper within the snow pack and thus provide depth-integrated information, which can be measured through clouds and does not relay on solar illumination.Fundamental studies examining the influence of snow properties on signals from various parts of the electromagnetic spectrum continue in part because of the promise of new remote sensors with higher spectral and spatial accuracy. Information in the visible and near-infrared parts of the spectrum comprise nearly all available data with high spatial resolution. Current passive microwave sensors have poor spatial resolution and the data are problematic where the scenes consist of mixed landscape features, but they offer timely observations that are independent of cloud cover and solar illumination
μκ³΅κ° ν΄μλ ν₯μμ ν΅ν μμ λ³ν λͺ¨λν°λ§
νμλ
Όλ¬Έ(λ°μ¬) -- μμΈλνκ΅λνμ : νκ²½λνμ νλκ³Όμ μ‘°κ²½ν, 2023. 2. λ₯μλ ¬.μ‘μ μνκ³μμ λκΈ°κΆκ³Ό μλ¬ΌκΆμ μνΈ μμ©μ μ΄ν΄νκΈ° μν΄μλ μμ λ³νμ λͺ¨λν°λ§μ΄ νμνλ€. μ΄ λ, μμ±μμμ μ§νλ©΄μ κ΄μΈ‘νμ¬ μμμ§λλ₯Ό μ 곡ν μ μμ§λ§, μ§νλ³νμ μμΈν μ 보λ ꡬλ¦μ΄λ μμ± μ΄λ―Έμ§μ κ³΅κ° ν΄μλμ μν΄ μ νλμλ€. λν μμ±μμμ μκ³΅κ° ν΄μλκ° μμμ§λλ₯Ό ν΅ν κ΄ν©μ± λͺ¨λν°λ§μ λ―ΈμΉλ μν₯μ μμ ν λ°νμ§μ§ μμλ€.
λ³Έ λ
Όλ¬Έμμλ κ³ ν΄μλ μμ μ§λλ₯Ό μΌλ¨μλ‘ μμ±νκΈ° μμ± μμμ μκ³΅κ° ν΄μλλ₯Ό ν₯μμν€λ κ²μ λͺ©νλ‘ νμλ€. κ³ ν΄μλ μμ±μμμ νμ©ν μμ λ³ν λͺ¨λν°λ§μ μ곡κ°μ μΌλ‘ νμ₯νκΈ° μν΄ 1) μ μ§κΆ€λ μμ±μ νμ©ν μμμ΅ν©μ ν΅ν΄ μκ°ν΄μλ ν₯μ, 2) μ λμ μμ±λ€νΈμν¬λ₯Ό νμ©ν 곡κ°ν΄μλ ν₯μ, 3) μ곡κ°ν΄μλκ° λμ μμ±μμμ ν μ§νΌλ³΅μ΄ κ· μ§νμ§ μμ 곡κ°μμ μλ¬Ό κ΄ν©μ± λͺ¨λν°λ§μ μννμλ€. μ΄μ²λΌ, μμ±κΈ°λ° μ격νμ§μμ μλ‘μ΄ κΈ°μ μ΄ λ±μ₯ν¨μ λ°λΌ νμ¬ λ° κ³Όκ±°μ μμ±μμμ μκ³΅κ° ν΄μλ μΈ‘λ©΄μμ ν₯μλμ΄ μμ λ³νμ λͺ¨λν°λ§ ν μ μλ€.
μ 2μ₯μμλ μ μ§κΆ€λμμ±μμμ νμ©νλ μκ³΅κ° μμμ΅ν©μΌλ‘ μλ¬Όμ κ΄ν©μ±μ λͺ¨λν°λ§ νμ λ, μκ°ν΄μλκ° ν₯μλ¨μ 보μλ€. μκ³΅κ° μμμ΅ν© μ, ꡬλ¦νμ§, μλ°©ν₯ λ°μ¬ ν¨μ μ‘°μ , κ³΅κ° λ±λ‘, μκ³΅κ° μ΅ν©, μκ³΅κ° κ²°μΈ‘μΉ λ³΄μ λ±μ κ³Όμ μ κ±°μΉλ€. μ΄ μμμ΅ν© μ°μΆλ¬Όμ κ²½μκ΄λ¦¬ λ±μΌλ‘ μμ μ§μμ μ°κ° λ³λμ΄ ν° λ μ₯μ(λκ²½μ§μ λμ½μλ¦Ό)μμ νκ°νμλ€. κ·Έ κ²°κ³Ό, μκ³΅κ° μμμ΅ν© μ°μΆλ¬Όμ κ²°μΈ‘μΉ μμ΄ νμ₯κ΄μΈ‘μ μμΈ‘νμλ€ (R2 = 0.71, μλ νΈν₯ = 5.64% λκ²½μ§; R2 = 0.79, μλ νΈν₯ = -13.8%, νμ½μλ¦Ό). μκ³΅κ° μμμ΅ν©μ μμ μ§λμ μκ³΅κ° ν΄μλλ₯Ό μ μ§μ μΌλ‘ κ°μ νμ¬, μλ¬Ό μμ₯κΈ°λμ μμ±μμμ΄ νμ₯ κ΄μΈ‘μ κ³Όμ νκ°λ₯Ό μ€μλ€. μμμ΅ν©μ λμ μκ³΅κ° ν΄μλλ‘ κ΄ν©μ± μ§λλ₯Ό μΌκ°κ²©μΌλ‘ μμ±νκΈ°μ μ΄λ₯Ό νμ©νμ¬ μμ± μμμ μ νλ μκ³΅κ° ν΄μλλ‘ λ°νμ§μ§ μμ μλ¬Όλ³νμ κ³Όμ μ λ°κ²¬νκΈΈ κΈ°λνλ€.
μμμ 곡κ°λΆν¬μ μ λ°λμ
κ³Ό ν μ§ νΌλ³΅ λ³ν λͺ¨λν°λ§μ μν΄ νμμ μ΄λ€. κ³ ν΄μλ μμ±μμμΌλ‘ μ§κ΅¬ νλ©΄μ κ΄μΈ‘νλ κ²μ μ©μ΄νκ² ν΄μ‘λ€. νΉν Planet Fusionμ μ΄μνμμ±κ΅° λ°μ΄ν°λ₯Ό μ΅λν νμ©ν΄ λ°μ΄ν° κ²°μΈ‘μ΄ μλ 3m κ³΅κ° ν΄μλμ μ§ν νλ©΄ λ°μ¬λμ΄λ€. κ·Έλ¬λ κ³Όκ±° μμ± μΌμ(Landsatμ κ²½μ° 30~60m)μ κ³΅κ° ν΄μλλ μμμ 곡κ°μ λ³νλ₯Ό μμΈ λΆμνλ κ²μ μ ννλ€. μ 3μ₯μμλ Landsat λ°μ΄ν°μ κ³΅κ° ν΄μλλ₯Ό ν₯μνκΈ° μν΄ Planet Fusion λ° Landsat 8 λ°μ΄ν°λ₯Ό μ¬μ©νμ¬ μ΄μ€ μ λμ μμ± λ€νΈμν¬(the dual RSS-GAN)λ₯Ό νμ΅μμΌ, κ³ ν΄μλ μ κ·ν μμ μ§μ(NDVI)μ μλ¬Ό κ·Όμ μΈμ λ°μ¬(NIRv)λλ₯Ό μμ±νλ νλ€. νμκΈ°λ° νμ₯ μμμ§μ(μ΅λ 8λ
)μ λλ‘ κΈ°λ° μ΄λΆκ΄μ§λλ‘ the dual RSS-GANμ μ±λ₯μ λνλ―Όκ΅ λ΄ λ λμμ§(λκ²½μ§μ νμ½μλ¦Ό)μμ νκ°νλ€. The dual RSS-GANμ Landsat 8 μμμ 곡κ°ν΄μλλ₯Ό ν₯μμμΌ κ³΅κ° ννμ 보μνκ³ μμ μ§μμ κ³μ μ λ³νλ₯Ό ν¬μ°©νλ€(R2> 0.96). κ·Έλ¦¬κ³ the dual RSS-GANμ Landsat 8 μμ μ§μκ° νμ₯μ λΉν΄ κ³Όμ νκ°λλ κ²μ μννλ€. νμ₯ κ΄μΈ‘μ λΉν΄ μ΄μ€ RSS-GANκ³Ό Landsat 8μ μλ νΈν₯ κ° κ°κ° -0.8% μμ -1.5%, -10.3% μμ -4.6% μλ€. μ΄λ¬ν κ°μ μ Planet Fusionμ 곡κ°μ 보λ₯Ό μ΄μ€ RSS-GANλ‘ νμ΅νμκΈ°μ κ°λ₯νλ€. ν€λΉ μ°κ΅¬ κ²°κ³Όλ Landsat μμμ κ³΅κ° ν΄μλλ₯Ό ν₯μμμΌ μ¨κ²¨μ§ κ³΅κ° μ 보λ₯Ό μ 곡νλ μλ‘μ΄ μ κ·Ό λ°©μμ΄λ€.
κ³ ν΄μλμμ μλ¬Ό κ΄ν©μ± μ§λλ ν μ§νΌλ³΅μ΄ 볡μ‘ν 곡κ°μμ νμ μν λͺ¨λν°λ§μ νμμ μ΄λ€. κ·Έλ¬λ Sentinel-2, Landsat λ° MODISμ κ°μ΄ νμ λμ‘° κΆ€λμ μλ μμ±μ κ³΅κ° ν΄μλκ° λκ±°λ μκ° ν΄μλ λμ μμ±μμλ§ μ 곡ν μ μλ€. μ΅κ·Ό λ°μ¬λ μ΄μνμμ±κ΅°μ μ΄λ¬ν ν΄μλ νκ³μ 극볡ν μ μλ€. νΉν Planet Fusionμ μ΄μνμμ± μλ£μ μκ³΅κ° ν΄μλλ‘ μ§νλ©΄μ κ΄μΈ‘ν μ μλ€. 4μ₯μμ, Planet Fusion μ§νλ°μ¬λλ₯Ό μ΄μ©νμ¬ μμμμ λ°μ¬λ κ·Όμ μΈμ 볡μ¬(NIRvP)λ₯Ό 3m ν΄μλ μ§λλ₯Ό μΌκ°κ²©μΌλ‘ μμ±νλ€. κ·Έλ° λ€μ λ―Έκ΅ μΊλ¦¬ν¬λμμ£Ό μν¬λΌλ©ν -μ νΈμν¨ λΈνμ νλμ€ νμ λ€νΈμν¬ λ°μ΄ν°μ λΉκ΅νμ¬ μλ¬Ό κ΄ν©μ±μ μΆμ νκΈ° μν NIRvP μ§λμ μ±λ₯μ νκ°νμλ€. μ 체μ μΌλ‘ NIRvP μ§λλ μ΅μ§μ μ¦μ μμ λ³νμλ λΆκ΅¬νκ³ κ°λ³ λμμ§μ μλ¬Ό κ΄ν©μ±μ μκ°μ λ³νλ₯Ό ν¬μ°©νμλ€. κ·Έλ¬λ λμμ§ μ 체μ λν NIRvP μ§λμ μλ¬Ό κ΄ν©μ± μ¬μ΄μ κ΄κ³λ NIRvP μ§λλ₯Ό νλμ€ νμ κ΄μΈ‘λ²μμ μΌμΉμν¬ λλ§ λμ μκ΄κ΄κ³λ₯Ό 보μλ€. κ΄μΈ‘λ²μλ₯Ό μΌμΉμν¬ κ²½μ°, NIRvP μ§λλ μλ¬Ό κ΄ν©μ±μ μΆμ νλ λ° μμ΄ νμ₯ NIRvPλ³΄λ€ μ°μν μ±λ₯μ 보μλ€. μ΄λ¬ν μ±λ₯ μ°¨μ΄λ νλμ€ νμ κ΄μΈ‘λ²μλ₯Ό μΌμΉμν¬ λ, μ°κ΅¬ λμμ§ κ°μ NIRvP-μλ¬Ό κ΄ν©μ± κ΄κ³μ κΈ°μΈκΈ°κ° μΌκ΄μ±μ 보μκΈ° λλ¬Έμ΄λ€. λ³Έ μ°κ΅¬ κ²°κ³Όλ μμ± κ΄μΈ‘μ νλμ€ νμ κ΄μΈ‘λ²μμ μΌμΉμν€λ κ²μ μ€μμ±μ 보μ¬μ£Όκ³ λμ μκ³΅κ° ν΄μλλ‘ μλ¬Ό κ΄ν©μ±μ μ격μΌλ‘ λͺ¨λν°λ§νλ μ΄μνμμ±κ΅° μλ£μ μ μ¬λ ₯μ 보μ¬μ€λ€.Monitoring changes in terrestrial vegetation is essential to understanding interactions between atmosphere and biosphere, especially terrestrial ecosystem. To this end, satellite remote sensing offer maps for examining land surface in different scales. However, the detailed information was hindered under the clouds or limited by the spatial resolution of satellite imagery. Moreover, the impacts of spatial and temporal resolution in photosynthesis monitoring were not fully revealed.
In this dissertation, I aimed to enhance the spatial and temporal resolution of satellite imagery towards daily gap-free vegetation maps with high spatial resolution. In order to expand vegetation change monitoring in time and space using high-resolution satellite images, I 1) improved temporal resolution of satellite dataset through image fusion using geostationary satellites, 2) improved spatial resolution of satellite dataset using generative adversarial networks, and 3) showed the use of high spatiotemporal resolution maps for monitoring plant photosynthesis especially over heterogeneous landscapes. With the advent of new techniques in satellite remote sensing, current and past datasets can be fully utilized for monitoring vegetation changes in the respect of spatial and temporal resolution.
In Chapter 2, I developed the integrated system that implemented geostationary satellite products in the spatiotemporal image fusion method for monitoring canopy photosynthesis. The integrated system contains the series of process (i.e., cloud masking, nadir bidirectional reflectance function adjustment, spatial registration, spatiotemporal image fusion, spatial gap-filling, temporal-gap-filling). I conducted the evaluation of the integrated system over heterogeneous rice paddy landscape where the drastic land cover changes were caused by cultivation management and deciduous forest where consecutive changes occurred in time. The results showed that the integrated system well predict in situ measurements without data gaps (R2 = 0.71, relative bias = 5.64% at rice paddy site; R2 = 0.79, relative bias = -13.8% at deciduous forest site). The integrated system gradually improved the spatiotemporal resolution of vegetation maps, reducing the underestimation of in situ measurements, especially during peak growing season. Since the integrated system generates daily canopy photosynthesis maps for monitoring dynamics among regions of interest worldwide with high spatial resolution. I anticipate future efforts to reveal the hindered information by the limited spatial and temporal resolution of satellite imagery.
Detailed spatial representations of terrestrial vegetation are essential for precision agricultural applications and the monitoring of land cover changes in heterogeneous landscapes. The advent of satellite-based remote sensing has facilitated daily observations of the Earths surface with high spatial resolution. In particular, a data fusion product such as Planet Fusion has realized the delivery of daily, gap-free surface reflectance data with 3-m pixel resolution through full utilization of relatively recent (i.e., 2018-) CubeSat constellation data. However, the spatial resolution of past satellite sensors (i.e., 30β60 m for Landsat) has restricted the detailed spatial analysis of past changes in vegetation. In Chapter 3, to overcome the spatial resolution constraint of Landsat data for long-term vegetation monitoring, we propose a dual remote-sensing super-resolution generative adversarial network (dual RSS-GAN) combining Planet Fusion and Landsat 8 data to simulate spatially enhanced long-term time-series of the normalized difference vegetation index (NDVI) and near-infrared reflectance from vegetation (NIRv). We evaluated the performance of the dual RSS-GAN against in situ tower-based continuous measurements (up to 8 years) and remotely piloted aerial system-based maps of cropland and deciduous forest in the Republic of Korea. The dual RSS-GAN enhanced spatial representations in Landsat 8 images and captured seasonal variation in vegetation indices (R2 > 0.95, for the dual RSS-GAN maps vs. in situ data from all sites). Overall, the dual RSS-GAN reduced Landsat 8 vegetation index underestimations compared with in situ measurements; relative bias values of NDVI ranged from β3.2% to 1.2% and β12.4% to β3.7% for the dual RSS-GAN and Landsat 8, respectively. This improvement was caused by spatial enhancement through the dual RSS-GAN, which captured fine-scale information from Planet Fusion. This study presents a new approach for the restoration of hidden sub-pixel spatial information in Landsat images.
Mapping canopy photosynthesis in both high spatial and temporal resolution is essential for carbon cycle monitoring in heterogeneous areas. However, well established satellites in sun-synchronous orbits such as Sentinel-2, Landsat and MODIS can only provide either high spatial or high temporal resolution but not both. Recently established CubeSat satellite constellations have created an opportunity to overcome this resolution trade-off. In particular, Planet Fusion allows full utilization of the CubeSat data resolution and coverage while maintaining high radiometric quality. In Chapter 4, I used the Planet Fusion surface reflectance product to calculate daily, 3-m resolution, gap-free maps of the near-infrared radiation reflected from vegetation (NIRvP). I then evaluated the performance of these NIRvP maps for estimating canopy photosynthesis by comparing with data from a flux tower network in Sacramento-San Joaquin Delta, California, USA. Overall, NIRvP maps captured temporal variations in canopy photosynthesis of individual sites, despite changes in water extent in the wetlands and frequent mowing in the crop fields. When combining data from all sites, however, I found that robust agreement between NIRvP maps and canopy photosynthesis could only be achieved when matching NIRvP maps to the flux tower footprints. In this case of matched footprints, NIRvP maps showed considerably better performance than in situ NIRvP in estimating canopy photosynthesis both for daily sum and data around the time of satellite overpass (R2 = 0.78 vs. 0.60, for maps vs. in situ for the satellite overpass time case). This difference in performance was mostly due to the higher degree of consistency in slopes of NIRvP-canopy photosynthesis relationships across the study sites for flux tower footprint-matched maps. Our results show the importance of matching satellite observations to the flux tower footprint and demonstrate the potential of CubeSat constellation imagery to monitor canopy photosynthesis remotely at high spatio-temporal resolution.Chapter 1. Introduction 2
1. Background 2
1.1 Daily gap-free surface reflectance using geostationary satellite products 2
1.2 Monitoring past vegetation changes with high-spatial-resolution 3
1.3 High spatiotemporal resolution vegetation photosynthesis maps 4
2. Purpose of Research 4
Chapter 2. Generating daily gap-filled BRDF adjusted surface reflectance product at 10 m resolution using geostationary satellite product for monitoring daily canopy photosynthesis 6
1. Introduction 6
2. Methods 11
2.1 Study sites 11
2.2 In situ measurements 13
2.3 Satellite products 14
2.4 Integrated system 17
2.5 Canopy photosynthesis 21
2.6 Evaluation 23
3. Results and discussion 24
3.1 Comparison of STIF NDVI and NIRv with in situ NDVI and NIRv 24
3.2 Comparison of STIF NIRvP with in situ NIRvP 28
4. Conclusion 31
Chapter 3. Super-resolution of historic Landsat imagery using a dual Generative Adversarial Network (GAN) model with CubeSat constellation imagery for monitoring vegetation changes 32
1. Introduction 32
2. Methods 38
2.1 Real-ESRGAN model 38
2.2 Study sites 40
2.3 In situ measurements 42
2.4 Vegetation index 44
2.5 Satellite data 45
2.6 Planet Fusion 48
2.7 Dual RSS-GAN via fine-tuned Real-ESRGAN 49
2.8 Evaluation 54
3. Results 57
3.1 Comparison of NDVI and NIRv maps from Planet Fusion, Sentinel 2 NBAR, and Landsat 8 NBAR data with in situ NDVI and NIRv 57
3.2 Comparison of dual RSS-SRGAN model results with Landsat 8 NDVI and NIRv 60
3.3 Comparison of dual RSS-GAN model results with respect to in situ time-series NDVI and NIRv 63
3.4 Comparison of the dual RSS-GAN model with NDVI and NIRv maps derived from RPAS 66
4. Discussion 70
4.1 Monitoring changes in terrestrial vegetation using the dual RSS-GAN model 70
4.2 CubeSat data in the dual RSS-GAN model 72
4.3 Perspectives and limitations 73
5. Conclusion 78
Appendices 79
Supplementary material 82
Chapter 4. Matching high resolution satellite data and flux tower footprints improves their agreement in photosynthesis estimates 85
1. Introduction 85
2. Methods 89
2.1 Study sites 89
2.2 In situ measurements 92
2.3 Planet Fusion NIRvP 94
2.4 Flux footprint model 98
2.5 Evaluation 98
3. Results 105
3.1 Comparison of Planet Fusion NIRv and NIRvP with in situ NIRv and NIRvP 105
3.2 Comparison of instantaneous Planet Fusion NIRv and NIRvP with against tower GPP estimates 108
3.3 Daily GPP estimation from Planet Fusion -derived NIRvP 114
4. Discussion 118
4.1 Flux tower footprint matching and effects of spatial and temporal resolution on GPP estimation 118
4.2 Roles of radiation component in GPP mapping 123
4.3 Limitations and perspectives 126
5. Conclusion 133
Appendix 135
Supplementary Materials 144
Chapter 5. Conclusion 153
Bibliography 155
Abstract in Korea 199
Acknowledgements 202λ°
- β¦