Structure Preserving Large Imagery Reconstruction

With the explosive growth of web-based cameras and mobile devices, billions of photographs are uploaded to the internet. We can trivially collect a huge number of photo streams for various goals, such as image clustering, 3D scene reconstruction, and other big data applications. However, such tasks are not easy due to the fact the retrieved photos can have large variations in their view perspectives, resolutions, lighting, noises, and distortions. Fur-thermore, with the occlusion of unexpected objects like people, vehicles, it is even more challenging to find feature correspondences and reconstruct re-alistic scenes. In this paper, we propose a structure-based image completion algorithm for object removal that produces visually plausible content with consistent structure and scene texture. We use an edge matching technique to infer the potential structure of the unknown region. Driven by the estimated structure, texture synthesis is performed automatically along the estimated curves. We evaluate the proposed method on different types of images: from highly structured indoor environment to natural scenes. Our experimental results demonstrate satisfactory performance that can be potentially used for subsequent big data processing, such as image localization, object retrieval, and scene reconstruction. Our experiments show that this approach achieves favorable results that outperform existing state-of-the-art techniques

Automatic Objects Removal for Scene Completion

With the explosive growth of web-based cameras and mobile devices, billions of photographs are uploaded to the internet. We can trivially collect a huge number of photo streams for various goals, such as 3D scene reconstruction and other big data applications. However, this is not an easy task due to the fact the retrieved photos are neither aligned nor calibrated. Furthermore, with the occlusion of unexpected foreground objects like people, vehicles, it is even more challenging to find feature correspondences and reconstruct realistic scenes. In this paper, we propose a structure based image completion algorithm for object removal that produces visually plausible content with consistent structure and scene texture. We use an edge matching technique to infer the potential structure of the unknown region. Driven by the estimated structure, texture synthesis is performed automatically along the estimated curves. We evaluate the proposed method on different types of images: from highly structured indoor environment to the natural scenes. Our experimental results demonstrate satisfactory performance that can be potentially used for subsequent big data processing: 3D scene reconstruction and location recognition.Comment: 6 pages, IEEE International Conference on Computer Communications (INFOCOM 14), Workshop on Security and Privacy in Big Data, Toronto, Canada, 201

A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation

Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluating scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.Comment: Includes supplementary materia

Explaining holistic image regressors and classifiers in urban analytics with plausible counterfactuals

We propose a new form of plausible counterfactual explanation designed to explain the behaviour of computer vision systems used in urban analytics that make predictions based on properties across the entire image, rather than specific regions of it. We illustrate the merits of our approach by explaining computer vision models used to analyse street imagery, which are now widely used in GeoAI and urban analytics. Such explanations are important in urban analytics as researchers and practioners are increasingly reliant on it for decision making. Finally, we perform a user study that demonstrate our approach can be used by non-expert users, who might not be machine learning experts, to be more confident and to better understand the behaviour of image-based classifiers/regressors for street view analysis. Furthermore, the method can potentially be used as an engagement tool to visualise how public spaces can plausibly look like. The limited realism of the counterfactuals is a concern which we hope to improve in the future

Depth-Assisted Semantic Segmentation, Image Enhancement and Parametric Modeling

This dissertation addresses the problem of employing 3D depth information on solving a number of traditional challenging computer vision/graphics problems. Humans have the abilities of perceiving the depth information in 3D world, which enable humans to reconstruct layouts, recognize objects and understand the geometric space and semantic meanings of the visual world. Therefore it is significant to explore how the 3D depth information can be utilized by computer vision systems to mimic such abilities of humans. This dissertation aims at employing 3D depth information to solve vision/graphics problems in the following aspects: scene understanding, image enhancements and 3D reconstruction and modeling. In addressing scene understanding problem, we present a framework for semantic segmentation and object recognition on urban video sequence only using dense depth maps recovered from the video. Five view-independent 3D features that vary with object class are extracted from dense depth maps and used for segmenting and recognizing different object classes in street scene images. We demonstrate a scene parsing algorithm that uses only dense 3D depth information to outperform using sparse 3D or 2D appearance features. In addressing image enhancement problem, we present a framework to overcome the imperfections of personal photographs of tourist sites using the rich information provided by large-scale internet photo collections (IPCs). By augmenting personal 2D images with 3D information reconstructed from IPCs, we address a number of traditionally challenging image enhancement techniques and achieve high-quality results using simple and robust algorithms. In addressing 3D reconstruction and modeling problem, we focus on parametric modeling of flower petals, the most distinctive part of a plant. The complex structure, severe occlusions and wide variations make the reconstruction of their 3D models a challenging task. We overcome these challenges by combining data driven modeling techniques with domain knowledge from botany. Taking a 3D point cloud of an input flower scanned from a single view, each segmented petal is fitted with a scale-invariant morphable petal shape model, which is constructed from individually scanned 3D exemplar petals. Novel constraints based on botany studies are incorporated into the fitting process for realistically reconstructing occluded regions and maintaining correct 3D spatial relations. The main contribution of the dissertation is in the intelligent usage of 3D depth information on solving traditional challenging vision/graphics problems. By developing some advanced algorithms either automatically or with minimum user interaction, the goal of this dissertation is to demonstrate that computed 3D depth behind the multiple images contains rich information of the visual world and therefore can be intelligently utilized to recognize/ understand semantic meanings of scenes, efficiently enhance and augment single 2D images, and reconstruct high-quality 3D models

구글 스트릿뷰를 이용한 도시 협곡 내 평균복사온도 추정

학위논문 (석사) -- 서울대학교 대학원 : 농업생명과학대학 생태조경학과, 2021. 2. 이동근.도시개발로 인해 보행자의 에너지 균형을 변화시키며 도시공간의 열 쾌적성이 악화되는 등 열 환경문제가 발생하고 있다. 선행연구에서는 도시 공간 내 열 쾌적성을 정량적으로 평가하기 위해 인간의 가장 중요한 생체 기상 변수 중 하나인 평균복사온도를 산정하는 연구가 진행되고 있다. 하지만 산정식이 복잡하거나, 넓은 범위에서의 공간 데이터 취득이 어렵기 때문에, 커뮤니티 단위에서 고해상도의 평균복사온도를 추정하는 것은 어렵다. 따라서 본 연구에서는 구글스트릿뷰 이미지를 사용하여 도시 거리 협곡내 평균복사 온도를 추정하는 방법을 제시하고, 도시 스케일에서 도시열섬 분석을 위해 많은 연구가 진행된 지표면 온도와 평균복사온도간 관계를 공간패턴 측면에서 분석하였다. 우선 평균복사온도 추정식에 큰 영향을 미치는 천공률은 파노라마 이미지를 바탕으로 딥러닝을 활용하여 도시 요인별(건물, 나무, 하늘 등)분류하고, 어안렌즈 이미지로 변환하여 도출하였다. 또한 어안렌즈 이미지를 중심으로 태양경로 알고리즘을 활용하여 시간별 그림자의 유무를 판단하였다. 마지막으로 기후요인, 시간, 위치 등 데이터를 활용하여 장파, 단파 복사를 도출하여 평균복사온도를 산정하였다. 제안된 평균복사온도 추정 방법과 실측간 비교(7 곳) 결과 단파, 장파 값의 R^2값이 각각 0.97, 0.77로 나타났다. 다른 모델과 비교한 결과, 높은 정확도를 확인할 수 있으며 복잡한 도시 환경에서의 활용가능성을 확인할 수 있다. 도시규모에서 지표면온도, 평균복사온도를 공간패턴 측면에서 비교한 결과 천공률, 빌딩 뷰팩터가 각각 0.6~1.0, 0.35-0.5인 오픈스페이스 혹은 저층 밀집지역에서 높은 평균복사온도(>59.4°C)를 보였다. 반면 높은 빌딩이 밀집된 지역의 경우(빌딩 뷰팩터 :0.4-0.6, 나무 뷰팩터 0.6-0.9) 낮은 평균복사온도(<47.6°C)를 보였다. 특히 거리의 방향이 동-서 인 경우에는 천공률이 0.3-0.55 일지라도 높은 평균복사온도를 확인할 수 있었다. 추가적으로 평균복사온도와 지표면 온도간 비교결과 전반적으로 높은 온도 값을 가진 공간이 유사하였으나, 저층 고밀도 건물 지역 혹은 초지 지역에서 상반된 결과를 확인할 수 있었다. 본 연구에서는 도시스케일에서 높은 해상도로 평균복사온도를 추정하는 방법을 딥러닝을 활용하여 제시하였으며, 지표면 온도와 공간패턴별 분석을 통해 실제 보행자가 체감하는 열 환경을 개선하기 위한 방안을 제시할 수 있는 기초자료를 제공하였다. 이는 도시 열 환경을 고려한 지속가능한 도시 공간 설계 및 환경 계획 측면에서 활용 될 수 있으며, 특히 공간데이터 취득이 어려운 곳에서의 높은 활용성을 기대해 볼 수 있다.This paper presents a method for estimating Mean Radiant Temperature (MRT) of street canyons using Google Street View (GSV) images and investigates its spatial patterns in street-level on large scale. We used image segmentation using deep learning, project panorama to fisheye image and sun path algorithms to estimate MRT using GSV. Verification of proposed method can be explained by total of 7 field measurements in clear-sky of street-level, since the estimated shortwave and longwave radiation of value is 0.97, 0.77 respectively. The method proposed in this study is suitable for actual complex urban environment consisting of buildings, tree and streets. Additionally, we compared calculated MRT and LST (Land Surface Temperature) from Landsat 8 in a city scale. As a result of investigating spatial patterns of MRT in Seoul, We found that Higher MRT of street canyons ( >59.4℃) is mainly distributed in open space areas and compact low-rise density building where SVF (Sky View Factor) is 0.6–1.0 and BVF(Building View Factor) is 0.35–0.5, or West-East orientation street canyons with SVF(0.3–0.55). On the other hand, high density building (BVF is 0.4–0.6) or high density tree areas (TVF (Tree View Factor) is 0.6–0.99) showed Low MRT ( < 47.6). The mapped MRT results had similar spatial distribution with LST, but the MRT(?) lower (?) than LST in low tree density or low-rise high-density building areas. And it will help decision makers how to improve thermal comfort at the street-level.Chapter 1. Introduction １ 1.1. Study Background １ 1.2. Literature review ４ 1.2.1 Mean radiant temperature formula ４ 1.2.2 Surface temperature simulation model ５ Chapter 2. Study area and data １０ 2.1. Study area １０ 2.2. Data collection １１ Chapter 3. Method １３ 3.1. Research flow １３ 3.2. MRT simulation １４ 3.2.1. Schematic flow for MRT simulation １４ 3.2.2. Urban canyon geometry calculation using GSV images (Phase I: built geometry data) １６ 3.2.3. Street canyon solar radiation calculation (Phase II:radiation transfer calculation.) １７ 3.2.3.1 Calculation of street-level shortwave radiation １７ 3.2.3.2 Calculation of street-level long-wave radiation １９ 3.2.4. Phase III mean radiation temperature calculation ２１ Chapter 4. Result and Discussion ２２ 4.1. verification of solar radiation estimated in street-level ２２ 4.2. Validation of Long-wave radiation ２４ 4.3. Comparison between LST and MRT estimated using GSV ２６ 4.4. Comparison of GSV_MRT with other models ２９ 4.5. limitations and future development ３２ Chapter 5. Conclusion ３４ Bibliography ３６ Abstract in Korean ４３ Appendix ４５Maste

Radiation techniques for urban thermal simulation with the Finite Element Method

Modern societies are increasingly organized in cities. In the present times, more than half of the world’s population lives in urban settlements. In this context, architectural and building scale works have the need of extending their scope to the urban environment. One of the main challenges of these times is understanting all the thermal exchanges that happen in the city. The radiative part appears as the less developed one; its characterization and interaction with built structures has gained attention for building physics, architecture and environmental engineering. Providing a linkage between these areas, the emerging field of urban physics has become important for tackling studies of such nature. Urban thermal studies are intrinsically linked to multidisciplinary work approaches. Performing full-scale measurements is hard, and prototype models are difficult to develop. Therefore, computational simulations are essential in order to understand how the city behaves and to evaluate projected modifications. The methodological and algorithmic improvement of simulation is one of the mainlines of work for computational physics and many areas of computer science. The field of computer graphics has addressed the adaptation of rendering algorithms to daylighting using physically-based radiation models on architectural scenes. The Finite Element Method (FEM) has been widely used for thermal analysis. The maturity achieved by FEM software allows for treating very large models with a high geometrical detail and complexity. However, computing radiation exchanges in this context implies a hard computational challenge, and forces to push the limits of existing physical models. Computer graphics techniques can be adapted to FEM to estimate solar loads. In the thermal radiation range, the memory requirements for storing the interaction between the elements grows because all the urban surfaces become radiation sources. In this thesis, a FEM-based methodology for urban thermal analysis is presented. A set of radiation techniques (both for solar and thermal radiation) are developed and integrated into the FEM software Cast3m. Radiosity and ray tracing are used as the main algorithms for radiation computations. Several studies are performed for different city scenes. The FEM simulation results are com-pared with measured temperature results obtained by means of urban thermography. Post-processing techniques are used to obtain rendered thermograms, showing that the proposed methodology pro-duces accurate results for the cases analyzed. Moreover, its good computational performance allows for performing this kind of study using regular desktop PCs.Las sociedades modernas están cada vez más organizadas en ciudades. Más de la mitad de la población mundial vive en asentamientos urbanos en la actualidad. En este contexto, los trabajos a escala arquitectónica y de edificio deben extender su alcance al ambiente urbano. Uno de los mayores desafíos de estos tiempos consiste en entender todos los intercambios térmicos que suceden en la ciudad. La parte radiativa es la menos desarrollada; su caracterización y su interacción con edificaciones ha ganado la atención de la física de edificios, la arquitectura y la ingeniería ambiental. Como herramienta de conexión entre estas áreas, la física urbana es un área que resulta importante para atacar estudios de tal naturaleza. Los estudios térmicos urbanos están intrinsecamente asociados a trabajos multidisciplinarios. Llevar a cabo mediciones a escala real resulta difícil, y el desarrollo de prototipos de menor escala es complejo. Por lo tanto, la simulación computacional es esencial para entender el comportamiento de la ciudad y para evaluar modificaciones proyectadas. La mejora metodológica y algorítmica de las simulaciones es una de las mayores líneas de trabajo para la física computacional y muchas áreas de las ciencias de la computación. El área de la computación gráfica ha abordado la adaptación de algoritmos de rendering para cómputo de iluminación natural, utilizando modelos de radiación basados en la física y aplicándolos sobre escenas arquitectónicas. El Método de Elementos Finitos (MEF) ha sido ampliamente utilizado para análisis térmico. La madurez alcanzada por soluciones de software MEF permite tratar grandes modelos con un alto nivel de detalle y complejidad geométrica. Sin embargo, el cómputo del intercambio radiativo en este contexto implica un desafío computacional, y obliga a empujar los límites de las descripciones físicas conocidas. Algunas técnicas de computación gráfica pueden ser adaptadas a MEF para estimar las cargas solares. En el espectro de radiación térmica, los requisitos de memoria necesarios para almacenar la interacción entre los elementos crecen debido a que todas las superficies urbanas se transforman en fuentes emisoras de radiación. En esta tesis se presenta una metodología basada en MEF para el análisis térmico de escenas urbanas. Un conjunto de técnicas de radiación (para radiación solar y térmica) son desarrolladas e integradas en el software MEF Cast3m. Los algoritmos de radiosidad y ray tracing son utilizados para el cómputo radiativo. Se presentan varios estudios que utilizan diferentes modelos de ciudades. Los resultados obtenidos mediante MEF son comparados con temperaturas medidas por medio de termografías urbanas. Se utilizan técnicas de post-procesamiento para renderizar imágenes térmicas, que permiten concluir que la metodología propuesta produce resultados precisos para los casos analizados. Asimismo, su buen desempeño computacional posibilita realizar este tipo de estudios en computadoras personales

Urban Visual Intelligence: Studying Cities with AI and Street-level Imagery

The visual dimension of cities has been a fundamental subject in urban studies, since the pioneering work of scholars such as Sitte, Lynch, Arnheim, and Jacobs. Several decades later, big data and artificial intelligence (AI) are revolutionizing how people move, sense, and interact with cities. This paper reviews the literature on the appearance and function of cities to illustrate how visual information has been used to understand them. A conceptual framework, Urban Visual Intelligence, is introduced to systematically elaborate on how new image data sources and AI techniques are reshaping the way researchers perceive and measure cities, enabling the study of the physical environment and its interactions with socioeconomic environments at various scales. The paper argues that these new approaches enable researchers to revisit the classic urban theories and themes, and potentially help cities create environments that are more in line with human behaviors and aspirations in the digital age