Beyond Geo-localization: Fine-grained Orientation of Street-view Images by Cross-view Matching with Satellite Imagery

Street-view imagery provides us with novel experiences to explore different places remotely. Carefully calibrated street-view images (e.g. Google Street View) can be used for different downstream tasks, e.g. navigation, map features extraction. As personal high-quality cameras have become much more affordable and portable, an enormous amount of crowdsourced street-view images are uploaded to the internet, but commonly with missing or noisy sensor information. To prepare this hidden treasure for "ready-to-use" status, determining missing location information and camera orientation angles are two equally important tasks. Recent methods have achieved high performance on geo-localization of street-view images by cross-view matching with a pool of geo-referenced satellite imagery. However, most of the existing works focus more on geo-localization than estimating the image orientation. In this work, we re-state the importance of finding fine-grained orientation for street-view images, formally define the problem and provide a set of evaluation metrics to assess the quality of the orientation estimation. We propose two methods to improve the granularity of the orientation estimation, achieving 82.4% and 72.3% accuracy for images with estimated angle errors below 2 degrees for CVUSA and CVACT datasets, corresponding to 34.9% and 28.2% absolute improvement compared to previous works. Integrating fine-grained orientation estimation in training also improves the performance on geo-localization, giving top 1 recall 95.5%/85.5% and 86.8%/80.4% for orientation known/unknown tests on the two datasets.Comment: This paper has been accepted by ACM Multimedia 2022. The version contains additional supplementary material

Object-based multi-view façade matching in SAR images of dense urban areas

Building information retrieval from remote sensing imagery is an important research topic in the remote sensing field. The extracted building footprints, heights and façade structures help us better understand our living areas. This information also gives great assistance for large urban area monitoring and future planning. With the availability of meter-resolution spaceborne synthetic aperture radar (SAR) sensors such as TerraSAR-X, SAR data have also been extensively exploited for building 3-D reconstruction. On one hand, SAR interferometric (InSAR) methods use the phase information of a time series of observations to retrieve the topography of urban areas. InSAR methods can provide 3-D information with detail up to individual floors, given a relative large stack of images. On the other hand, another group of methods extracts building information by solely exploiting the amplitude of a few SAR images, e.g. stereo radargrammetry. These methods require fewer images than InSAR methods. But the retrieved 3-D information is so far only limited to a single building roof height or the height of a set of sparse points in the image. Therefore, this thesis proposed a new method that provides precise 3-D height estimates of individual floors of a building façade using only three SAR amplitude images of different incidence angles from either all ascending or all descending orbits. The firsthand information of a building can be obtained within a short time span. The method detects and extracts building façade with regular patterns, and performs an object-based façade matching and floor height estimation. The proposed method requires no coregistration between the three images. This avoids additional coregistration error, as well as the azimuth disparity introduced by coregistering images acquired from nonparallel orbits. The algorithm has been tested on real and simulated data. The object-based façade matching and height estimation is robust against the appearance of strong reflective scatterers that do not belong to the building façade. In addition, the algorithm is applied to individual buildings whose heights of each floor are referred to their ground floor respectively, so that the effect of atmospheric delay is insignificant

Human emotion recognition to improve driving safety

The emotion recognition program is designed to recognize drivers' emotional states by using real-time video. As an important part of the future automobile auto-control system, the emotion recognition program could provide the information about the drivers' emotional state to the system so that it could identify the aggressive behavior of the driver and make necessary correction to avoid accidents. In this emotion recognition program, ELM machine learning algorithm and various image processing operations are used. A specific face detector used in the driving environment and an emotion classifier is developed. The offline testing and the real-time emotion recognition program could effectively detect the face region and classify the emotional state into positive or aggressive. This project is a very important study for the future driving safety improvement.Bachelor of Engineerin

Reduced sensitivity to neutral feedback versus negative feedback in subjects with mild depression: Evidence from event-related potentials study

Many previous event-related potential (ERP) studies have linked the feedback related negativity (FRN) component with medial frontal cortex processing and associated this component with depression. Few if any studies have investigated the processing of neutral feedback in mildly depressive subjects in the normal population. Two experiments compared brain responses to neutral feedback with behavioral performance in mildly depressed subjects who scored highly on the Beck Depression Inventory (high BDI) and a control group with lower BDI scores (low BDI). In the first study, the FRN component was recorded when neutral, negative or positive feedback was pseudo-randomly delivered to the two groups in a time estimation task. In the second study, real feedback was provided to the two groups in the same task in order to measure their actual accuracy of performance. The results of experiment one (Exp. 1) revealed that a larger FRN effect was elicited by neutral feedback than by negative feedback in the low BDI group, but no significant difference was found between neutral condition and negative condition in the High BDI group. The present findings demonstrated that depressive tendencies influence the processing of neutral feedback in medial frontal cortex. The FRN effect may work as a helpful index for investigating cognitive bias in depression in future studies