There is a wide array of online photographic content that is not geotagged. Algorithms for efficient and accurate geographical estimation of an image are needed to geolocate these photos. This paper presents a general model for using both textual metadata and visual features of photos to automatically place them on a world map

Davies, Jamie

Dupplaw, David

Hare, Jonathon

Jain, Neha

Lewis, Paul H.

Preston, John

Samangooei, Sina

English

Southampton (e-Prints Soton)

Identifying the Geographic Location of an Image with aMultimodal Probability Density FunctionJamie Daviesjagd1g11@ecs.soton.ac.ukJonathon Harejsh2@ecs.soton.ac.ukSina Samangooeiss@ecs.soton.ac.ukJohn Prestonjlp1g11@ecs.soton.ac.ukNeha Jainnj1g12@ecs.soton.ac.ukDavid Dupplawdpd@ecs.soton.ac.ukElectronics and Computer Science, University of Southampton, United KingdomABSTRACTThere is a wide array of online photographic content thatis not geotagged. Algorithms for ecient and accurate ge-ographical estimation of an image are needed to geolocatethese photos. This paper presents a general model for us-ing both textual metadata and visual features of photos toautomatically place them on a world map.1. INTRODUCTION AND MOTIVATIONThe primary goal of the 2013 MediaEval placing task [3]was to develop techniques for accurately predicting the geo-location of a set of Flickr images in terms of latitude andlongitude. In addition, a secondary goal was to enhance pre-dictions by estimating the error of the predicted location ofeach image. The task organisers provided a set of approxi-mately 8.5 million images with metadata and locations fortraining, and a set of 262,000 images without geotags fortesting.The motivation for the techniques we have developed forthe task was twofold; we rstly wanted to develop a tech-nique that can operate using either the visual content orthe metadata, but which also seamlessly allowed blending ofinformation across modalities and allowed information fromexternal gazetteers to be incorporated. Secondly, we wantedour technique to be scalable and ecient, with the aim ofbeing able to estimate the position of an image in well undera second using standard desktop hardware.2. OVERALL METHODOLOGYThe basic idea of our approach is that we estimate a con-tinuous probability density function (PDF) over the surfaceof the Earth from a number of features extracted from thequery image and/or its metadata. To estimate the PDF,each feature provides a xed size set of points (latitude,longitude) which are then combined, and a kernel densityestimator can be used to estimate the probability density atany arbitrary position. By nding the modes of the PDF wecreate an estimate of the location of the photograph fromthe position of the mode with the highest probability. Bytting a univariate Gaussian over the support of the high-est probability mode, we can estimate the accuracy of theestimated geolocation as a function of the variance.Copyright is held by the author/owner(s).MediaEval 2013 Workshop, October 18-19, 2013, Barcelona, SpainIn practice, density estimation and mode-nding can becombined by applying the mean-shift algorithm. Mean-shifthas been used in the context of geolocation estimation in thepast; for example, Hays and Efros [4] used mean-shift on theresults of content-based image search to determine proba-ble locations. Our approach diers from that of Hays andEfros's, because, whereas they only considered single (highrecall/low precision) content-based features, we consider thefusion of multiple features from dierent modalities. In ad-dition, Hays and Efros used the mean-shift algorithm forcoarse-grained location estimation, with a very large kernelbandwidth, whilst in our technique, because of the way weare using features we are able to use a much smaller kernelbandwidth for ne-grained location estimation.3. EXPERIMENTSThe implementation of our methodology was realised inJava using OpenIMAJ1 [1] and Lucene2. For speed, we usedan approximate mean-shift implementation inspired by theone in scikit-learn3. The approximations stem from using aregular grid for determining the seed points from which toseek modes (rather than using the actual data), and usingnearest-neighbours to assign data points to modes, ratherthan actually assigning them to the mode they converge to.A KD-Tree is used for ecient nearest neighbour lookup.3.1 FeaturesThe following features were used in our experiments. Eachfeature provides a set of geographic points in response to aquery image:Location Prior. A constant prior feature built by sampling1000 geographical coordinates from the training data.Tags. Every tag in the query image is associated with thecoordinates of the training images in which the tag appeared.If a tag in the query was unseen in the training data, thenit contributes no points. Each tag is considered to be anindependent feature. No ltering of tags was performed.PQ-CEDD. In order to provide a high-recall/low-precisionimage search we indexed the provided CEDD features witha product quantiser (18 products of 256 clusters) to enablefast in-memory search of the complete training data usingthe asymmetric distance computation method [5]. The lo-1http://openimaj.org2http://lucene.apache.org3http://scikit-learn.org/stable/modules/clustering.html#mean-shiftTable 1: Results of the ve runsRun 1 Run 2 Run 3 Run 4 Run 5No. est. within 1km 53449 (20.55%) 891 (0.34%) 60190 (23.15%) 68050 (26.17%) 61631 (23.70%)No. est. within 10km 81988 (31.53%) 1453 (0.56%) 98032 (37.70%) 106370 (40.9%) 100009 (38.47%)No. est. within 100km 93838 (36.09%) 2711 (0.10%) 113937 (43.82%) 123233 (47.40%) 114986 (44.23%)No. est. within 500km 109117 (41.97%) 9174 (3.52%) 132655 (51.02%) 139136 (53.51%) 129721 (49.89%)No. est. within 1000km 122752 (47.21%) 19129 (7.36%) 147443 (56.71%) 151876 (58.41%) 141767 (54.53%)Median error in km 1352.897154 6898.266561 451.8928271 254.4838372 540.109773Linear correlation of error 0.1568 0.0594 0.3693 0.3721 0.0406Table 2: The feature conguration for the run submissions.Prior Tags PQ-CEDD SIFT-LSHRun 1 X X X XRun 2 X X XRun 3 X XRun 4 X X XRun 5 X X X Xcations of the 100 top images to a query formed the pointset returned by the feature.LSH-SIFT. High-precision (low recall) image content searchwas performed using a variant of the approach we developedin [2]. DoG-SIFT features were extracted from the imagesand hashed using Locality Sensitive Hashing. A graph wasbuilt with the images as the nodes, and edge weights werebased on the number of collisions. In order to perform aquery, the directly connected nodes to the query were de-tected in the graph, and their geo-coordinates returned.3.2 RunsFor all of our submitted runs, the methodology describedin Section 2 was applied. In all runs we used a at ker-nel with bandwidth of 0:01. All runs except for run #5used a constant number of 1000 points per feature (eec-tively making all features have a xed weight); any featurethat produced more or less points was randomly and uni-formly sub- or super-sampled to reach 1000 points. In total,5 runs were submitted. The congurations are summarisedin Table 2. Notes about each run are listed below:Runs 1-3: Provided data. The rst three runs used fea-tures extracted from the provided dataset only. No externaldata was used.Run 4: Text+Visual, bigger dataset. The fourth runused features extracted from a larger dataset of approxi-mately 46 million geotagged Flickr images we collected lastyear. As with the provided data, only images with an accu-racy of 16 were crawled. For the purposes of fair experimen-tation, we removed all photos from the users who appearedin the test set.Run 5: Text+Visual, provided data with tag boost-ing. The fth run was the same as run 1, but we usedthe GeoNames4 gazetteer to boost the weight of tag fea-tures that were likely to belong to a specic geographic lo-cation; any textual tag that could be matched against thethe GeoNames\name"or\alternate-name"eld was boostedby doubling its number of points from 1000 to 2000. Nontextual features, and tags that didn't match remained at4http://www.geonames.org1000 points/feature.3.3 Results, Discussion and Future WorkOur results are shown in Table 1. The rst thing to noteis that visual features alone perform relatively poorly for ex-actly predicting locations; this is be be expected as the vastmajority of images do not contain recognisable places. Inter-estingly our rst and fourth runs (text and visual) performedworse than the third (text only). This was unexpected andwarrants further investigation as our experiments had in-dicated that the visual features should help boost perfor-mance. Run 4 indicates that using additional data helpsour approach. We'd expect our error estimation to improve(higher correlation coecient) as we become more accurate;this trend can be seen to an extent, although run 5 is a def-inite outlier. We had started to experiment with product-quantised PCA VLAD encodings of SIFT features, temporalfeatures and query expansion of LSH-SIFT features, how-ever we ran out of time to optimise and include these inthe runs. It will be interesting to investigate these further,together with other features such as GIST in the future. Itwould also be interesting to try some other approaches toincorporating structured knowledge from GeoNames.4. ACKNOWLEDGMENTSThe described work was funded by the European UnionSeventh Framework Programme (FP7/2007-2013) under grantagreements 270239 (ARCOMEM), and 287863 (TrendMiner).5. ADDITIONAL AUTHORSAdditional author: Paul Lewis (email: phl@ecs.soton.ac.uk)6. REFERENCES[1] J. S. Hare, S. Samangooei, and D. P. Dupplaw.OpenIMAJ and ImageTerrier: Java libraries and toolsfor scalable multimedia analysis and indexing ofimages. In ACM MM'11, pages 691{694. ACM, 2011.[2] J. S. Hare, S. Samangooei, D. P. Dupplaw, and P. H.Lewis. Twitter's visual pulse. In ICMR'13, pages297{298, New York, NY, USA, 2013. ACM.[3] C. Hau, B. Thomee, and M. Trevisiol. Working Notesfor the Placing Task at MediaEval 2013. In MediaEval2013 Workshop, Barcelona, Spain, October 18-19 2013.[4] J. Hays and A. A. Efros. im2gps: estimating geographicinformation from a single image. In CVPR' 08, 2008.[5] H. Jegou, M. Douze, and C. Schmid. Productquantization for nearest neighbor search. IEEE Trans.Pattern Anal. Mach. Intell., 33(1):117{128, Jan. 2011.

im2gps: estimating geographic information from a single image.

OpenIMAJ and ImageTerrier: Java libraries and tools for scalable multimedia analysis and indexing of images.

Product quantization for nearest neighbor search.

Twitter's visual pulse.

Working Notes for the Placing Task at MediaEval

Identifying the Geographic Location of an Image with a Multimodal Probability Density Function

Identifying the Geographic Location of an Image with a Multimodal Probability Density Function

Abstract

Similar works

Full text

Available Versions

Southampton (e-Prints Soton)