Crowd-sourced Pictures Geo-localization Method Based on 3D Reconstruction

doi:10.11947/j.AGCS.2018.20170365

Abstract

Abstract: People are increasingly becoming accustomed to taking photos of everyday life in modern cities and uploading them on major photo-sharing social media sites. These sites contain numerous pictures, but many have incomplete or blurred location information. The geo-localization of crowd-sourced pictures enriches the information contained therein, and is applicable to activities such as urban construction, urban landscape analysis, and crime tracking. However, geo-localization faces huge technical challenges. This paper proposes a method for large-scale geo-localization of crowd-sourced pictures. Our approach uses structured, organized Street View images as a reference dataset and employs a three-step strategy of coarse geo-localization by image retrieval, selecting reliable matches by image registration, and fine geo-localization by 3D reconstruction to attach geographic tags to pictures from unidentified sources. 3D reconstruction based on close-range photogrammetry is used to restore the 3D geographical information of the crowd-sourced pictures, resulting in the proposed method improving the median error from 256.7 m to 69.0 m, and the percentage of the geo-localized query pictures under a 50 m error requirement from 17.2% to 43.2% compared with the previous method. Another discovery of the proposed method is that, regarding the causes of reconstruction error, closer distances from the query cameras to the main objects in query pictures tend to produce smaller errors. The proposed method is not limited to small areas, and could be expanded to cities and larger areas owing to its flexible parameters.

Key words: street view images, geo-localization, image retrieval, 3D reconstruction, accuracy analysis

CLC Number:

P237

YUAN Yi, CHENG Liang, ZONG Wenwen, LI Shuyi, LI Manchun. Crowd-sourced Pictures Geo-localization Method Based on 3D Reconstruction[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(5): 631-643.

References

[1] LI Songnian, DRAGICEVIC S, CASTRO F A, et al. Geospatial Big Data Handling Theory and Methods:A Review and Research Challenges[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 115:119-133.
[2] HEIPKE C. Crowdsourcing Geospatial Data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(6):550-557.
[3] ARTH C, PIRCHHEIM C, VENTURA J, et al. Instant Outdoor Localization and SLAM Initialization from 2.5D Maps[J]. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(11):1309-1318.
[4] YIN Li, CHENG Qimin, WANG Zhenxin, et al. ‘Big Data’ for Pedestrian Volume:Exploring the Use of Google Street View Images for Pedestrian Counts[J]. Applied Geography, 2015, 63:337-345.
[5] SALMEN J, HOUBEN S, SCHLIPSING M. Google Street View Images Support the Development of Vision-based Driver Assistance Systems[C]//Proceedings of 2012 IEEE Intelligent Vehicles Symposium. Alcala de Henares, Spain:IEEE, 2012:891-895.
[6] TSAI V J D. Traffic Sign Detection and Positioning from Google Street View Streamlines[C]//Proceedings of ASPRS 2015 Annual Conference. Tampa, Florida:ASPRS, 2015.
[7] LIU Yue, WANG Yongtian. AR-view:An Augmented Reality Device for Digital Reconstruction of Yuangmingyuan[C]//Proceedings of 2009 IEEE International Symposium on Mixed and Augmented Reality-arts, Media and Humanities. Orlando, FL:IEEE, 2009:3-7.
[8] BADLAND H M, OPIT S, WITTEN K, et al. Can Virtual Streetscape Audits Reliably Replace Physical Streetscape Audits?[J]. Journal of Urban Health, 2010, 87(6):1007-1016.
[9] LI Xiaojiang, ZHANG Chuanrong, LI Weidong, et al. Assessing Street-level Urban Greenery Using Google Street View and a Modified Green View Index[J]. Urban Forestry & Urban Greening, 2015, 14(3):675-685.
[10] YIN Li, WANG Zhenxin. Measuring Visual Enclosure for Street Walkability:Using Machine Learning Algorithms and Google Street View Imagery[J]. Applied Geography, 2016, 76:147-153.
[11] HARA K, AZENKOT S, CAMPBELL M, et al. Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Locations with Google Street View:An Extended Analysis[J]. ACM Transactions on Accessible Computing (TACCESS), 2015, 6(2):5.
[12] ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization[M]. Cham:Springer, 2016.
[13] LOWE D G. Distinctive Image Features from Scale-invariant Keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[14] SATTLER T, LEIBE B, KOBBELT L. Fast Image-based Localization Using Direct 2D-to-3D Matching[C]//Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona, Spain:IEEE, 2011:667-674.
[15] CHEN D M, BAATZ G, KÖSER K, et al. City-scale Landmark Identification on Mobile Devices[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO:IEEE, 2011:737-744.
[16] KNOPP J, SIVIC J, PAJDLA T. Avoiding Confusing Features in Place Recognition[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:748-761.
[17] NISTER D, STEWENIUS H. Scalable Recognition with a Vocabulary Tree[C]//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York:IEEE, 2006:2161-2168.
[18] ZAMIR A R, SHAH M. Accurate Image Localization Based on Google Maps Street View[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:255-268.
[19] ZAMIR A R, SHAH M. Image Geo-localization Based on Multiplenearest Neighbor Feature Matching Using Generalized Graphs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(8):1546-1558.
[20] SCHINDLER G, BROWN M, SZELISKI R. City-scale Location Recognition[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis:IEEE, 2007:1-7.
[21] KNOPP J, SIVIC J, PAJDLA T. Avoiding Confusing Features in Place Recognition[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:748-761.
[22] HAYS J, EFROS A A. IM2GPS:Estimating Geographic Information from a Single Image[C]//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage:IEEE, 2008:1-8.
[23] YAZAWA N, UCHIYAMA H, SAITO H, et al. Image Based View Localization System Retrieving from a Panorama Database by SURF[C]//Proceedings of 2009 IAPR Conference on Machine Vision Applications. Yokohama, Japan:[s.n.], 2009:3632-3636.
[24] KE Yan, SUKTHANKAR R. PCA-SIFT:A More Distinctive Representation for Local Image Descriptors[C]//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington DC:IEEE, 2004:506-513.
[25] PARK H, MOON K S. Fast Feature Matching by Coarse-to-fine Comparison of Rearranged SURF Descriptors[J]. IEICE Transactions on Information and Systems, 2015, E98-D(1):210-213.
[26] KAMENCAY P, BREZNAN M, JELSOVKA D, et al. Improved Face Recognition Method Based on Segmentation Algorithm Using SIFT-PCA[C]//Proceedings of the 35th International Conference on Telecommunications and Signal Processing. Prague, Czech Republic:IEEE, 2012:758-762.
[27] TANG Gefu, XIAO Zhifeng, LIU Qing, et al. A Novel Airport Detection Method via Line Segment Classification and Texture Classification[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(12):2408-2412.
[28] XIAO Zhifeng, LIU Qing, TANG Gefu, et al. Elliptic Fourier Transformation-based Histograms of Oriented Gradients for Rotationally Invariant Object Detection in Remote-Sensing Images[J]. International Journal of Remote Sensing, 2015, 36(2):618-644.
[29] BLASCHKE T. Object Based Image Analysis for Remote Sensing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(1):2-16.
[30] SE S, NADEAU C, WOOD S. Automated UAV-based Video Exploitation Using Service Oriented Architecture Framework[C]//Proceedings of Volume 8020, Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications VⅢ. Orlando, FL:SPIE, 2011:80200Y.
[31] SZLÁVIK Z, SZIRÁNYI T, HAVASI L. Video Camera Registration Using Accumulated Co-motion Maps[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2007, 61(5):298-306.
[32] LI Yunpeng, SNAVELY N, HUTTENLOCHER D, et al. Worldwide Pose Estimation Using 3D Point Clouds[C]//Proceedings of the 12th European Conference on Computer Vision. Florence, Italy:Springer, 2012:15-29.
[33] SVÄRM L, ENQVIST O, OSKARSSON M, et al. Accurate Localization and Pose Estimation for Large 3D Models[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:532-539.
[34] GERNHARDT S, AUER S, EDER K. Persistent Scatterers at Building Facades-evaluation of Appearance and Localization Accuracy[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 100:92-105.
[35] SINGH G, KOŠECKÁ J. Semantically Guided Geo-location and Modeling in Urban Environments[M]//ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization. Cham:Springer, 2016:101-120.
[36] CRANDALL D J, LI Yunpeng, LEE S, et al. Recognizing Landmarks in Large-scale Social Image Collections[M]//ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization. Cham:Springer, 2016:121-144.
[37] NIEVERGELT J, HINTERBERGER H, SEVCIK K C. The Grid File:An Adaptable, Symmetric Multikey File Structure[J]. ACM Transactions on Database Systems (TODS), 1984, 9(1):38-71.
[38] ROBINSON J T. The K-D-B-Tree:A Search Structure for Large Multidimensional Dynamic Indexes[C]//Proceedings of 1981 ACM SIGMOD International Conference on Management of Data. Ann Arbor, Michigan:ACM, 1981:10-18.
[39] SNAVELY N, SEITZ S M, SZELISKI R. Modeling the World from Internet Photo Collections[J]. International Journal of Computer Vision, 2008, 80(2):189-210.
[40] WU Changchang. Towards Linear-time Incremental Structure from Motion[C]//Proceedings of 2013 International Conference on 3D Vision-3DV 2013. Seattle:IEEE, 2013:127-134.
[41] MATAS J, CHUM O, URBAN M, et al. Robust Wide-baseline Stereo from Maximally Stable Extremal Regions[J]. Image and Vision Computing, 2004, 22(10):761-767.
[42] 程亮, 龚健雅, 宋小刚, 等. 面向宽基线立体影像匹配的高质量仿射不变特征提取方法[J]. 测绘学报, 2008, 37(1):77-82. CHENG Liang, GONG Jianya, SONG Xiaogang, et al. A Method for Affine Invariant Feature Extraction with High Quality for Wide Baseline Stereo Image Matching[J]. Acta Geodaetica et Cartographica Sinica, 2008, 37(1):77-82.
[43] INDYK P, MOTWANI R. Approximate Nearest Neighbors:Towards Removing the Curse of Dimensionality[C]//Proceedings of the 30th Annual ACM Symposium on Theory of Computing. Dallas, Texas:ACM, 1998:604-613.
[44] ZHANG Zhengyou, DERICHE R, FAUGERAS O, et al. A Robust Technique for Matching Two Uncalibrated Images through the Recovery of the Unknown Epipolar Geometry[J]. Artificial Intelligence, 1995, 78(1-2):87-119.
[45] WESTOBY M J, BRASINGTON J, GLASSER N F, et al. ‘Structure-from-Motion’ Photogrammetry:A Low-cost, Effective Tool for Geoscience Applications[J]. Geomorphology, 2012, 179:300-314.
[46] LOURAKIS M I A, ARGYROS A A. SBA:A Software Package for Generic Sparse Bundle Adjustment[J]. ACM Transactions on Mathematical Software, 2009, 36(1):2.
[47] 刘颖真, 贾奋励, 万刚, 等. 非专业弱关联影像的地理配准及其精度评估[J]. 测绘学报, 2015, 44(9):1014-1021, 1028. DOI:10.11947/j.AGCS.2015.20140394. LIU Yingzhen, JIA Fenli, WAN Gang, et al. Geo-registration of Unprofessional and Weakly-related Image and Precision Evaluation[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(9):1014-1021, 1028. DOI:10.11947/j.AGCS.2015.20140394.
[48] 陈爱军, 徐光祐, 史元春. 基于城市航空立体像对的全自动3维建筑物建模[J]. 测绘学报, 2002, 31(1):54-59. CHEN Aijun, XU Guangyou, SHI Yuanchun. Automated 3D Building Modeling Based on Urban Aerial Stereopair[J]. Acta Geodaetica et Cartographica Sinica, 2002, 31(1):54-59.