互联网众源照片的三维重建定位技术

袁一; 程亮; 宗雯雯; 李舒怡; 李满春

doi:10.11947/j.AGCS.2018.20170365

测绘学报 >

2018 , Vol. 47 >Issue 5: 631 - 643

DOI: https://doi.org/10.11947/j.AGCS.2018.20170365

摄影测量学与遥感

互联网众源照片的三维重建定位技术

袁一 ,
程亮 ,
宗雯雯 ,
李舒怡 ,
李满春

展开

1. 南京大学江苏省地理信息技术重点实验室, 江苏南京 210023;
2. 南京大学中国南海研究协同创新中心, 江苏南京 210023;
3. 南京大学软件新技术与产业化协同创新中心, 江苏南京 210023;
4. 南京大学地理与海洋科学学院, 江苏南京 210023;
5. 南京师范大学江苏省地理信息资源开发与利用协同创新中心, 江苏南京 210023

袁一(1993-),女,硕士生,研究方向为激光雷达与摄影测量数据集成处理与应用。E-mail:yuanyi_1213@163.com

收稿日期: 2017-06-28

修回日期: 2018-03-01

网络出版日期: 2018-06-01

基金资助

国家重点研发计划（2017YFB0504205）；国家自然科学基金（41622109；41371017）

收起

Crowd-sourced Pictures Geo-localization Method Based on 3D Reconstruction

YUAN Yi ,
CHENG Liang ,
ZONG Wenwen ,
LI Shuyi ,
LI Manchun

Expand

1. Jiangsu Provincial Key Laboratory of Geographic Information Science and Technology, Nanjing University, Nanjing 210023, China;
2. Collaborative Innovation Center for the South Sea Studies, Nanjing University, Nanjing 210023, China;
3. Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing University, Nanjing 210023, China;
4. School of Geographic and Oceanographic Sciences, Nanjing University, Nanjing 210023, China;
5. Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing Normal University, Nanjing 210023, China

Received date: 2017-06-28

Revised date: 2018-03-01

Online published: 2018-06-01

Supported by

The National Key Research and Development Program (No.2017YFB0504205);The National Natural Science Foundation of China (Nos.41622109;41371017)

Fold

摘要

随着电子产品的普及和互联网发展，人们习惯性地将照片上传到主流图片分享网站和社交媒体。这些照片很多都没有地理坐标或者仅有模糊的位置信息。如果互联网上任意来源的电子照片能被恢复出真实地理位置，它们将可以应用在城市建设、城市景观分析和犯罪追踪等很多方面。本文提出了一种互联网众筹照片的三维重建定位技术，该技术将结构化组织的街景数据作为参考数据集，使用三步走策略：图像检索粗定位、图像匹配细筛选和三维重建精定位，来给不明来源的众筹照片附上精确的地理标签。本文通过摄影测量原理恢复待查询照片周围的三维空间信息，较之前Zamir和Shah的方法，定位精度中值从256.7 m提升到69.0 m，平均值从350.4 m提升到206.0 m，在50 m精度要求下的照片数量占比从17.2%提升到43.2%。另一个发现是在重建误差成因方面，待查询照片的相机距离主拍摄目标越近，定位总误差越小。本文所提出的照片定位技术提供了灵活的参数，使其应用范围不仅仅局限于小区域，也可以扩大至城市、国家尺度。

关键词： 街景数据; 照片定位; 图像检索; 三维重建; 精度分析

本文引用格式

袁一 , 程亮 , 宗雯雯 , 李舒怡 , 李满春 . 互联网众源照片的三维重建定位技术[J]. 测绘学报, 2018 , 47(5) : 631 -643 . DOI: 10.11947/j.AGCS.2018.20170365

Abstract

People are increasingly becoming accustomed to taking photos of everyday life in modern cities and uploading them on major photo-sharing social media sites. These sites contain numerous pictures, but many have incomplete or blurred location information. The geo-localization of crowd-sourced pictures enriches the information contained therein, and is applicable to activities such as urban construction, urban landscape analysis, and crime tracking. However, geo-localization faces huge technical challenges. This paper proposes a method for large-scale geo-localization of crowd-sourced pictures. Our approach uses structured, organized Street View images as a reference dataset and employs a three-step strategy of coarse geo-localization by image retrieval, selecting reliable matches by image registration, and fine geo-localization by 3D reconstruction to attach geographic tags to pictures from unidentified sources. 3D reconstruction based on close-range photogrammetry is used to restore the 3D geographical information of the crowd-sourced pictures, resulting in the proposed method improving the median error from 256.7 m to 69.0 m, and the percentage of the geo-localized query pictures under a 50 m error requirement from 17.2% to 43.2% compared with the previous method. Another discovery of the proposed method is that, regarding the causes of reconstruction error, closer distances from the query cameras to the main objects in query pictures tend to produce smaller errors. The proposed method is not limited to small areas, and could be expanded to cities and larger areas owing to its flexible parameters.

Key words： street view images; geo-localization; image retrieval; 3D reconstruction; accuracy analysis

参考文献

[1] LI Songnian, DRAGICEVIC S, CASTRO F A, et al. Geospatial Big Data Handling Theory and Methods:A Review and Research Challenges[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 115:119-133.
[2] HEIPKE C. Crowdsourcing Geospatial Data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(6):550-557.
[3] ARTH C, PIRCHHEIM C, VENTURA J, et al. Instant Outdoor Localization and SLAM Initialization from 2.5D Maps[J]. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(11):1309-1318.
[4] YIN Li, CHENG Qimin, WANG Zhenxin, et al. ‘Big Data’ for Pedestrian Volume:Exploring the Use of Google Street View Images for Pedestrian Counts[J]. Applied Geography, 2015, 63:337-345.
[5] SALMEN J, HOUBEN S, SCHLIPSING M. Google Street View Images Support the Development of Vision-based Driver Assistance Systems[C]//Proceedings of 2012 IEEE Intelligent Vehicles Symposium. Alcala de Henares, Spain:IEEE, 2012:891-895.
[6] TSAI V J D. Traffic Sign Detection and Positioning from Google Street View Streamlines[C]//Proceedings of ASPRS 2015 Annual Conference. Tampa, Florida:ASPRS, 2015.
[7] LIU Yue, WANG Yongtian. AR-view:An Augmented Reality Device for Digital Reconstruction of Yuangmingyuan[C]//Proceedings of 2009 IEEE International Symposium on Mixed and Augmented Reality-arts, Media and Humanities. Orlando, FL:IEEE, 2009:3-7.
[8] BADLAND H M, OPIT S, WITTEN K, et al. Can Virtual Streetscape Audits Reliably Replace Physical Streetscape Audits?[J]. Journal of Urban Health, 2010, 87(6):1007-1016.
[9] LI Xiaojiang, ZHANG Chuanrong, LI Weidong, et al. Assessing Street-level Urban Greenery Using Google Street View and a Modified Green View Index[J]. Urban Forestry & Urban Greening, 2015, 14(3):675-685.
[10] YIN Li, WANG Zhenxin. Measuring Visual Enclosure for Street Walkability:Using Machine Learning Algorithms and Google Street View Imagery[J]. Applied Geography, 2016, 76:147-153.
[11] HARA K, AZENKOT S, CAMPBELL M, et al. Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Locations with Google Street View:An Extended Analysis[J]. ACM Transactions on Accessible Computing (TACCESS), 2015, 6(2):5.
[12] ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization[M]. Cham:Springer, 2016.
[13] LOWE D G. Distinctive Image Features from Scale-invariant Keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[14] SATTLER T, LEIBE B, KOBBELT L. Fast Image-based Localization Using Direct 2D-to-3D Matching[C]//Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona, Spain:IEEE, 2011:667-674.
[15] CHEN D M, BAATZ G, KÖSER K, et al. City-scale Landmark Identification on Mobile Devices[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO:IEEE, 2011:737-744.
[16] KNOPP J, SIVIC J, PAJDLA T. Avoiding Confusing Features in Place Recognition[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:748-761.
[17] NISTER D, STEWENIUS H. Scalable Recognition with a Vocabulary Tree[C]//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York:IEEE, 2006:2161-2168.
[18] ZAMIR A R, SHAH M. Accurate Image Localization Based on Google Maps Street View[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:255-268.
[19] ZAMIR A R, SHAH M. Image Geo-localization Based on Multiplenearest Neighbor Feature Matching Using Generalized Graphs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(8):1546-1558.
[20] SCHINDLER G, BROWN M, SZELISKI R. City-scale Location Recognition[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis:IEEE, 2007:1-7.
[21] KNOPP J, SIVIC J, PAJDLA T. Avoiding Confusing Features in Place Recognition[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:748-761.
[22] HAYS J, EFROS A A. IM2GPS:Estimating Geographic Information from a Single Image[C]//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage:IEEE, 2008:1-8.
[23] YAZAWA N, UCHIYAMA H, SAITO H, et al. Image Based View Localization System Retrieving from a Panorama Database by SURF[C]//Proceedings of 2009 IAPR Conference on Machine Vision Applications. Yokohama, Japan:[s.n.], 2009:3632-3636.
[24] KE Yan, SUKTHANKAR R. PCA-SIFT:A More Distinctive Representation for Local Image Descriptors[C]//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington DC:IEEE, 2004:506-513.
[25] PARK H, MOON K S. Fast Feature Matching by Coarse-to-fine Comparison of Rearranged SURF Descriptors[J]. IEICE Transactions on Information and Systems, 2015, E98-D(1):210-213.
[26] KAMENCAY P, BREZNAN M, JELSOVKA D, et al. Improved Face Recognition Method Based on Segmentation Algorithm Using SIFT-PCA[C]//Proceedings of the 35th International Conference on Telecommunications and Signal Processing. Prague, Czech Republic:IEEE, 2012:758-762.
[27] TANG Gefu, XIAO Zhifeng, LIU Qing, et al. A Novel Airport Detection Method via Line Segment Classification and Texture Classification[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(12):2408-2412.
[28] XIAO Zhifeng, LIU Qing, TANG Gefu, et al. Elliptic Fourier Transformation-based Histograms of Oriented Gradients for Rotationally Invariant Object Detection in Remote-Sensing Images[J]. International Journal of Remote Sensing, 2015, 36(2):618-644.
[29] BLASCHKE T. Object Based Image Analysis for Remote Sensing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(1):2-16.
[30] SE S, NADEAU C, WOOD S. Automated UAV-based Video Exploitation Using Service Oriented Architecture Framework[C]//Proceedings of Volume 8020, Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications VⅢ. Orlando, FL:SPIE, 2011:80200Y.
[31] SZLÁVIK Z, SZIRÁNYI T, HAVASI L. Video Camera Registration Using Accumulated Co-motion Maps[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2007, 61(5):298-306.
[32] LI Yunpeng, SNAVELY N, HUTTENLOCHER D, et al. Worldwide Pose Estimation Using 3D Point Clouds[C]//Proceedings of the 12th European Conference on Computer Vision. Florence, Italy:Springer, 2012:15-29.
[33] SVÄRM L, ENQVIST O, OSKARSSON M, et al. Accurate Localization and Pose Estimation for Large 3D Models[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:532-539.
[34] GERNHARDT S, AUER S, EDER K. Persistent Scatterers at Building Facades-evaluation of Appearance and Localization Accuracy[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 100:92-105.
[35] SINGH G, KOŠECKÁ J. Semantically Guided Geo-location and Modeling in Urban Environments[M]//ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization. Cham:Springer, 2016:101-120.
[36] CRANDALL D J, LI Yunpeng, LEE S, et al. Recognizing Landmarks in Large-scale Social Image Collections[M]//ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization. Cham:Springer, 2016:121-144.
[37] NIEVERGELT J, HINTERBERGER H, SEVCIK K C. The Grid File:An Adaptable, Symmetric Multikey File Structure[J]. ACM Transactions on Database Systems (TODS), 1984, 9(1):38-71.
[38] ROBINSON J T. The K-D-B-Tree:A Search Structure for Large Multidimensional Dynamic Indexes[C]//Proceedings of 1981 ACM SIGMOD International Conference on Management of Data. Ann Arbor, Michigan:ACM, 1981:10-18.
[39] SNAVELY N, SEITZ S M, SZELISKI R. Modeling the World from Internet Photo Collections[J]. International Journal of Computer Vision, 2008, 80(2):189-210.
[40] WU Changchang. Towards Linear-time Incremental Structure from Motion[C]//Proceedings of 2013 International Conference on 3D Vision-3DV 2013. Seattle:IEEE, 2013:127-134.
[41] MATAS J, CHUM O, URBAN M, et al. Robust Wide-baseline Stereo from Maximally Stable Extremal Regions[J]. Image and Vision Computing, 2004, 22(10):761-767.
[42] 程亮, 龚健雅, 宋小刚, 等. 面向宽基线立体影像匹配的高质量仿射不变特征提取方法[J]. 测绘学报, 2008, 37(1):77-82. CHENG Liang, GONG Jianya, SONG Xiaogang, et al. A Method for Affine Invariant Feature Extraction with High Quality for Wide Baseline Stereo Image Matching[J]. Acta Geodaetica et Cartographica Sinica, 2008, 37(1):77-82.
[43] INDYK P, MOTWANI R. Approximate Nearest Neighbors:Towards Removing the Curse of Dimensionality[C]//Proceedings of the 30th Annual ACM Symposium on Theory of Computing. Dallas, Texas:ACM, 1998:604-613.
[44] ZHANG Zhengyou, DERICHE R, FAUGERAS O, et al. A Robust Technique for Matching Two Uncalibrated Images through the Recovery of the Unknown Epipolar Geometry[J]. Artificial Intelligence, 1995, 78(1-2):87-119.
[45] WESTOBY M J, BRASINGTON J, GLASSER N F, et al. ‘Structure-from-Motion’ Photogrammetry:A Low-cost, Effective Tool for Geoscience Applications[J]. Geomorphology, 2012, 179:300-314.
[46] LOURAKIS M I A, ARGYROS A A. SBA:A Software Package for Generic Sparse Bundle Adjustment[J]. ACM Transactions on Mathematical Software, 2009, 36(1):2.
[47] 刘颖真, 贾奋励, 万刚, 等. 非专业弱关联影像的地理配准及其精度评估[J]. 测绘学报, 2015, 44(9):1014-1021, 1028. DOI:10.11947/j.AGCS.2015.20140394. LIU Yingzhen, JIA Fenli, WAN Gang, et al. Geo-registration of Unprofessional and Weakly-related Image and Precision Evaluation[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(9):1014-1021, 1028. DOI:10.11947/j.AGCS.2015.20140394.
[48] 陈爱军, 徐光祐, 史元春. 基于城市航空立体像对的全自动3维建筑物建模[J]. 测绘学报, 2002, 31(1):54-59. CHEN Aijun, XU Guangyou, SHI Yuanchun. Automated 3D Building Modeling Based on Urban Aerial Stereopair[J]. Acta Geodaetica et Cartographica Sinica, 2002, 31(1):54-59.

Options

文章导航

模态框（Modal）标题

摘要

本文引用格式

Abstract

参考文献