互联网众源照片的三维重建定位技术

doi:10.11947/j.AGCS.2018.20170365

测绘学报 ›› 2018, Vol. 47 ›› Issue (5): 631-643.doi: 10.11947/j.AGCS.2018.20170365

互联网众源照片的三维重建定位技术

袁一^1,4, 程亮^1,2,3,4, 宗雯雯^1,4, 李舒怡^1,4, 李满春^1,2,4,5

1. 南京大学江苏省地理信息技术重点实验室, 江苏南京 210023;
2. 南京大学中国南海研究协同创新中心, 江苏南京 210023;
3. 南京大学软件新技术与产业化协同创新中心, 江苏南京 210023;
4. 南京大学地理与海洋科学学院, 江苏南京 210023;
5. 南京师范大学江苏省地理信息资源开发与利用协同创新中心, 江苏南京 210023

收稿日期:2017-06-28 修回日期:2018-03-01 出版日期:2018-05-20 发布日期:2018-06-01
通讯作者: 程亮 E-mail:lcheng@nju.edu.cn
作者简介:袁一(1993-),女,硕士生,研究方向为激光雷达与摄影测量数据集成处理与应用。E-mail:yuanyi_1213@163.com
基金资助:
国家重点研发计划（2017YFB0504205）；国家自然科学基金（41622109；41371017）

Crowd-sourced Pictures Geo-localization Method Based on 3D Reconstruction

YUAN Yi^1,4, CHENG Liang^1,2,3,4, ZONG Wenwen^1,4, LI Shuyi^1,4, LI Manchun^1,2,4,5

1. Jiangsu Provincial Key Laboratory of Geographic Information Science and Technology, Nanjing University, Nanjing 210023, China;
2. Collaborative Innovation Center for the South Sea Studies, Nanjing University, Nanjing 210023, China;
3. Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing University, Nanjing 210023, China;
4. School of Geographic and Oceanographic Sciences, Nanjing University, Nanjing 210023, China;
5. Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing Normal University, Nanjing 210023, China

Received:2017-06-28 Revised:2018-03-01 Online:2018-05-20 Published:2018-06-01
Supported by:
The National Key Research and Development Program (No.2017YFB0504205);The National Natural Science Foundation of China (Nos.41622109;41371017)

摘要/Abstract

摘要： 随着电子产品的普及和互联网发展，人们习惯性地将照片上传到主流图片分享网站和社交媒体。这些照片很多都没有地理坐标或者仅有模糊的位置信息。如果互联网上任意来源的电子照片能被恢复出真实地理位置，它们将可以应用在城市建设、城市景观分析和犯罪追踪等很多方面。本文提出了一种互联网众筹照片的三维重建定位技术，该技术将结构化组织的街景数据作为参考数据集，使用三步走策略：图像检索粗定位、图像匹配细筛选和三维重建精定位，来给不明来源的众筹照片附上精确的地理标签。本文通过摄影测量原理恢复待查询照片周围的三维空间信息，较之前Zamir和Shah的方法，定位精度中值从256.7 m提升到69.0 m，平均值从350.4 m提升到206.0 m，在50 m精度要求下的照片数量占比从17.2%提升到43.2%。另一个发现是在重建误差成因方面，待查询照片的相机距离主拍摄目标越近，定位总误差越小。本文所提出的照片定位技术提供了灵活的参数，使其应用范围不仅仅局限于小区域，也可以扩大至城市、国家尺度。

关键词: 街景数据, 照片定位, 图像检索, 三维重建, 精度分析

Abstract: People are increasingly becoming accustomed to taking photos of everyday life in modern cities and uploading them on major photo-sharing social media sites. These sites contain numerous pictures, but many have incomplete or blurred location information. The geo-localization of crowd-sourced pictures enriches the information contained therein, and is applicable to activities such as urban construction, urban landscape analysis, and crime tracking. However, geo-localization faces huge technical challenges. This paper proposes a method for large-scale geo-localization of crowd-sourced pictures. Our approach uses structured, organized Street View images as a reference dataset and employs a three-step strategy of coarse geo-localization by image retrieval, selecting reliable matches by image registration, and fine geo-localization by 3D reconstruction to attach geographic tags to pictures from unidentified sources. 3D reconstruction based on close-range photogrammetry is used to restore the 3D geographical information of the crowd-sourced pictures, resulting in the proposed method improving the median error from 256.7 m to 69.0 m, and the percentage of the geo-localized query pictures under a 50 m error requirement from 17.2% to 43.2% compared with the previous method. Another discovery of the proposed method is that, regarding the causes of reconstruction error, closer distances from the query cameras to the main objects in query pictures tend to produce smaller errors. The proposed method is not limited to small areas, and could be expanded to cities and larger areas owing to its flexible parameters.

Key words: street view images, geo-localization, image retrieval, 3D reconstruction, accuracy analysis

中图分类号:

P237

袁一, 程亮, 宗雯雯, 李舒怡, 李满春. 互联网众源照片的三维重建定位技术[J]. 测绘学报, 2018, 47(5): 631-643.

YUAN Yi, CHENG Liang, ZONG Wenwen, LI Shuyi, LI Manchun. Crowd-sourced Pictures Geo-localization Method Based on 3D Reconstruction[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(5): 631-643.

参考文献

[1] LI Songnian, DRAGICEVIC S, CASTRO F A, et al. Geospatial Big Data Handling Theory and Methods:A Review and Research Challenges[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 115:119-133.
[2] HEIPKE C. Crowdsourcing Geospatial Data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(6):550-557.
[3] ARTH C, PIRCHHEIM C, VENTURA J, et al. Instant Outdoor Localization and SLAM Initialization from 2.5D Maps[J]. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(11):1309-1318.
[4] YIN Li, CHENG Qimin, WANG Zhenxin, et al. ‘Big Data’ for Pedestrian Volume:Exploring the Use of Google Street View Images for Pedestrian Counts[J]. Applied Geography, 2015, 63:337-345.
[5] SALMEN J, HOUBEN S, SCHLIPSING M. Google Street View Images Support the Development of Vision-based Driver Assistance Systems[C]//Proceedings of 2012 IEEE Intelligent Vehicles Symposium. Alcala de Henares, Spain:IEEE, 2012:891-895.
[6] TSAI V J D. Traffic Sign Detection and Positioning from Google Street View Streamlines[C]//Proceedings of ASPRS 2015 Annual Conference. Tampa, Florida:ASPRS, 2015.
[7] LIU Yue, WANG Yongtian. AR-view:An Augmented Reality Device for Digital Reconstruction of Yuangmingyuan[C]//Proceedings of 2009 IEEE International Symposium on Mixed and Augmented Reality-arts, Media and Humanities. Orlando, FL:IEEE, 2009:3-7.
[8] BADLAND H M, OPIT S, WITTEN K, et al. Can Virtual Streetscape Audits Reliably Replace Physical Streetscape Audits?[J]. Journal of Urban Health, 2010, 87(6):1007-1016.
[9] LI Xiaojiang, ZHANG Chuanrong, LI Weidong, et al. Assessing Street-level Urban Greenery Using Google Street View and a Modified Green View Index[J]. Urban Forestry & Urban Greening, 2015, 14(3):675-685.
[10] YIN Li, WANG Zhenxin. Measuring Visual Enclosure for Street Walkability:Using Machine Learning Algorithms and Google Street View Imagery[J]. Applied Geography, 2016, 76:147-153.
[11] HARA K, AZENKOT S, CAMPBELL M, et al. Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Locations with Google Street View:An Extended Analysis[J]. ACM Transactions on Accessible Computing (TACCESS), 2015, 6(2):5.
[12] ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization[M]. Cham:Springer, 2016.
[13] LOWE D G. Distinctive Image Features from Scale-invariant Keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[14] SATTLER T, LEIBE B, KOBBELT L. Fast Image-based Localization Using Direct 2D-to-3D Matching[C]//Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona, Spain:IEEE, 2011:667-674.
[15] CHEN D M, BAATZ G, KÖSER K, et al. City-scale Landmark Identification on Mobile Devices[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO:IEEE, 2011:737-744.
[16] KNOPP J, SIVIC J, PAJDLA T. Avoiding Confusing Features in Place Recognition[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:748-761.
[17] NISTER D, STEWENIUS H. Scalable Recognition with a Vocabulary Tree[C]//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York:IEEE, 2006:2161-2168.
[18] ZAMIR A R, SHAH M. Accurate Image Localization Based on Google Maps Street View[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:255-268.
[19] ZAMIR A R, SHAH M. Image Geo-localization Based on Multiplenearest Neighbor Feature Matching Using Generalized Graphs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(8):1546-1558.
[20] SCHINDLER G, BROWN M, SZELISKI R. City-scale Location Recognition[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis:IEEE, 2007:1-7.
[21] KNOPP J, SIVIC J, PAJDLA T. Avoiding Confusing Features in Place Recognition[C]//Proceedings of the 11th European Conference on Computer Vision. Heraklion, Greece:Springer, 2010:748-761.
[22] HAYS J, EFROS A A. IM2GPS:Estimating Geographic Information from a Single Image[C]//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage:IEEE, 2008:1-8.
[23] YAZAWA N, UCHIYAMA H, SAITO H, et al. Image Based View Localization System Retrieving from a Panorama Database by SURF[C]//Proceedings of 2009 IAPR Conference on Machine Vision Applications. Yokohama, Japan:[s.n.], 2009:3632-3636.
[24] KE Yan, SUKTHANKAR R. PCA-SIFT:A More Distinctive Representation for Local Image Descriptors[C]//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington DC:IEEE, 2004:506-513.
[25] PARK H, MOON K S. Fast Feature Matching by Coarse-to-fine Comparison of Rearranged SURF Descriptors[J]. IEICE Transactions on Information and Systems, 2015, E98-D(1):210-213.
[26] KAMENCAY P, BREZNAN M, JELSOVKA D, et al. Improved Face Recognition Method Based on Segmentation Algorithm Using SIFT-PCA[C]//Proceedings of the 35th International Conference on Telecommunications and Signal Processing. Prague, Czech Republic:IEEE, 2012:758-762.
[27] TANG Gefu, XIAO Zhifeng, LIU Qing, et al. A Novel Airport Detection Method via Line Segment Classification and Texture Classification[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(12):2408-2412.
[28] XIAO Zhifeng, LIU Qing, TANG Gefu, et al. Elliptic Fourier Transformation-based Histograms of Oriented Gradients for Rotationally Invariant Object Detection in Remote-Sensing Images[J]. International Journal of Remote Sensing, 2015, 36(2):618-644.
[29] BLASCHKE T. Object Based Image Analysis for Remote Sensing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(1):2-16.
[30] SE S, NADEAU C, WOOD S. Automated UAV-based Video Exploitation Using Service Oriented Architecture Framework[C]//Proceedings of Volume 8020, Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications VⅢ. Orlando, FL:SPIE, 2011:80200Y.
[31] SZLÁVIK Z, SZIRÁNYI T, HAVASI L. Video Camera Registration Using Accumulated Co-motion Maps[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2007, 61(5):298-306.
[32] LI Yunpeng, SNAVELY N, HUTTENLOCHER D, et al. Worldwide Pose Estimation Using 3D Point Clouds[C]//Proceedings of the 12th European Conference on Computer Vision. Florence, Italy:Springer, 2012:15-29.
[33] SVÄRM L, ENQVIST O, OSKARSSON M, et al. Accurate Localization and Pose Estimation for Large 3D Models[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:532-539.
[34] GERNHARDT S, AUER S, EDER K. Persistent Scatterers at Building Facades-evaluation of Appearance and Localization Accuracy[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 100:92-105.
[35] SINGH G, KOŠECKÁ J. Semantically Guided Geo-location and Modeling in Urban Environments[M]//ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization. Cham:Springer, 2016:101-120.
[36] CRANDALL D J, LI Yunpeng, LEE S, et al. Recognizing Landmarks in Large-scale Social Image Collections[M]//ZAMIR A R, HAKEEM A, VAN GOOL L, et al. Large-scale Visual Geo-localization. Cham:Springer, 2016:121-144.
[37] NIEVERGELT J, HINTERBERGER H, SEVCIK K C. The Grid File:An Adaptable, Symmetric Multikey File Structure[J]. ACM Transactions on Database Systems (TODS), 1984, 9(1):38-71.
[38] ROBINSON J T. The K-D-B-Tree:A Search Structure for Large Multidimensional Dynamic Indexes[C]//Proceedings of 1981 ACM SIGMOD International Conference on Management of Data. Ann Arbor, Michigan:ACM, 1981:10-18.
[39] SNAVELY N, SEITZ S M, SZELISKI R. Modeling the World from Internet Photo Collections[J]. International Journal of Computer Vision, 2008, 80(2):189-210.
[40] WU Changchang. Towards Linear-time Incremental Structure from Motion[C]//Proceedings of 2013 International Conference on 3D Vision-3DV 2013. Seattle:IEEE, 2013:127-134.
[41] MATAS J, CHUM O, URBAN M, et al. Robust Wide-baseline Stereo from Maximally Stable Extremal Regions[J]. Image and Vision Computing, 2004, 22(10):761-767.
[42] 程亮, 龚健雅, 宋小刚, 等. 面向宽基线立体影像匹配的高质量仿射不变特征提取方法[J]. 测绘学报, 2008, 37(1):77-82. CHENG Liang, GONG Jianya, SONG Xiaogang, et al. A Method for Affine Invariant Feature Extraction with High Quality for Wide Baseline Stereo Image Matching[J]. Acta Geodaetica et Cartographica Sinica, 2008, 37(1):77-82.
[43] INDYK P, MOTWANI R. Approximate Nearest Neighbors:Towards Removing the Curse of Dimensionality[C]//Proceedings of the 30th Annual ACM Symposium on Theory of Computing. Dallas, Texas:ACM, 1998:604-613.
[44] ZHANG Zhengyou, DERICHE R, FAUGERAS O, et al. A Robust Technique for Matching Two Uncalibrated Images through the Recovery of the Unknown Epipolar Geometry[J]. Artificial Intelligence, 1995, 78(1-2):87-119.
[45] WESTOBY M J, BRASINGTON J, GLASSER N F, et al. ‘Structure-from-Motion’ Photogrammetry:A Low-cost, Effective Tool for Geoscience Applications[J]. Geomorphology, 2012, 179:300-314.
[46] LOURAKIS M I A, ARGYROS A A. SBA:A Software Package for Generic Sparse Bundle Adjustment[J]. ACM Transactions on Mathematical Software, 2009, 36(1):2.
[47] 刘颖真, 贾奋励, 万刚, 等. 非专业弱关联影像的地理配准及其精度评估[J]. 测绘学报, 2015, 44(9):1014-1021, 1028. DOI:10.11947/j.AGCS.2015.20140394. LIU Yingzhen, JIA Fenli, WAN Gang, et al. Geo-registration of Unprofessional and Weakly-related Image and Precision Evaluation[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(9):1014-1021, 1028. DOI:10.11947/j.AGCS.2015.20140394.
[48] 陈爱军, 徐光祐, 史元春. 基于城市航空立体像对的全自动3维建筑物建模[J]. 测绘学报, 2002, 31(1):54-59. CHEN Aijun, XU Guangyou, SHI Yuanchun. Automated 3D Building Modeling Based on Urban Aerial Stereopair[J]. Acta Geodaetica et Cartographica Sinica, 2002, 31(1):54-59.

互联网众源照片的三维重建定位技术

Crowd-sourced Pictures Geo-localization Method Based on 3D Reconstruction

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	叶发茂, 孟祥龙, 董萌, 聂运菊, 葛芸, 陈晓勇. 遥感图像蚁群算法和加权图像到类距离检索法[J]. 测绘学报, 2021, 50(5): 612-620.
[2]	邓非, 陈欣, 颜青松, 曲英杰. 带线约束的摄影测量网格变分精化算法[J]. 测绘学报, 2020, 49(4): 469-479.
[3]	张春森, 张会, 郭丙轩, 彭哲. 城市场景结构感知的网格模型简化算法[J]. 测绘学报, 2020, 49(3): 334-342.
[4]	邓非, 颜青松, 肖腾. 多视密集匹配并行传播GPU-PatchMatch算法[J]. 测绘学报, 2020, 49(2): 181-190.
[5]	朱艳, 颜青松, 曲英杰, 陈欣, 邓非. 影像一致性优化的视图选取策略[J]. 测绘学报, 2020, 49(11): 1463-1472.
[6]	朱永兴, 谭述森, 杜兰, 贾小林. 顾及粗差影响的全球电离层克里金插值及精度分析[J]. 测绘学报, 2019, 48(7): 840-848.
[7]	张一, 姜挺, 江刚武, 余岸竹, 于英. 特征法视觉SLAM逆深度滤波的三维重建[J]. 测绘学报, 2019, 48(6): 708-717.
[8]	范军, 李涛, 左小清, 陈乾福, 张祥, 禄競. 利用参数独立分解的星载SAR干涉测量检校方法[J]. 测绘学报, 2019, 48(6): 737-746.
[9]	尚洋, 孙晓亮, 张跃强, 李由, 于起峰. 三维目标位姿跟踪与模型修正[J]. 测绘学报, 2018, 47(6): 799-808.
[10]	房亚男, 辛景民, 曾光, 王家松, 李杰. 模糊度固定的北斗卫星多系统融合精密轨道确定[J]. 测绘学报, 2018, 47(3): 341-347.
[11]	王阅兵, 甘卫军, 陈为涛, 连尉平, 游新兆. 北斗导航系统精密单点定位在地壳运动监测中的应用分析[J]. 测绘学报, 2018, 47(1): 48-56.
[12]	赵传, 张保明, 陈小卫, 郭海涛, 卢俊. 一种利用点云邻域信息的建筑物屋顶面高精度自动提取方法[J]. 测绘学报, 2017, 46(9): 1123-1134.
[13]	汪捷, 何锡扬. 北斗卫星伪距多路径系统偏差的修正方法及其对单频PPP的影响分析[J]. 测绘学报, 2017, 46(7): 841-847.
[14]	朱宁宁. 柱面全景图像拼接方法的仿真分析[J]. 测绘学报, 2017, 46(4): 487-497.
[15]	李国元, 唐新明. 资源三号02星激光测高精度分析与验证[J]. 测绘学报, 2017, 46(12): 1939-1949.