面向大视角差的无人机影像序列学习型特征匹配

doi:10.11947/j.AGCS.2023.20210472

摘要/Abstract

摘要： 针对无人机大视角差影像之间存在仿射变形大、遮挡严重、视角差异显著等问题导致的同名点匹配存在多解和大量误匹配难题，本文提出了一种适用于大视角差影像稳健匹配方法。利用改进的具有双头通信机制的D2-Net卷积神经网络提取倾斜影像的学习型特征，在之后的同名点匹配搜索阶段，为解决唯一匹配点受到较多潜在可行解干扰的问题，设计了一种由粗到精的提纯策略，在稳健匹配同名点对的同时大幅降低匹配开销成本。将HPatches数据集中多组不同场景的影像序列和实地采集的无人机大视角差影像序列作为数据源对提出的方法进行测试，并与具有代表性的基于手工设计的ASIFT方法和基于深度学习的多种方法进行了比较。结果表明，本文方法能够提取稳健的大视角差影像序列仿射不变学习型特征，在正确匹配点数、匹配点正确率、匹配点均方根误差和匹配时间开销方面具有优势。

关键词: 影像匹配, 仿射变换, 深度学习特征, 卷积神经网络, 无人机

Abstract: Aiming at the problems of large affine deformation, serious occlusion, and obvious viewpoint difference, a method of robust matching is proposed to solve the problems of multiple solutions and a number of mismatches in UAV image sequences matching with significantly different viewpoints. First, the improved dual-head communication D2-Net convolutional neural network is used to extract the learning features of the image sequences. In the subsequent matching search stage of the corresponding image points, a coarse-to-fine matching purification strategy is designed to solve the problem that the unique matching point is interfered by many potential feasible points, which achieves the robust matching and greatly reduces matching time cost. The proposed algorithm was tested using multiple sets of sequence images of different scenes in the HPatches dataset and field-collected images with large different viewpoints, and compared with the representative ASIFT method based on the hand-crafted and some methods based on deep learning. The results show that the proposed method can extract robust affine invariant deep learning features, and has advantages in terms of the number of correct matching points, the correct rate of matching points, the RMSE of matching points and the cost of matching time.

Key words: image matching, affine transformation, deep learning feature, convolutional neural network, UAV

中图分类号:

P236

张永显, 马国锐, 崔志祥, 张志军. 面向大视角差的无人机影像序列学习型特征匹配[J]. 测绘学报, 2023, 52(2): 230-243.

ZHANG Yongxian, MA Guorui, CUI Zhixiang, ZHANG Zhijun. Learning feature matching for UAV image sequences with significantly different viewpoints[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(2): 230-243.

参考文献

[1] 李德仁, 肖雄武, 郭丙轩, 等. 倾斜影像自动空三及其在城市真三维模型重建中的应用[J]. 武汉大学学报(信息科学版), 2016, 41(6):711-721. LI Deren, XIAO Xiongwu, GUO Bingxuan, et al. Oblique image based automatic aerotriangulation and its application in 3D city model reconstruction[J]. Geomatics and Information Science of Wuhan University, 2016, 41(6):711-721.
[2] 张永生, 张振超, 童晓冲, 等. 地理空间智能研究进展和面临的若干挑战[J]. 测绘学报, 2021, 50(9):1137-1146.DOI:10.11947/j.AGCS.2021.20200420. ZHANG Yongsheng, ZHANG Zhenchao, TONG Xiaochong, et al. Progress and challenges of geospatial artificial intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(9):1137-1146.DOI:10.11947/j.AGCS.2021.20200420.
[3] 张卡, 宿东, 王蓬勃, 等. 深度学习技术在影像密集匹配方面的进展与应用[J]. 科学技术与工程, 2020, 20(30):12268-12278. ZHANG Ka, SU Dong, WANG Pengbo, et al. Progress and application of deep learning in image dense matching[J]. Science Technology and Engineering, 2020, 20(30):12268-12278.
[4] 赵红蕊, 陆胜寒. 基于特征尺度分布与对极几何约束的高清影像快速密集匹配方法[J]. 测绘学报, 2018, 47(6):790-798.DOI:10.11947/j.AGCS.2018.20170630. ZHAO Hongrui, LU Shenghan. Dense high-definition image matching strategy based on scale distribution of feature and geometric constraint[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):790-798.DOI:10.11947/j.AGCS.2018.20170630.
[5] 余美. 倾斜立体影像匹配若干问题研究[D]. 徐州:中国矿业大学, 2018. YU Mei. Research on several issues of oblique stereo image matching[D]. Xuzhou:China University of Mining and Technology, 2018.
[6] YANG Huachao, ZHANG Shubi, WANG Yongbo. Robust and precise registration of oblique images based on scale-invariant feature transformation algorithm[J]. IEEE Geoscience and Remote Sensing Letters, 2012, 9(4):783-787.
[7] ZHANG Qiang, WANG Yabin, WANG Long. Registration of images with affine geometric distortion based on maximally stable extremal regions and phase congruency[J]. Image and Vision Computing, 2015, 36:23-39.
[8] 肖雄武, 郭丙轩, 李德仁, 等. 一种具有仿射不变性的倾斜影像快速匹配方法[J]. 测绘学报, 2015, 44(4):414-421.DOI:10.11947/j.AGCS.2015.20140048. XIAO Xiongwu, GUO Bingxuan, LI Deren, et al. A quick and affine invariance matching method for oblique images[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(4):414-421.DOI:10.11947/j.AGCS.2015.20140048.
[9] 肖雄武, 李德仁, 郭丙轩, 等. 一种具有视点不变性的倾斜影像快速匹配方法[J]. 武汉大学学报(信息科学版), 2016, 41(9):1151-1159. XIAO Xiongwu, LI Deren, GUO Bingxuan, et al. A robust and rapid viewpoint-invariant matching method for oblique images[J]. Geomatics and Information Science of Wuhan University, 2016, 41(9):1151-1159.
[10] 姜三, 许志海, 张峰, 等. 面向无人机倾斜影像的高效SfM重建方案[J]. 武汉大学学报(信息科学版), 2019, 44(8):1153-1161. JIANG San, XU Zhihai, ZHANG Feng, et al. Solution for efficient SfM reconstruction of oblique UAV images[J]. Geomatics and Information Science of Wuhan University, 2019, 44(8):1153-1161.
[11] MOREL J M, YU Guoshen. ASIFT:a new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2):438-469.
[12] FAN Dazhao, DONG Yang, ZHANG Yongsheng. Satellite image matching method based on deep convolutional neural network[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):90-100.
[13] ŽBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Boston, MA, USA:IEEE, 2015:1592-1599.
[14] HAN Xufeng, LEUNG T, JIA Yangqing, et al. MatchNet:unifying feature and metric learning for patch-based matching[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA:IEEE, 2015:3279-3286.
[15] MISHCHUK A, MISHKIN D, RADENOVIC F, et al. Working hard to know your neighbor's margins:local descriptor learning loss[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York,NY,USA:ACM, 2017:4829-4840.
[16] DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint:self-supervised interest point detection and description[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Salt Lake City, UT, USA:IEEE, 2018:337-33712.
[17] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-net:a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach, CA, USA:IEEE, 2020:8084-8093.
[18] REVAUD J, WEINZAEPFEL P, SOUZA C D, et al. R2D2:repeatable and reliable detector and descriptor[C]//Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition workshop.[S.l.]:IEEE, 2019.
[19] JIANG San, JIANG Wanshou, GUO Bingxuan, et al. Learned local features for structure from motion of UAV images:a comparative evaluation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14:10583-10597.
[20] 张传辉, 姚国标, 张力, 等. 多分支网络联合的倾斜影像仿射不变特征匹配[J]. 测绘学报, 2021, 50(5):641-651. DOI:10.11947/j.AGCS.2021.20200506. ZHANG Chuanhui, YAO Guobiao, ZHANG Li, et al. Affine invariant feature matching of oblique images based on multi-branch network[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(5):641-651.DOI:10.11947/j.AGCS.2021.20200506.
[21] 蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2):189-202.DOI:10.11947/j.AGCS.2021.20200048. LAN Chaozhen, LU Wanjie, YU Junming, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):189-202.DOI:10.11947/j.AGCS.2021.20200048.
[22] SNAVELY N, SEITZ S M, SZELISKI R. Photo tourism:exploring photo collections in 3D[J]. ACM Transactions on Graphics, 2006, 25(3):835-846.
[23] SINGH PARIHAR U, GUJARATHI A, MEHTA K, et al. RoRD:rotation-robust descriptors and orthographic views for local feature matching[C]//Proceedings of 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Prague, Czech Republic:IEEE, 2021:1593-1600.
[24] 蓝朝桢, 阎晓东, 崔志祥, 等. 用于无人机自主绝对定位的实时特征匹配方法[J]. 测绘科学技术学报, 2020, 37(3):264-268, 274. LAN Chaozhen, YAN Xiaodong, CUI Zhixiang, et al. Real-time feature matching method for the autonomous absolute location of UAV[J]. Journal of Geomatics Science and Technology, 2020, 37(3):264-268, 274.
[25] LUO Zixin, ZHOU Lei, BAI Xuyang, et al. ASLFeat:learning local features of accurate shape and localization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA:IEEE, 2020:6588-6597.
[26] 张永生, 于英, 薛武. 无人机地理影像直播技术[M]. 北京:科学出版社:2020. ZHANG Yongsheng, YU Ying, XUE Wu. UAV geographic image live broadcast technology[M]. Beijing:Science Press, 2020.