Learning feature matching for UAV image sequences with significantly different viewpoints

doi:10.11947/j.AGCS.2023.20210472

Abstract

Abstract: Aiming at the problems of large affine deformation, serious occlusion, and obvious viewpoint difference, a method of robust matching is proposed to solve the problems of multiple solutions and a number of mismatches in UAV image sequences matching with significantly different viewpoints. First, the improved dual-head communication D2-Net convolutional neural network is used to extract the learning features of the image sequences. In the subsequent matching search stage of the corresponding image points, a coarse-to-fine matching purification strategy is designed to solve the problem that the unique matching point is interfered by many potential feasible points, which achieves the robust matching and greatly reduces matching time cost. The proposed algorithm was tested using multiple sets of sequence images of different scenes in the HPatches dataset and field-collected images with large different viewpoints, and compared with the representative ASIFT method based on the hand-crafted and some methods based on deep learning. The results show that the proposed method can extract robust affine invariant deep learning features, and has advantages in terms of the number of correct matching points, the correct rate of matching points, the RMSE of matching points and the cost of matching time.

Key words: image matching, affine transformation, deep learning feature, convolutional neural network, UAV

CLC Number:

P236

ZHANG Yongxian, MA Guorui, CUI Zhixiang, ZHANG Zhijun. Learning feature matching for UAV image sequences with significantly different viewpoints[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(2): 230-243.

References

[1] 李德仁, 肖雄武, 郭丙轩, 等. 倾斜影像自动空三及其在城市真三维模型重建中的应用[J]. 武汉大学学报(信息科学版), 2016, 41(6):711-721. LI Deren, XIAO Xiongwu, GUO Bingxuan, et al. Oblique image based automatic aerotriangulation and its application in 3D city model reconstruction[J]. Geomatics and Information Science of Wuhan University, 2016, 41(6):711-721.
[2] 张永生, 张振超, 童晓冲, 等. 地理空间智能研究进展和面临的若干挑战[J]. 测绘学报, 2021, 50(9):1137-1146.DOI:10.11947/j.AGCS.2021.20200420. ZHANG Yongsheng, ZHANG Zhenchao, TONG Xiaochong, et al. Progress and challenges of geospatial artificial intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(9):1137-1146.DOI:10.11947/j.AGCS.2021.20200420.
[3] 张卡, 宿东, 王蓬勃, 等. 深度学习技术在影像密集匹配方面的进展与应用[J]. 科学技术与工程, 2020, 20(30):12268-12278. ZHANG Ka, SU Dong, WANG Pengbo, et al. Progress and application of deep learning in image dense matching[J]. Science Technology and Engineering, 2020, 20(30):12268-12278.
[4] 赵红蕊, 陆胜寒. 基于特征尺度分布与对极几何约束的高清影像快速密集匹配方法[J]. 测绘学报, 2018, 47(6):790-798.DOI:10.11947/j.AGCS.2018.20170630. ZHAO Hongrui, LU Shenghan. Dense high-definition image matching strategy based on scale distribution of feature and geometric constraint[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):790-798.DOI:10.11947/j.AGCS.2018.20170630.
[5] 余美. 倾斜立体影像匹配若干问题研究[D]. 徐州:中国矿业大学, 2018. YU Mei. Research on several issues of oblique stereo image matching[D]. Xuzhou:China University of Mining and Technology, 2018.
[6] YANG Huachao, ZHANG Shubi, WANG Yongbo. Robust and precise registration of oblique images based on scale-invariant feature transformation algorithm[J]. IEEE Geoscience and Remote Sensing Letters, 2012, 9(4):783-787.
[7] ZHANG Qiang, WANG Yabin, WANG Long. Registration of images with affine geometric distortion based on maximally stable extremal regions and phase congruency[J]. Image and Vision Computing, 2015, 36:23-39.
[8] 肖雄武, 郭丙轩, 李德仁, 等. 一种具有仿射不变性的倾斜影像快速匹配方法[J]. 测绘学报, 2015, 44(4):414-421.DOI:10.11947/j.AGCS.2015.20140048. XIAO Xiongwu, GUO Bingxuan, LI Deren, et al. A quick and affine invariance matching method for oblique images[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(4):414-421.DOI:10.11947/j.AGCS.2015.20140048.
[9] 肖雄武, 李德仁, 郭丙轩, 等. 一种具有视点不变性的倾斜影像快速匹配方法[J]. 武汉大学学报(信息科学版), 2016, 41(9):1151-1159. XIAO Xiongwu, LI Deren, GUO Bingxuan, et al. A robust and rapid viewpoint-invariant matching method for oblique images[J]. Geomatics and Information Science of Wuhan University, 2016, 41(9):1151-1159.
[10] 姜三, 许志海, 张峰, 等. 面向无人机倾斜影像的高效SfM重建方案[J]. 武汉大学学报(信息科学版), 2019, 44(8):1153-1161. JIANG San, XU Zhihai, ZHANG Feng, et al. Solution for efficient SfM reconstruction of oblique UAV images[J]. Geomatics and Information Science of Wuhan University, 2019, 44(8):1153-1161.
[11] MOREL J M, YU Guoshen. ASIFT:a new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2):438-469.
[12] FAN Dazhao, DONG Yang, ZHANG Yongsheng. Satellite image matching method based on deep convolutional neural network[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):90-100.
[13] ŽBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Boston, MA, USA:IEEE, 2015:1592-1599.
[14] HAN Xufeng, LEUNG T, JIA Yangqing, et al. MatchNet:unifying feature and metric learning for patch-based matching[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA:IEEE, 2015:3279-3286.
[15] MISHCHUK A, MISHKIN D, RADENOVIC F, et al. Working hard to know your neighbor's margins:local descriptor learning loss[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York,NY,USA:ACM, 2017:4829-4840.
[16] DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint:self-supervised interest point detection and description[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Salt Lake City, UT, USA:IEEE, 2018:337-33712.
[17] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-net:a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach, CA, USA:IEEE, 2020:8084-8093.
[18] REVAUD J, WEINZAEPFEL P, SOUZA C D, et al. R2D2:repeatable and reliable detector and descriptor[C]//Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition workshop.[S.l.]:IEEE, 2019.
[19] JIANG San, JIANG Wanshou, GUO Bingxuan, et al. Learned local features for structure from motion of UAV images:a comparative evaluation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14:10583-10597.
[20] 张传辉, 姚国标, 张力, 等. 多分支网络联合的倾斜影像仿射不变特征匹配[J]. 测绘学报, 2021, 50(5):641-651. DOI:10.11947/j.AGCS.2021.20200506. ZHANG Chuanhui, YAO Guobiao, ZHANG Li, et al. Affine invariant feature matching of oblique images based on multi-branch network[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(5):641-651.DOI:10.11947/j.AGCS.2021.20200506.
[21] 蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2):189-202.DOI:10.11947/j.AGCS.2021.20200048. LAN Chaozhen, LU Wanjie, YU Junming, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):189-202.DOI:10.11947/j.AGCS.2021.20200048.
[22] SNAVELY N, SEITZ S M, SZELISKI R. Photo tourism:exploring photo collections in 3D[J]. ACM Transactions on Graphics, 2006, 25(3):835-846.
[23] SINGH PARIHAR U, GUJARATHI A, MEHTA K, et al. RoRD:rotation-robust descriptors and orthographic views for local feature matching[C]//Proceedings of 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Prague, Czech Republic:IEEE, 2021:1593-1600.
[24] 蓝朝桢, 阎晓东, 崔志祥, 等. 用于无人机自主绝对定位的实时特征匹配方法[J]. 测绘科学技术学报, 2020, 37(3):264-268, 274. LAN Chaozhen, YAN Xiaodong, CUI Zhixiang, et al. Real-time feature matching method for the autonomous absolute location of UAV[J]. Journal of Geomatics Science and Technology, 2020, 37(3):264-268, 274.
[25] LUO Zixin, ZHOU Lei, BAI Xuyang, et al. ASLFeat:learning local features of accurate shape and localization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA:IEEE, 2020:6588-6597.
[26] 张永生, 于英, 薛武. 无人机地理影像直播技术[M]. 北京:科学出版社:2020. ZHANG Yongsheng, YU Ying, XUE Wu. UAV geographic image live broadcast technology[M]. Beijing:Science Press, 2020.