Deep learning algorithm for feature matching of cross modality remote sensing images

doi:10.11947/j.AGCS.2021.20200048

Abstract

Abstract: Focusing on the problem of difficulty in matching due to the differences in imaging modality, time phases, and resolutions of cross modality remote sensing images, a new deep learning feature matching method named CMM-Net is proposed. First, a convolutional neural network is used to extract high-dimensional feature maps of the cross modality remote sensing images. The key points are selected according to the conditions that both the channel maximum and local maximum are met, and the 512-dimensional descriptors in corresponding location are extracted on the feature map to complete the feature extraction. In the matching stage, after completing the fast-nearest neighbor searching, in order to solve the problem of lots of mismatched points, a purification algorithm with dynamic adaptive Euclidean distance and RANSAC constraints is proposed to ensure that the mismatches are effectively eliminated while retaining the correct matches. The algorithm was tested using multiple sets of cross modality remote sensing images and compared with other algorithms. The results show that the proposed algorithm has the ability to extract similar scale invariant features in cross modality images, and has strong adaptability and robustness.

Key words: deep learning, image matching, cross modality image, convolution neural network, satellite image

CLC Number:

P236

LAN Chaozhen, LU Wanjie, YU Junming, XU Qing. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2): 189-202.

References

[1] ZHAO Y, JIANG M, MA Z F. Integration of SAR polarimetric features and multi-spectral data for object-based land cover classification[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(4):64-72.
[2] 李德仁. 展望大数据时代的地球空间信息学[J]. 测绘学报, 2016, 45(4):379-384. DOI:10.11947/j.AGCS.2016. 20160057. LI Deren. Towards geo-spatial information science in big data era[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(4):379-384. DOI:10.11947/j.AGCS.2016.20160057.
[3] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[4] 戴激光, 宋伟东, 李玉. 渐进式异源光学卫星影像SIFT匹配方法[J]. 测绘学报, 2014, 43(7):746-752. DOI:10.13485/j.cnki.11-2089.2014.00. DAI Jiguang, SONG Weidong, LI Yu. Progressive SIFT matching algorithm for multi-source optical satellite images[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(7):746-752. DOI:10.13485/j.cnki.11-2089.2014.00.
[5] 王峰, 尤红建, 傅兴玉, 等. 应用于多源SAR图像匹配的级联SIFT算法[J]. 电子学报, 2016, 44(3):548-554. WANG Feng, YOU Hongjian, FU Xingyu, et al. Cascade SIFT matching method for multi-source SAR images[J]. Acta Electronica Sinica, 2016, 44(3):548-554.
[6] 叶沅鑫, 单杰, 熊金鑫, 等. 一种结合SIFT和边缘信息的多源遥感影像匹配方法[J]. 武汉大学学报(信息科学版), 2013, 38(10):1148-1151, 1260. YE Yuanxin, SHAN Jie, XIONG Jinxin, et al. A node localization method in wireless sensor network based on k-means cluster[J]. Geomatics and Information Science of Wuhan University, 2013, 38(10):1148-1151, 1260.
[7] 王瑞瑞, 马建文, 陈雪. 多传感器影像配准中基于虚拟匹配窗口的SIFT算法[J]. 武汉大学学报(信息科学版), 2011, 36(2):163-166. WANG Ruirui, MA Jianwen, CHEN Xue. SIFT algorithm based on visual matching window for registration between multi-sensor imagery[J]. Geomatics and Information Science of Wuhan University, 2011, 36(2):163-166.
[8] DELLINGER F, DELON Julie, GOUSSEAU Y, et al. SAR-SIFT:a SIFT-like algorithm for SAR images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(1):453-466.
[9] 凌志刚, 梁彦, 程咏梅, 等. 一种稳健的多源遥感图像特征配准方法[J]. 电子学报, 2010, 38(12):2892-2897. LING Zhigang, LIANG Yan, CHENG Yongmei, et al. A robust multi-source remote-sensing image registration method based on feature matching[J]. Acta Electronica Sinica, 2010, 38(12):2892-2897.
[10] KOVESI P. Image features from phase congruency[J]. Journal of Computer Vision Research, 1999, 1(3):1-26.
[11] WONG A, CLAUSI D A. ARRSI:automatic registration of remote-sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 45(5):1483-1493.
[12] 陈敏, 朱庆, 朱军, 等. 多光谱遥感影像亮度空间相位一致性特征点检测[J]. 测绘学报, 2016, 45(2):178-185. DOI:10.11947/j.AGCS.2016.20150030. CHEN Min, ZHU Qing, ZHU Jun, et al. Interest point detection for multispectral remote sensing image using phase congruency in illumination space[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(2):178-185. DOI:10.11947/j.AGCS.2016.20150030.
[13] 叶沅鑫, 单杰, 彭剑威, 等. 利用局部自相似进行多光谱遥感图像自动配准[J]. 测绘学报, 2014, 43(3):268-275. DOI:10.13485/j.cnki.11-2089.2014.0039. YE Yuanxin, SHAN Jie, PENG Jianwei, et al. Automated multispectral remote sensing image registration using local self-similarity[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(3):268-275. DOI:10.13485/j.cnki.11-2089.2014.0039.
[14] 闫利, 王紫琦, 叶志云. 顾及灰度和梯度信息的多模态影像配准算法[J]. 测绘学报, 2018, 47(1):71-81. DOI:10.11947/j.AGCS.2018.20170368. YAN Li, WANG Ziqi, YE Zhiyun. Multimodal image registration algorithm considering grayscale and gradient information[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(1):71-81. DOI:10.11947/j.AGCS.2018.20170368.
[15] LI Jiayuan, HU Qingwu, AI Mingyao. RIFT:multi-modal image matching based on radiation-variation insensitive feature transform[J]. IEEE Transactions on Image Processing, 2020, 29:3296-3310.
[16] DOSOVITSKIY A, FISCHER P, SPRINGENBERG J T, et al. Discriminative unsupervised feature learning with exemplar convolutional neural networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(9):1734-1747.
[17] 龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640. GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640.
[18] ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA:IEEE, 2015:4353-4361.
[19] 范大昭, 董杨, 张永生. 卫星影像匹配的深度卷积神经网络方法[J]. 测绘学报, 2018, 47(6):844-853. DOI:10.11947/j.AGCS.2018.20170627. FAN Dazhao, DONG Yang, ZHANG Yongsheng. Satellite image matching method based on deep convolution neural network[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):844-853. DOI:10.11947/j.AGCS.2018.20170627.
[20] YANG Zhuoqian, DAN Tingting, YANG Yang. Multi-temporal remote sensing image registration using deep convolutional features[J]. IEEE Access, 2018, 6:38544-38555.
[21] YE Famao, SU Yanfei, XIAO Hui, et al. Remote sensing image registration using convolutional neural network features[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(2):232-236.
[22] GONG J Y, JI S P. Photogrammetry and deep learning[J]. Journal of Geodesy and Geoinformation Science, 2018, 1(1):1-15.
[23] YI K M, TRULLS E, LEPETIT V, et al. LIFT:learned invariant feature transform[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, Netherlands:Springer, 2016:467-483.
[24] DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint:self-supervised interest point detection and description[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City, UT:IEEE, 2018:224-236.
[25] NOH H, ARAUJO A, SIM J, et al. Large-scale image retrieval with attentive deep local features[C]//2017 IEEE International Conference on Computer Vision (ICCV). Venice:IEEE, 2017:3456-3465.
[26] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-Net:a trainable CNN for joint description and detection of local features[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA:IEEE, 2019.
[27] LUO Zixin, SHEN Tianwei, ZHOU Lei, et al. ContextDesc:local descriptor augmentation with cross-modality context[C]//The Processdings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA:IEEE, 2019:2522-2531.
[28] KAREN Simonyan, REW Zisserman. Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the International Conference on Learning Representation. San-diego, California, USA:[s.n.],2014.
[29] MISHCHUK A, MISHKIN D, RADENOVIĆ F, et al. Working hard to know your neighbor's margins:local descriptor learning loss[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach:Curran Associates Inc., 2017.
[30] LI Zhengqi, SNAVELY N. MegaDepth:learning single-view depth prediction from internet photos[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT:IEEE, 2018.
[31] SCHÖNBERGER J L, FRAHM J M. Structure-from-motion revisited[C]//The Processdings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV:IEEE, 2016:4104-4113.