Multi-source remote sensing image bidirectional consistent registration based on learning feature

doi:10.11947/j.AGCS.2023.20220412

Abstract

Abstract: A robust registration method with bidirectional consistent transformation is proposed to solve the problem of multi-source remote sensing image relatively poor registration effect due to large nonlinear radiation and geometric distortion. First, the fine-tuned ResNet101 network model was utilized to extract images learning features. To improve the reliability of the corresponding feature matching, we designed a bidirectional consistency feature matching model in the feature matching stage. Then, robust registration was achieved by using a parametric regression transformation model with weight based on a small lightweight network. In our experiments, we tested the proposed algorithm using Google Earth images, satellite images, UAV images and Google Earth-satellite-UAV images, and compared it with the several typical methods. The results show that the proposed method has advantages in the accuracy, efficiency and robustness, and achieves automatic registration accuracy almost within 2 pixels.

Key words: learning feature, feature matching, multi-source images, image registration, bidirectional consistent transformation

CLC Number:

P236

ZHANG Yongxian, MA Guorui, ZI Shuanjin, MEN Hang. Multi-source remote sensing image bidirectional consistent registration based on learning feature[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(11): 1906-1916.

References

[1] 冯蕊涛, 杜清运, 罗恒, 等. 基于光流校正的复杂地形区多时相遥感影像配准[J]. 遥感学报, 2021, 25(2):630-640. FENG Ruitao, DU Qingyun, LUO Heng, et al. A registration algorithm based on optical flow modification for multi-temporal remote sensing images covering the complex-terrain region[J]. National Remote Sensing Bulletin, 2021, 25(2):630-640.
[2] TONG Xinyi, XIA Guisong, HU Fan, et al. Exploiting deep features for remote sensing image retrieval:a systematic investigation[J]. IEEE Transactions on Big Data, 2020, 6(3):507-521.
[3] CHENG Jian, LENG Cong, WU Jiaxiang, et al. Fast and accurate image matching with cascade hashing for 3D reconstruction[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus:IEEE, 2014:1-8.
[4] BRUNNER D, LEMOINE G, BRUZZONE L. Earthquake damage assessment of buildings using VHR optical and SAR imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(5):2403-2420.
[5] ZHANG Xu, YU F X, KARAMAN S, et al. Learning discriminative and transformation covariant local feature detectors[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu:IEEE, 2017:4923-4931.
[6] FAN Bin, HUO Chunlei, PAN Chunhong, et al. Registration of optical and SAR satellite images by exploring the spatial relationship of the improved SIFT[J]. IEEE Geoscience and Remote Sensing Letters, 2013, 10(4):657-661.
[7] XIANG Yuming, WANG Feng, YOU Hongjian. OS-SIFT:a robust SIFT-like algorithm for high-resolution optical-to-SAR image registration in suburban areas[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(6):3078-3090.
[8] YU Qiuze, NI Dawen, JIANG Yuxuan, et al. Universal SAR and optical image registration via a novel SIFT framework based on nonlinear diffusion and a polar spatial-frequency descriptor[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171:1-17.
[9] ZHANG Wannan. Combination of SIFT and canny edge detection for registration between SAR and optical images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19:1-5.
[10] 李培, 姜刚, 马千里, 等. 结合张量与互信息的混合模型多模态图像配准方法[J]. 测绘学报, 2021, 50(7):916-929. DOI:10.11947/j.AGCS.2021.20200492. LI Pei, JIANG Gang, MA Qianli, et al. A hybrid model combining tensor and mutual information for multi-modal image registration[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(7):916-929. DOI:10.11947/j.AGCS.2021.20200492.
[11] 叶沅鑫, 单杰, 彭剑威, 等. 利用局部自相似进行多光谱遥感图像自动配准[J]. 测绘学报, 2014, 43(3):268-275. YE Yuanxin, SHAN Jie, PENG Jianwei, et al. Automated multispectral remote sensing image registration using local self-similarity[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(3):268-275.
[12] YE Yuanxin, BRUZZONE L, SHAN Jie, et al. Fast and robust matching for multimodal remote sensing image registration[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(11):9059-9070.
[13] YE Yuanxin, SHAN Jie, BRUZZONE L, et al. Robust registration of multimodal remote sensing images based on structural similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5):2941-2958.
[14] 张力, 刘玉轩, 孙洋杰, 等. 数字航空摄影三维重建理论与技术发展综述[J]. 测绘学报, 2022, 51(7):1437-1457. DOI:10.11947/j.AGCS.2022.20220130. ZHANG Li, LIU Yuxuan, SUN Yangjie, et al. A review of developments in the theory and technology of three-dimensional reconstruction in digital aerial photogrammetry[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7):1437-1457. DOI:10.11947/j.AGCS.2022.20220130.
[15] 蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2):189-202. DOI:10.11947/j.AGCS.2021.20200048. LAN Chaozhen, LU Wanjie, YU Junming, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):189-202. DOI:10.11947/j.AGCS.2021.20200048.
[16] FAN Rongbo, HOU Bochuan, LIU Jinbao, et al. Registration of multiresolution remote sensing images based on L2-siamese model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 14:237-248.
[17] ZHANG Han, NI Weiping, YAN Weidong, et al. Registration of multimodal remote sensing image based on deep fully convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(8):3028-3042.
[18] BÜRGMANN T, KOPPE W, SCHMITT M. Matching of TerraSAR-X derived ground control points to optical image patches using deep learning[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 158:241-248.
[19] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-net:a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach:IEEE, 2020:8084-8093.
[20] SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue:learning feature matching with graph neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle:IEEE, 2020:4938-4947.
[21] CHEN Hongkai, LUO Zixin, ZHANG Jiahui, et al. Learning to match features with seeded graph matching network[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal:IEEE, 2022:6281-6290.
[22] WANG Shuang, QUAN Dou, LIANG Xuefeng, et al. A deep learning framework for remote sensing image registration[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 145:148-164.
[23] HUGHES L H, MARCOS D, LOBRY S, et al. A deep learning framework for matching of SAR and optical imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 169:166-179.
[24] 朱庆涛, 殷君君, 曾亮, 等. 基于邻域一致性的极化SAR图像仿射配准[J]. 雷达学报, 2021, 10(1):49-60. ZHU Qingtao, YIN Junjun, ZENG Liang, et al. Polarimetric SAR image affine registration based on neighborhood consensus[J]. Journal of Radars, 2021, 10(1):49-60.
[25] ROCCO I, ARANDJELOVIC' R, SIVIC J. Convolutional neural network architecture for geometric matching[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11):2553-2567.
[26] PARK J H, NAM W J, LEE S W. A two-stream symmetric network with bidirectional ensemble for aerial image matching[J]. Remote Sensing, 2020, 12(3):465.
[27] LI Jiayuan, HU Qingwu, AI Mingyao. RIFT:multi-modal image matching based on radiation-variation insensitive feature transform[J]. IEEE Transactions on Image Processing, 2020, 29:3296-3310.