Multi-modal image matching based on modality reconstruction and feature perturbation learning

doi:10.11947/j.AGCS.2026.20250297

Abstract

Abstract:

Multi-modal image matching is a crucial foundational task in multi-source remote sensing earth observation. Due to differences in imaging principles, spectral characteristics, temporal phases, and other factors, multi-modal images exhibit significant geometric distortions and nonlinear radiometric differences, which make the expression of common features and matching challenging. Existing methods mostly focus on forced alignment of multi-modal images in the feature space, failing to fully explore cross-modal transformation mapping relationships and lacking comprehensive consideration of interference factors in complex matching scenarios, thus limiting their robustness. To address this, we propose a multi-modal image matching method based on modality reconstruction and feature perturbation learning. First, a cross-modal local common feature expression model is constructed. Positive and negative samples are created using local regions of matched and mismatched points to implement feature contrastive learning. On the basis of supervision by radiometric differences in raw data, a perturbation sample augmentation supervision mechanism is introduced to drive the model to learn interference-resistant feature expressions. Then, a modality reconstruction decoding module is designed to reconstruct local features of one modal image into a pseudo-image of another modality. Additional supervision signals are provided by optimizing the correlation between the reconstructed image and the original image. Finally, through the above multi-objective training, the model can effectively extract local common features invariant to radiometric and geometric variations, thereby achieving accurate multi-modal image matching. Experiments on visible-infrared, visible-SAR, and other modal datasets demonstrate that the proposed method can effectively extract common features resistant to radiometric differences and geometric distortions. It outperforms current state-of-the-art methods in metrics such as reprojection error, success rate, and area under the curve. Moreover, it is verified to be suitable for scenarios with 0°~360° rotation angle differences, providing robust technical support for multi-source remote sensing collaborative tasks.

Key words: image matching, multi-modal image, deep learning, multi-source remote sensing

CLC Number:

P237

Tengfeng Tang, Renyuan Liu, Chang Liu, Yangang Zhao, Dan Pan, Yuanxin Ye. Multi-modal image matching based on modality reconstruction and feature perturbation learning[J]. Acta Geodaetica et Cartographica Sinica, 2026, 55(8): 1343-1356.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: http://xb.chinasmp.com/EN/10.11947/j.AGCS.2026.20250297

http://xb.chinasmp.com/EN/Y2026/V55/I8/1343

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Tab. 1

Tab. 2

Tab. 3

Tab. 4

Fig. 7

Fig. 8

Fig. 9

Tab. 5

Tab. 6

Tab. 7

References 33

[1]	朱柏, 叶沅鑫. 多模态遥感图像配准方法研究综述[J]. 中国图象图形学报, 2024, 29(8): 2137-2161.
	Zhu Bai, Ye Yuanxin. Multimodal remote sensing image registration: a survey[J]. Journal of Image and Graphics, 2024, 29(8): 2137-2161.
[2]	眭海刚, 刘畅, 干哲, 等. 多模态遥感图像匹配方法综述[J]. 测绘学报, 2022, 51(9): 1848-1861. DOI: . doi: 10.11947/j.AGCS.2022.20220126
	Sui Haigang, Liu Chang, Gan Zhe, et al. Overview of multi-modal remote sensing image matching methods[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(9): 1848-1861. DOI: . doi: 10.11947/j.AGCS.2022.20220126
[3]	Li Liangzhi, Han Ling, Ye Yuanxin, et al. Deep learning in remote sensing image matching: a survey[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2025, 225: 88-112.
[4]	唐霖峰, 张浩, 徐涵, 等. 基于深度学习的图像融合方法综述[J]. 中国图象图形学报, 2023, 28(1): 3-36.
	Tang Linfeng, Zhang Hao, Xu Han, et al. Deep learning-based image fusion: a survey[J]. Journal of Image and Graphics, 2023, 28(1): 3-36.
[5]	Ye Yuanxin, Wang Mengmeng, Zhou Liang, et al. Adjacent-level feature cross-fusion with 3D CNN for remote sensing image change detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5618214.
[6]	蔺小虎, 杨鑫, 姚顽强, 等. 面向复杂地下空间的多源传感器数据动态加权融合SLAM方法[J]. 测绘学报, 2025, 54(3): 523-535. DOI: . doi: 10.11947/j.AGCS.2025.20230586
	Lin Xiaohu, Yang Xin, Yao Wanqiang, et al. A dynamic weighted fusion SLAM method using multi-source sensor data in complex underground spaces[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(3): 523-535. DOI: . doi: 10.11947/j.AGCS.2025.20230586
[7]	Emery W J, Baldwin D, Matthews D. Maximum cross correlation automatic satellite image navigation and attitude corrections for open-ocean image navigation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2003, 41(1): 33-42.
[8]	Suri S, Reinartz P. Mutual-information-based registration of TerraSAR-X and Ikonos imagery in urban areas[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(2): 939-949.
[9]	Ye Y X, Bruzzone L, Shan J, et al. Fast and robust matching for multimodal remote sensing image registration[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(11): 9059-9070.
[10]	樊仲藜, 张力, 王庆栋, 等. SAR影像和光学影像梯度方向加权的快速匹配方法[J]. 测绘学报, 2021, 50(10): 1390-1403. DOI: . doi: 10.11947/j.AGCS.2021.20200587
	Fan Zhongli, Zhang Li, Wang Qingdong, et al. A fast matching method of SAR and optical images using angular weighted orientated gradients[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(10): 1390-1403. DOI: . doi: 10.11947/j.AGCS.2021.20200587
[11]	南轲, 齐华, 叶沅鑫. 深度卷积特征表达的多模态遥感影像模板匹配方法[J]. 测绘学报, 2019, 48(6): 727-736. DOI: . doi: 10.11947/j.AGCS.2019.20180432
	Nan Ke, Qi Hua, Ye Yuanxin. A template matching method of multimodal remote sensing images based on deep convolutional feature representation[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 727-736. DOI: . doi: 10.11947/j.AGCS.2019.20180432
[12]	Yang Chao, Gong Guoqing, Liu Chang, et al. RMSO-ConvNeXt: a lightweight CNN network for robust SAR and optical image matching under strong noise interference[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5208013.
[13]	Zhou Liang, Ye Yuanxin, Tang Tengfeng, et al. Robust matching for SAR and optical images using multiscale convolutional gradient features[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 4017605.
[14]	李加元. 鲁棒性遥感影像特征匹配关键问题研究[D]. 武汉: 武汉大学, 2018.
	Li Jiayuan. Research on key techniques of robust remote sensing image feature matching[D]. Wuhan: Wuhan University, 2018.
[15]	Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[16]	Li Jiayuan, Hu Qingwu, Ai Mingyao. RIFT: multi-modal image matching based on radiation-variation insensitive feature transform[J]. IEEE Transactions on Image Processing, 2020, 29: 3296-3310.
[17]	Wan Genyi, Ye Zhen, Xu Yusheng, et al. Multimodal remote sensing image matching based on weighted structure saliency feature[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 4700816.
[18]	Mishchuk A, Mishkin D, Radenovic F, et al. Working hard to know your neighbor's margins: local descriptor learning loss[C]//Proceedings of the 31st Annual Conference on Neural Information Processing Systems. Long Beach: NIPS Foundation, 2017: 4827-4838.
[19]	Tian Yurun, Barroso-Laguna A, Ng T, et al. HyNet: learning local descriptor with hybrid similarity measure and triplet loss[C]//Proceedings of the 34th Conference on Neural Information Processing Systems. [S.l.]: NIPS Foundation, 2020.
[20]	Detone D, Malisiewicz T, Rabinovich A. SuperPoint: self-supervised interest point detection and description[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE, 2018: 337-349.
[21]	Dusmanu M, Rocco I, Pajdla T, et al. D2-Net: a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 8084-8093.
[22]	蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2): 189-202. DOI: . doi: 10.11947/j.AGCS.2021.20200048
	Lan Chaozhen, Lu Wanjie, Yu Junming, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2): 189-202. DOI: . doi: 10.11947/j.AGCS.2021.20200048
[23]	Sarlin P E, Detone D, Malisiewicz T, et al. SuperGlue: learning feature matching with graph neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 4937-4946.
[24]	Lindenberger P, Sarlin P, Pollefeys M. LightGlue: local feature matching at light speed[C]//Proceedings of 2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 17581-17592.
[25]	Sun Jiaming, Shen Zehong, Wang Yuang, et al. LoFTR: detector-free local feature matching with Transformers[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 8918-8927.
[26]	Wang Yifan, He Xingyi, Peng Sida, et al. Efficient LoFTR: semi-dense local feature matching with sparse-like speed[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 21666-21675.
[27]	Tuzcuoğlu Ö, Köksal A, Sofu B, et al. XoFTR: cross-modal feature matching transformer[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle: IEEE, 2024: 4275-4286.
[28]	Edstedt J, Sun Qiyu, Bökman G, et al. RoMa: robust dense feature matching[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 19790-19800.
[29]	Ren Jiangwei, Jiang Xingyu, Li Zizhuo, et al. MINIMA: modality invariant image matching[C]//Proceedings of 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Tennessee: IEEE, 2025: 23059-23068.
[30]	Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[31]	Sun Yiming, Cao Bing, Zhu Pengfei, et al. Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(10): 6700-6713.
[32]	Xiang Yuming, Tao Rongshu, Wang Feng, et al. Automatic registration of optical and SAR images via improved phase congruency model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 5847-5861.
[33]	Chum O, Werner T, Matas J. Two-view geometry estimation unaffected by a dominant plane[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego: IEEE, 2005: 772-779.

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.82	28.42	45.87	68.39
RIFT	2.51	24.22	40.83	62.33
WSSF	2.02	25.63	43.61	66.04
LightGlue	2.12	25.30	42.78	65.55
LoFTR	1.97	25.75	43.19	65.63
EfficientLoFTR	2.14	24.86	42.32	65.15
XoFTR	1.93	26.16	43.70	66.14

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.86	22.17	43.49	68.08
RIFT	5.54	8.14	18.14	37.48
WSSF	4.68	8.37	18.51	39.93
LightGlue	2.42	18.87	38.00	63.55
LoFTR	3.17	14.62	30.96	55.74
EfficientLoFTR	2.63	13.25	31.86	59.61
XoFTR	2.44	14.81	34.65	61.54

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.59	45.18	64.21	80.26
RIFT	6.45	15.39	27.79	45.43
WSSF	8.26	6.03	13.62	30.74
LightGlue	3.22	35.53	53.57	70.49
LoFTR	3.34	30.57	49.45	68.64
EfficientLoFTR	2.92	24.54	45.87	69.49
XoFTR	2.12	40.64	59.28	77.26

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.34	41.89	60.91	78.82
RIFT	6.31	12.39	24.33	45.21
WSSF	8.97	4.37	11.87	29.92
LightGlue	3.11	32.44	44.97	57.81
LoFTR	4.34	19.79	38.45	59.81
EfficientLoFTR	1.98	27.63	48.72	71.50
XoFTR	1.63	41.27	60.22	77.70

消融方案	描述	RMSE/像素	AUC/（%）
消融方案	描述	RMSE/像素	3像素	5像素	10像素
1	Baseline	2.47	19.03	40.87	63.24
2	Baseline+特征扰动	2.15	22.41	41.37	65.34
3	Baseline+模态重构	1.93	22.80	42.33	66.80
4	本文方法	1.86	22.17	43.49	68.08