模态重构与特征扰动学习相结合的多模态图像匹配方法

doi:10.11947/j.AGCS.2026.20250297

测绘学报 ›› 2026, Vol. 55 ›› Issue (8): 1343-1356.doi: 10.11947/j.AGCS.2026.20250297

• 影像大地测量前沿技术余智慧防灾创新应用 • 上一篇

模态重构与特征扰动学习相结合的多模态图像匹配方法

唐腾峰¹(), 刘人源¹, 刘畅¹, 赵彦刚², 潘丹³, 叶沅鑫¹()

^1.西南交通大学地球科学与工程学院，四川　成都　611756
^2.自然资源部第二地形测量队，陕西　西安　710054
^3.北京微纳星空科技股份有限公司，北京　100089

收稿日期:2025-07-24 修回日期:2025-12-05 发布日期:2026-09-09
通讯作者: 叶沅鑫 E-mail:ttf@my.swjtu.edu.cn;yeyuanxin@home.swjtu.edu.cn
作者简介:唐腾峰（2000—），男，博士生，研究方向为多源遥感图像匹配。E-mail：ttf@my.swjtu.edu.cn
基金资助:
国家重点研发计划(2024YFC3015402);国家自然科学基金(42271446);天津市科技计划项目(24YFYSHZ00080)

Multi-modal image matching based on modality reconstruction and feature perturbation learning

Tengfeng Tang¹(), Renyuan Liu¹, Chang Liu¹, Yangang Zhao², Dan Pan³, Yuanxin Ye¹()

^1.Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China
^2.The Second Topographic Surveying Brigade of Ministry Natural Resource, Xi'an 710054, China
^3.Beijing Weina Starry Sky Technology Co., Ltd., Beijing 100089, China

Received:2025-07-24 Revised:2025-12-05 Published:2026-09-09
Contact: Yuanxin Ye E-mail:ttf@my.swjtu.edu.cn;yeyuanxin@home.swjtu.edu.cn
About author:Tang Tengfeng (2000—), male, PhD candidate, majors in multi-source remote sensing image matching. E-mail: ttf@my.swjtu.edu.cn
Supported by:
The National Key Research and Development Program of China(2024YFC3015402);The National Natural Science Foundation of China(42271446);Tianjin Science and Technology Plan Project(24YFYSHZ00080)

摘要/Abstract

摘要：

多模态图像匹配是多源遥感对地观测的重要基础工作。由于成像原理、光谱特性和时相等因素的差异，多模态图像间具有显著的几何畸变和非线性辐射差异，导致共性特征表达和匹配困难。现有方法多聚焦于多模态图像在特征空间强制对齐，未充分挖掘跨模态的转换映射关系，且缺乏对复杂匹配场景干扰因素的综合考虑，稳健性受限。为此，本文提出模态重构与特征扰动学习相结合的多模态图像匹配方法。首先，构建跨模态局部共性特征表达模型，通过匹配点与不匹配点的局部区域构造正负样本，实现特征对比学习，并在原始数据辐射差异监督基础上，引入扰动样本增强监督机制，驱动模型学习具备抗干扰的特征表达；然后，设计模态重构解码模块，将一种模态图像的局部特征重构为另一模态的伪图像，通过优化重构图像与原始图像的相关性提供额外监督信号；最后，通过上述多目标训练，模型能够有效提取辐射与几何不变的局部共性特征，进而实现精确的多模态图像匹配。在可见光-红外、可见光-SAR等模态数据集上的试验表明，本文方法能够有效提取抗辐射差异与几何畸变的共性特征，在重投影误差、成功率及曲线下面积等指标上优于当前先进方法，且验证了适用于0°~360°旋转角度差异场景，为多源遥感协同任务提供强稳健性技术支撑。

关键词: 图像匹配, 多模态图像, 深度学习, 多源遥感

Abstract:

Multi-modal image matching is a crucial foundational task in multi-source remote sensing earth observation. Due to differences in imaging principles, spectral characteristics, temporal phases, and other factors, multi-modal images exhibit significant geometric distortions and nonlinear radiometric differences, which make the expression of common features and matching challenging. Existing methods mostly focus on forced alignment of multi-modal images in the feature space, failing to fully explore cross-modal transformation mapping relationships and lacking comprehensive consideration of interference factors in complex matching scenarios, thus limiting their robustness. To address this, we propose a multi-modal image matching method based on modality reconstruction and feature perturbation learning. First, a cross-modal local common feature expression model is constructed. Positive and negative samples are created using local regions of matched and mismatched points to implement feature contrastive learning. On the basis of supervision by radiometric differences in raw data, a perturbation sample augmentation supervision mechanism is introduced to drive the model to learn interference-resistant feature expressions. Then, a modality reconstruction decoding module is designed to reconstruct local features of one modal image into a pseudo-image of another modality. Additional supervision signals are provided by optimizing the correlation between the reconstructed image and the original image. Finally, through the above multi-objective training, the model can effectively extract local common features invariant to radiometric and geometric variations, thereby achieving accurate multi-modal image matching. Experiments on visible-infrared, visible-SAR, and other modal datasets demonstrate that the proposed method can effectively extract common features resistant to radiometric differences and geometric distortions. It outperforms current state-of-the-art methods in metrics such as reprojection error, success rate, and area under the curve. Moreover, it is verified to be suitable for scenarios with 0°~360° rotation angle differences, providing robust technical support for multi-source remote sensing collaborative tasks.

Key words: image matching, multi-modal image, deep learning, multi-source remote sensing

中图分类号:

P237

唐腾峰, 刘人源, 刘畅, 赵彦刚, 潘丹, 叶沅鑫. 模态重构与特征扰动学习相结合的多模态图像匹配方法[J]. 测绘学报, 2026, 55(8): 1343-1356.

Tengfeng Tang, Renyuan Liu, Chang Liu, Yangang Zhao, Dan Pan, Yuanxin Ye. Multi-modal image matching based on modality reconstruction and feature perturbation learning[J]. Acta Geodaetica et Cartographica Sinica, 2026, 55(8): 1343-1356.

导出引用管理器 EndNote|Reference Manager|ProCite|BibTeX|RefWorks

链接本文: http://xb.chinasmp.com/CN/10.11947/j.AGCS.2026.20250297

http://xb.chinasmp.com/CN/Y2026/V55/I8/1343

图/表 16

图1

图2

图3

图4

图5

图6

表1

表2

表3

表4

图7

图8

图9

表5

表6

表7

参考文献 33

[1]	朱柏, 叶沅鑫. 多模态遥感图像配准方法研究综述[J]. 中国图象图形学报, 2024, 29(8): 2137-2161.
	Zhu Bai, Ye Yuanxin. Multimodal remote sensing image registration: a survey[J]. Journal of Image and Graphics, 2024, 29(8): 2137-2161.
[2]	眭海刚, 刘畅, 干哲, 等. 多模态遥感图像匹配方法综述[J]. 测绘学报, 2022, 51(9): 1848-1861. DOI: . doi: 10.11947/j.AGCS.2022.20220126
	Sui Haigang, Liu Chang, Gan Zhe, et al. Overview of multi-modal remote sensing image matching methods[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(9): 1848-1861. DOI: . doi: 10.11947/j.AGCS.2022.20220126
[3]	Li Liangzhi, Han Ling, Ye Yuanxin, et al. Deep learning in remote sensing image matching: a survey[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2025, 225: 88-112.
[4]	唐霖峰, 张浩, 徐涵, 等. 基于深度学习的图像融合方法综述[J]. 中国图象图形学报, 2023, 28(1): 3-36.
	Tang Linfeng, Zhang Hao, Xu Han, et al. Deep learning-based image fusion: a survey[J]. Journal of Image and Graphics, 2023, 28(1): 3-36.
[5]	Ye Yuanxin, Wang Mengmeng, Zhou Liang, et al. Adjacent-level feature cross-fusion with 3D CNN for remote sensing image change detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5618214.
[6]	蔺小虎, 杨鑫, 姚顽强, 等. 面向复杂地下空间的多源传感器数据动态加权融合SLAM方法[J]. 测绘学报, 2025, 54(3): 523-535. DOI: . doi: 10.11947/j.AGCS.2025.20230586
	Lin Xiaohu, Yang Xin, Yao Wanqiang, et al. A dynamic weighted fusion SLAM method using multi-source sensor data in complex underground spaces[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(3): 523-535. DOI: . doi: 10.11947/j.AGCS.2025.20230586
[7]	Emery W J, Baldwin D, Matthews D. Maximum cross correlation automatic satellite image navigation and attitude corrections for open-ocean image navigation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2003, 41(1): 33-42.
[8]	Suri S, Reinartz P. Mutual-information-based registration of TerraSAR-X and Ikonos imagery in urban areas[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(2): 939-949.
[9]	Ye Y X, Bruzzone L, Shan J, et al. Fast and robust matching for multimodal remote sensing image registration[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(11): 9059-9070.
[10]	樊仲藜, 张力, 王庆栋, 等. SAR影像和光学影像梯度方向加权的快速匹配方法[J]. 测绘学报, 2021, 50(10): 1390-1403. DOI: . doi: 10.11947/j.AGCS.2021.20200587
	Fan Zhongli, Zhang Li, Wang Qingdong, et al. A fast matching method of SAR and optical images using angular weighted orientated gradients[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(10): 1390-1403. DOI: . doi: 10.11947/j.AGCS.2021.20200587
[11]	南轲, 齐华, 叶沅鑫. 深度卷积特征表达的多模态遥感影像模板匹配方法[J]. 测绘学报, 2019, 48(6): 727-736. DOI: . doi: 10.11947/j.AGCS.2019.20180432
	Nan Ke, Qi Hua, Ye Yuanxin. A template matching method of multimodal remote sensing images based on deep convolutional feature representation[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 727-736. DOI: . doi: 10.11947/j.AGCS.2019.20180432
[12]	Yang Chao, Gong Guoqing, Liu Chang, et al. RMSO-ConvNeXt: a lightweight CNN network for robust SAR and optical image matching under strong noise interference[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5208013.
[13]	Zhou Liang, Ye Yuanxin, Tang Tengfeng, et al. Robust matching for SAR and optical images using multiscale convolutional gradient features[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 4017605.
[14]	李加元. 鲁棒性遥感影像特征匹配关键问题研究[D]. 武汉: 武汉大学, 2018.
	Li Jiayuan. Research on key techniques of robust remote sensing image feature matching[D]. Wuhan: Wuhan University, 2018.
[15]	Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[16]	Li Jiayuan, Hu Qingwu, Ai Mingyao. RIFT: multi-modal image matching based on radiation-variation insensitive feature transform[J]. IEEE Transactions on Image Processing, 2020, 29: 3296-3310.
[17]	Wan Genyi, Ye Zhen, Xu Yusheng, et al. Multimodal remote sensing image matching based on weighted structure saliency feature[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 4700816.
[18]	Mishchuk A, Mishkin D, Radenovic F, et al. Working hard to know your neighbor's margins: local descriptor learning loss[C]//Proceedings of the 31st Annual Conference on Neural Information Processing Systems. Long Beach: NIPS Foundation, 2017: 4827-4838.
[19]	Tian Yurun, Barroso-Laguna A, Ng T, et al. HyNet: learning local descriptor with hybrid similarity measure and triplet loss[C]//Proceedings of the 34th Conference on Neural Information Processing Systems. [S.l.]: NIPS Foundation, 2020.
[20]	Detone D, Malisiewicz T, Rabinovich A. SuperPoint: self-supervised interest point detection and description[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE, 2018: 337-349.
[21]	Dusmanu M, Rocco I, Pajdla T, et al. D2-Net: a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 8084-8093.
[22]	蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2): 189-202. DOI: . doi: 10.11947/j.AGCS.2021.20200048
	Lan Chaozhen, Lu Wanjie, Yu Junming, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2): 189-202. DOI: . doi: 10.11947/j.AGCS.2021.20200048
[23]	Sarlin P E, Detone D, Malisiewicz T, et al. SuperGlue: learning feature matching with graph neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 4937-4946.
[24]	Lindenberger P, Sarlin P, Pollefeys M. LightGlue: local feature matching at light speed[C]//Proceedings of 2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 17581-17592.
[25]	Sun Jiaming, Shen Zehong, Wang Yuang, et al. LoFTR: detector-free local feature matching with Transformers[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 8918-8927.
[26]	Wang Yifan, He Xingyi, Peng Sida, et al. Efficient LoFTR: semi-dense local feature matching with sparse-like speed[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 21666-21675.
[27]	Tuzcuoğlu Ö, Köksal A, Sofu B, et al. XoFTR: cross-modal feature matching transformer[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle: IEEE, 2024: 4275-4286.
[28]	Edstedt J, Sun Qiyu, Bökman G, et al. RoMa: robust dense feature matching[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 19790-19800.
[29]	Ren Jiangwei, Jiang Xingyu, Li Zizhuo, et al. MINIMA: modality invariant image matching[C]//Proceedings of 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Tennessee: IEEE, 2025: 23059-23068.
[30]	Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[31]	Sun Yiming, Cao Bing, Zhu Pengfei, et al. Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(10): 6700-6713.
[32]	Xiang Yuming, Tao Rongshu, Wang Feng, et al. Automatic registration of optical and SAR images via improved phase congruency model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 5847-5861.
[33]	Chum O, Werner T, Matas J. Two-view geometry estimation unaffected by a dominant plane[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego: IEEE, 2005: 772-779.

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.82	28.42	45.87	68.39
RIFT	2.51	24.22	40.83	62.33
WSSF	2.02	25.63	43.61	66.04
LightGlue	2.12	25.30	42.78	65.55
LoFTR	1.97	25.75	43.19	65.63
EfficientLoFTR	2.14	24.86	42.32	65.15
XoFTR	1.93	26.16	43.70	66.14

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.86	22.17	43.49	68.08
RIFT	5.54	8.14	18.14	37.48
WSSF	4.68	8.37	18.51	39.93
LightGlue	2.42	18.87	38.00	63.55
LoFTR	3.17	14.62	30.96	55.74
EfficientLoFTR	2.63	13.25	31.86	59.61
XoFTR	2.44	14.81	34.65	61.54

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.59	45.18	64.21	80.26
RIFT	6.45	15.39	27.79	45.43
WSSF	8.26	6.03	13.62	30.74
LightGlue	3.22	35.53	53.57	70.49
LoFTR	3.34	30.57	49.45	68.64
EfficientLoFTR	2.92	24.54	45.87	69.49
XoFTR	2.12	40.64	59.28	77.26

方法	RMSE/像素	AUC/（%）
方法	RMSE/像素	3像素	5像素	10像素
本文方法	1.34	41.89	60.91	78.82
RIFT	6.31	12.39	24.33	45.21
WSSF	8.97	4.37	11.87	29.92
LightGlue	3.11	32.44	44.97	57.81
LoFTR	4.34	19.79	38.45	59.81
EfficientLoFTR	1.98	27.63	48.72	71.50
XoFTR	1.63	41.27	60.22	77.70

消融方案	描述	RMSE/像素	AUC/（%）
消融方案	描述	RMSE/像素	3像素	5像素	10像素
1	Baseline	2.47	19.03	40.87	63.24
2	Baseline+特征扰动	2.15	22.41	41.37	65.34
3	Baseline+模态重构	1.93	22.80	42.33	66.80
4	本文方法	1.86	22.17	43.49	68.08

模态重构与特征扰动学习相结合的多模态图像匹配方法

Multi-modal image matching based on modality reconstruction and feature perturbation learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 33

相关文章 15

编辑推荐

Metrics

本文评价

局部区域尺寸/像素	RMSE/像素	AUC/（%）
局部区域尺寸/像素	RMSE/像素	3像素	5像素	10像素
64	8.59	5.43	11.89	20.74
96	3.22	20.01	36.41	56.05
128	2.35	20.56	40.47	64.89
160	1.86	22.17	43.49	68.08
192	2.03	19.42	41.34	67.38
224	2.17	18.79	39.95	66.60

方法	耗时/s
RIFT	3.823 5
WSSF	13.945 4
LightGlue	0.342 5
LoFTR	0.326 9
E-LoFTR	0.248 7
XoFTR	0.607 3
本文方法	1.102 7

[1]	李贺, 陈克杰, 崔文峰, 柴海山, 方荣新, 彭朝勇, 孙丽. 基于北斗三号PPP-B2b同震位移与深度学习的震级快速估算——以2025年定日Mw 7.1和曼德勒Mw 7.7地震为例[J]. 测绘学报, 2026, 55(7): 1171-1182.
[2]	赵曦, 袁强强, 徐佳丹. 面向复杂海洋环境的侧扫声呐目标样本生成与增强方法[J]. 测绘学报, 2026, 55(7): 1229-1239.
[3]	曾路兵, 李建胜, 王安成, 杨子迪, 高宇宁, 张嘉杰. 复杂光照环境下融合轻量级光照增强网络的视觉SLAM方法[J]. 测绘学报, 2026, 55(7): 1278-1292.
[4]	时天东, 赵玲, 赵文豪, 齐霁, 崔浩, 彭程里, 张新长. 时空信息显式引导的高分光学遥感影像可控生成方法[J]. 测绘学报, 2026, 55(5): 894-908.
[5]	李鹏, 张家涵, 汪志翰, 王厚杰, 李振洪. 潮间带地形重建方法综述：现状、挑战与趋势[J]. 测绘学报, 2026, 55(4): 571-587.
[6]	王家耀, 陈琳, 程士源, 王利军, 熊思奇. 人工智能赋能地图科学数智化[J]. 测绘学报, 2026, 55(3): 381-389.
[7]	禄小敏, 张志义, 闫浩文, 何毅, 苏小宁. 融合深度图信息最大化和多层感知机的建筑物群组模式识别方法[J]. 测绘学报, 2026, 55(3): 425-438.
[8]	季顺平, 刘瑾, 高建, 龚健雅. 多视影像深度学习密集匹配三维重建智能框架[J]. 测绘学报, 2025, 54(9): 1633-1646.
[9]	张继贤, 顾海燕, 倪欢, 李海涛, 杨懿, 丁少鹏, 隋淞蔓. 遥感智能变化检测的深度学习方法：演变与发展趋势[J]. 测绘学报, 2025, 54(8): 1347-1370.
[10]	方帅, 刘加恩, 张晶. 自适应参考特征引入与多尺度特征聚合的时空融合算法[J]. 测绘学报, 2025, 54(8): 1476-1488.
[11]	孟妮娜, 李凤梅, 周校东. 数据与认知双驱动的建筑物群制图综合结果与尺度一致性识别[J]. 测绘学报, 2025, 54(7): 1318-1331.
[12]	王亚青, 王中辉. 异构图卷积网络支持下的河系自动选取方法[J]. 测绘学报, 2025, 54(7): 1332-1345.
[13]	安晓亚, 郭伟茹, 张鹏鑫, 李欣欣, 石磊. 顾及几何位置和移动特征相似性的船舶轨迹聚类方法[J]. 测绘学报, 2025, 54(6): 1107-1121.
[14]	王超, 陈天宇, 张同, AhmedTanvir, 纪立强, 谢涛, 杨佳俊, 王帅. 基于全局差分增强模块和平衡惩罚损失的多源光学遥感影像变化检测[J]. 测绘学报, 2025, 54(5): 873-887.
[15]	罗卿莉, 李雪岩, 黄国满, 陈红辉, 薛铭龙, 李健. AOSN：α-最优网络模型的山区单通道SAR高程重建方法[J]. 测绘学报, 2025, 54(5): 888-898.