基于SAR与光学遥感影像融合的多标签场景分类方法

doi:10.11947/j.AGCS.2025.20240281

测绘学报 ›› 2025, Vol. 54 ›› Issue (5): 911-923.doi: 10.11947/j.AGCS.2025.20240281

基于SAR与光学遥感影像融合的多标签场景分类方法

赵一鸣¹(), 胡克林², 涂可龙¹, 卿雅娴³, 杨超², 祁昆仑¹^,²(), 吴华意³

^1.中国地质大学（武汉）地理与信息工程学院，湖北　武汉　430078
^2.国家地理信息系统工程技术研究中心，湖北　武汉　430078
^3.武汉大学测绘遥感信息工程全国重点实验室，湖北　武汉　430079

收稿日期:2024-07-10 修回日期:2025-03-20 出版日期:2025-06-23 发布日期:2025-06-23
通讯作者: 祁昆仑 E-mail:zym805805@cug.edu.cn;qikunlun@cug.edu.cn
作者简介:赵一鸣（1999—），男，硕士生，研究方向为多模态遥感数据融合。　E-mail：zym805805@cug.edu.cn
基金资助:
湖北省科技厅重大专项(2020AAA004);湖北珞珈实验室专项基金(220100034)

Multi-label scene classification method based on fusion of SAR and optical remote sensing images

Yiming ZHAO¹(), Kelin HU², Kelong TU¹, Yaxian QING³, Chao YANG², Kunlun QI¹^,²(), Huayi WU³

^1.School of Geography and Information Engineering, China University of Geosciences, Wuhan 430078, China
^2.National Research Center for Geographic Information System Engineering Technology, Wuhan 430078, China
^3.State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

Received:2024-07-10 Revised:2025-03-20 Online:2025-06-23 Published:2025-06-23
Contact: Kunlun QI E-mail:zym805805@cug.edu.cn;qikunlun@cug.edu.cn
About author:ZHAO Yiming (1999—), male, postgraduate, majors in multi-modal remote sensing image fusion.　E-mail: zym805805@cug.edu.cn
Supported by:
Hubei Key Research, and Development Program in China(2020AAA004);Hubei Lujia Laboratory Special Fund(220100034)

摘要/Abstract

摘要：

深度卷积神经网络已被证实是高分辨率遥感影像场景分类中最有效的方法之一。过去的研究大多关注于单一光学遥感影像的场景级分类，并且多为单标签分类。然而，单一光学遥感影像容易受到天气条件的限制，并且单标签的标注难以全面描述复杂的图像内容。因此，本文利用欧洲空间局于2020年获取的SAR和光学遥感图像，构建了武汉市多模态多标签场景分类数据集SEN12-MLRS，并设计了一种基于并行双注意力融合网络（PDANet）的多标签场景分类方法。PDANet通过双分支特征提取、自适应特征融合及多级特征融合，实现了光学和SAR图像的多模态与多层级的特征融合。试验结果表明，在SEN12-MLRS数据集上，PDANet相较于多种先进模型取得了最佳性能，并通过消融试验进一步验证了本文方法的有效性。

关键词: 多模态遥感影像融合, 注意力机制, 多标签分类, 特征融合

Abstract:

Deep convolutional neural networks have proven to be one of the most effective methods for scene classification of high-resolution remote sensing images. Most previous studies focus on scene-level classification of single optical remote sensing images and are primarily limited to single-label classification. However, single optical remote sensing images are often constrained by weather conditions, and single-label annotations cannot fully describe complex image contents. Therefore, in this paper, we constructed a multimodal, multi-label scene classification dataset called SEN12-MLRS, using SAR and optical remote sensing images acquired by the European Space Agency in 2020. We proposed a parallel dual attention fusion network (PDANet) for multi-label scene classification. PDANet achieves optical and SAR image feature extraction as well as multi-modal and multilevel feature fusion through two-branch feature extraction, adaptive feature fusion, and multilevel feature fusion. Experimental results demonstrate that PDANet achieves superior performance compared to many state-of-the-art models on the SEN12-MLRS dataset. The effectiveness of the proposed network and its modules is further validated through ablation experiments.

Key words: multi-modal remote sensing image fusion, attention mechanism, multi-label classification, feature fusion

中图分类号:

P236

赵一鸣, 胡克林, 涂可龙, 卿雅娴, 杨超, 祁昆仑, 吴华意. 基于SAR与光学遥感影像融合的多标签场景分类方法[J]. 测绘学报, 2025, 54(5): 911-923.

Yiming ZHAO, Kelin HU, Kelong TU, Yaxian QING, Chao YANG, Kunlun QI, Huayi WU. Multi-label scene classification method based on fusion of SAR and optical remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(5): 911-923.

图/表 12

图1

图2

图3

图4

图5

图6

表1

表2

表3

表4

表5

表6

参考文献 47

[1]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vega: IEEE, 2016: 770-778.
[2]	TAN Mingxing, LE Q V. EfficientNet: rethinking model scaling for convolutional neural networks[EB/OL]. [2024-02-03]. https://arxiv.org/abs/1905.11946.
[3]	LIU Zhuang, MAO Hanzi, WU Chaoyuan, et al. A ConvNet for the 2020s[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 11966-11976.
[4]	白坤, 慕晓冬, 陈雪冰, 等. 融合半监督学习的无监督遥感影像场景分类[J]. 测绘学报, 2022, 51(5): 691-702. DOI . doi: 10.11947/j.AGCS.2022.20210270
	BAI Kun, MU Xiaodong, CHEN Xuebing, et al. Unsupervised remote sensing image scene classification based on semi-supervised learning[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(5): 691-702. DOI . doi: 10.11947/j.AGCS.2022.20210270
[5]	李彦胜, 吴敏郎, 张永军. 知识图谱约束深度网络的高分辨率遥感影像场景分类[J]. 测绘学报, 2024, 53(4): 677-688. DOI . doi: 10.11947/j.AGCS.2024.20230125
	LI Yansheng, WU Minlang, ZHANG Yongjun. Knowledge graph-guided deep network for high-resolution remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(4): 677-688. DOI . doi: 10.11947/j.AGCS.2024.20230125
[6]	ZHANG Wei, TANG Ping, ZHAO Lijun. Remote sensing image scene classification using CNN-CapsNet[J]. Remote Sensing, 2019, 11(5): 494.
[7]	ZHANG Xinchang, SHI Qian, SUN Ying, et al. The review of land use/land cover mapping AI methodology and application in the era of remote sensing big data[J]. Journal of Geodesy and Geoinformation Science, 2024, 7(3):1-23.
[8]	尹国应, 张洪艳, 张良培. 2001—2019年长江中下游农业干旱遥感监测及植被敏感性分析[J]. 武汉大学学报(信息科学版), 2022, 47(8): 1245-1256.
	YIN Guoying, ZHANG Hongyan, ZHANG Liangpei. Remote sensing monitoring of agricultural drought in the middle and lower reaches of the Yangtze River and vegetation sensitivity analysis from 2001 to 2019[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1245-1256.
[9]	YUAN Gang, WANG Yunjia, ZHAO Feng, et al. Spatiotemporal correlation characteristics between thermal infrared remote sensing obtained surface thermal anomalies and reconstructed 4D temperature fields of underground coal fires[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-18.
[10]	LI Yansheng, ZHANG Yongjun, ZHU Zhihui. Error-tolerant deep learning for remote sensing image scene classification[J]. IEEE Transactions on Cybernetics, 2021, 51(4): 1756-1768.
[11]	BOUTELL M R, LUO Jiebo, SHEN Xipeng, et al. Learning multi-label scene classification[J]. Pattern Recognition, 2004, 37(9): 1757-1771.
[12]	DAI O E, DEMIR B, SANKUR B, et al. A novel system for content-based retrieval of single and multi-label high-dimensional remote sensing images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(7): 2473-2490.
[13]	QI Kunlun, WU Huayi, SHEN Chen, et al. Land-use scene classification in high-resolution remote sensing images using improved correlatons[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(12): 2403-2407.
[14]	LOWE D G. Distinctive image features from scale-invariant key points[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[15]	HARALICK R M, SHANMUGAM K, DINSTEIN I. Textural features for image classification[J]. IEEE Transactions on Systems, Man, and Cybernetics, 1973, SMC-3(6): 610-621.
[16]	JAIN A K, RATHA N K, LAKSHMANAN S. Object detection using Gabor filters[J]. Pattern Recognition, 1997, 30(2): 295-309.
[17]	OJALA T, PIETIKAINEN M, MAENPAA T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7): 971-987.
[18]	SWAIN M J, BALLARD D H. Color indexing[J]. International Journal of Computer Vision, 1991, 7(1): 11-32.
[19]	DALAL N, TRIGGS B. Histograms of oriented gradients for human detection[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego: IEEE, 2005: 886-893.
[20]	ZHU Qiqi, SUN Xiongli, ZHONG Yanfei, et al. High-resolution remote sensing image scene understanding: a review[C]//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE, 2019: 3061-3064.
[21]	CHENG Gong, XIE Xingxing, HAN Junwei, et al. Remote sensing image scene classification meets deep learning: challenges, methods, benchmarks, and opportunities[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 3735-3756.
[22]	PENATTI O A B, NOGUEIRA K, DOS SANTOS J A. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston: IEEE, 2015: 44-51.
[23]	龚希, 吴亮, 谢忠, 等. 融合全局和局部深度特征的高分辨率遥感影像场景分类方法[J]. 光学学报, 2019, 39(3): 19-29.
	GONG Xi, WU Liang, XIE Zhong, et al. Classification method of high-resolution remote sensing scenes based on fusion of global and local deep features[J]. Acta Optica Sinica, 2019, 39(3): 19-29.
[24]	LU Xiaoqiang, SUN Hao, ZHENG Xiangtao. A feature aggregation convolutional neural network for remote sensing scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(10): 7894-7906.
[25]	王梨名, 祁昆仑, 杨超, 等. 弱监督尺度自适应增强的高分辨率遥感影像场景分类[J]. 遥感学报, 2023, 27(12):2815-2830.
	WANG Liming, QI Kunlun, YANG Chao, et al. Weakly supervised scale adaptive enhancement for scene classification of high-resolution remote sensing images[J]. National Remote Sensing Bulletin, 2023, 27(12):2815-2830.
[26]	WANG Qi, LIU Shaoteng, CHANUSSOT J, et al. Scene classification with recurrent attention of VHR remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(2): 1155-1167.
[27]	LIN Daoyu, FU Kun, WANG Yang, et al. MARTA GANs: unsupervised representation learning for remote sensing image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(11): 2092-2096.
[28]	YU Yunlong, LI Xianzhi, LIU Fuxian. Attention GANs: unsupervised deep feature learning for aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(1): 519-531.
[29]	LÜ Pengyuan, WU Wenjun, ZHONG Yanfei, et al. SCViT: a spatial-channel feature preserving vision transformer for remote sensing image scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3157671.
[30]	吴倩倩, 倪康, 郑志忠. 基于双阶段高阶Transformer的遥感图像场景分类[J]. 遥感学报, 2025, 29(3): 792-807.
	WU Qianqian, NI Kang, ZHENG Zhizhong. Remote sensing image scene classification based on two-stage high-order Transformer[J]. National Remote Sensing Bulletin, 2025, 29(3): 792-807.
[31]	KARALAS K, TSAGKATAKIS G, ZERVAKIS M, et al. Land classification using remotely sensed data: going multilabel[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(6): 3548-3563.
[32]	QI Xiaoman, ZHU Panpan, WANG Yuebin, et al. MLRSNet: a multi-label high spatial resolution remote sensing dataset for semantic scene understanding[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 169: 337-350.
[33]	LI Yansheng, CHEN Ruixian, ZHANG Yongjun, et al. Multi-label remote sensing image scene classification by combining a convolutional neural network and a graph neural network[J]. Remote Sensing, 2020, 12(23): 4003.
[34]	KING M D, PLATNICK S, MENZEL W P, et al. Spatial and temporal distribution of clouds observed by MODIS onboard the terra and aqua satellites[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(7): 3826-3852.
[35]	LI Yansheng, WEI Fanyi, ZHANG Yongjun, et al. HS2P: hierarchical spectral and structure-preserving fusion network for multimodal remote sensing image cloud and shadow removal[J]. Information Fusion, 2023, 94: 215-228.
[36]	LI Xue, ZHANG Guo, CUI Hao, et al. MCANet: a joint semantic segmentation framework of optical and SAR images for land use classification[J]. International Journal of Applied Earth Observation and Geoinformation, 2022, 106: 102638.
[37]	ZHU Xiaoxiang, TUIA D, MOU Lichao, et al. Deep learning in remote sensing: a comprehensive review and list of resources[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(4): 8-36.
[38]	AUDEBERT N, LE SAUX B, LEFÈVRE S. Beyond RGB: very high resolution urban remote sensing with multimodal deep networks[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 140: 20-32.
[39]	HOFFMANN D S, CLASEN K N, DEMIR B. Transformer-based multi-modal learning for multi-label remote sensing image classification[C]//Proceedings of 2023 IEEE International Geoscience and Remote Sensing Symposium. Pasadena: IEEE, 2023: 4891-4894.
[40]	GAWLIKOWSKI J, SAHA S, NIEBLING J, et al. Handling unexpected inputs: incorporating source-wise out-of-distribution detection into SAR-optical data fusion for scene classification[J]. EURASIP Journal on Advances in Signal Processing, 2023, 2023(1): 47.
[41]	HONG Danfeng, ZHANG Bing, LI Xuyang, et al. SpectralGPT: spectral remote sensing foundation model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(8): 5227-5244.
[42]	GUO Xin, LAO Jiangwei, DANG Bo, et al. SkySense: a multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 27662-27673.
[43]	ZUHLKE M, FOMFERRA N, BROCKMANN C, et al. SNAP (Sentinel application platform) and the ESA Sentinel 3 toolbox[C]//Proceedings of Sentinel-3 for Science Workshop. Venice: [s.n.], 2015.
[44]	LI Chengyang, SONG Dan, TONG Ruofeng, et al. Illumination-aware faster R-CNN for robust multispectral pedestrian detection[J]. Pattern Recognition, 2019, 85: 161-171.
[45]	YU Dingjun, WANG Hanli, CHEN Peiqiu, et al. Mixed pooling for convolutional neural networks[C]//Proceedings of 2014 Rough Sets and Knowledge Technology. Cham: Springer International Publishing, 2014: 364-375.
[46]	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2024-01-03]. https://arxiv.org/abs/1412.6980.
[47]	SUMBUL G, CHARFUELAN M, DEMIR B, et al. Bigearthnet: a large-scale benchmark archive for remote sensing image understanding[C]//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE, 2019: 5901-5904.

方法	指标	水体	道路	建筑物	裸地	绿地	农田	平均值
SCTFusionViT^[39]	Precision	92.38	33.33	95.70	63.05	88.11	74.86	74.57
	Recall	96.22	15.13	80.10	76.54	95.60	69.83	72.24
	F₁值	94.26	20.81	87.21	69.14	91.70	72.26	72.56
	Micro-F₁值	—	—	—	—	—	—	85.93
OOD^[40]	Precision	94.26	25.53	94.15	68.06	87.61	84.59	75.70
	Recall	96.22	23.68	80.10	73.66	95.10	63.64	72.07
	F₁值	95.23	24.57	86.56	70.75	91.21	72.63	73.49
	Micro-F₁值	—	—	—	—	—	—	85.76
MCANet^[36]	Precision	96.23	58.41	98.96	78.66	90.03	88.92	85.20
	Recall	96.48	43.42	81.48	77.37	96.72	72.99	78.08
	F₁值	96.36	49.81	89.37	78.01	93.26	80.17	81.16
	Micro-F₁值	—	—	—	—	—	—	89.53
PDANet（本文方法）	Precision	95.59	74.22	98.44	85.45	91.86	91.64	89.81
	Recall	98.83	57.89	86.62	74.90	96.93	75.10	72.56
	F₁值	97.18	65.67	92.15	79.82	94.33	82.55	85.28
	Micro-F₁值	—	—	—	—	—	—	91.40

交叉注意力	自注意力	Precision	Recall	F₁值	Micro-F₁值
		86.94	82.8	84.53	90.89
√		88.08	81.75	84.57	91.08
	√	87.99	82.56	84.98	91.11
√	√	89.81	81.71	85.28	91.40

池化方法	Precision	Recall	F₁值	Micro-F₁值
平均池化	86.62	83.19	84.63	91.04
最大池化	88.48	82.87	85.31	91.14
ADR（本文方法）	89.81	81.71	85.28	91.40

数据输入	Precision	Recall	F₁值	Micro-F₁值
无云光学	84.51	82.90	83.26	90.14
有云光学	76.57	71.78	72.19	83.95
SAR	71.61	68.07	69.33	82.51
有云光学+SAR	78.5	74.29	75.90	87.45
无云光学+SAR	89.81	81.71	85.28	91.40

骨干网络	Precision	Recall	F₁值	Micro-F₁值
ResNet18	74.04	64.35	68.20	83.02
Efficient-b6	79.74	72.65	75.44	86.31
ConvNeXt-S	86.11	81.20	83.31	89.84
ResNet101	89.81	81.71	85.28	91.40

基于SAR与光学遥感影像融合的多标签场景分类方法

Multi-label scene classification method based on fusion of SAR and optical remote sensing images

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 47

相关文章 15

编辑推荐

Metrics

本文评价

方法	Precision	Recall	F₁值	Micro-F₁值
SCTFusionViT^[39]	62.35	55.16	55.42	71.35
OOD^[40]	63.96	54.28	56.05	70.71
Mcanet^[36]	55.58	47.42	48.98	67.14
PDANet（本文方法）	69.08	60.18	61.43	72.48

[1]	王艳军, 唐徐超, 王成, 蔡恒藩. 基于道路拓扑关联特征的城乡道路面精细提取网络[J]. 测绘学报, 2025, 54(1): 75-89.
[2]	龚良雄, 李星华, 程远明, 赵兴友, 谢仁平, 王红根. 时空差异增强与自适应特征融合的轻量级遥感影像变化检测网络[J]. 测绘学报, 2025, 54(1): 136-153.
[3]	唐建波, 胡致远, 彭举, 夏何炎, 丁俊杰, 张玉玉, 梅小明. 融合视觉特征与运动特征的众源轨迹数据道路交叉口识别方法[J]. 测绘学报, 2025, 54(1): 182-193.
[4]	刘万增, 陈杭, 任加新, 张兆江, 李然, 赵婷婷, 翟曦, 朱秀丽. 基于混合智能的街景影像知识提取方法[J]. 测绘学报, 2024, 53(9): 1817-1828.
[5]	张付兵, 孙群, 马京振, 孙士杰, 温伯威. 融合全局和局部特征的建筑物形状智能分类方法[J]. 测绘学报, 2024, 53(9): 1842-1852.
[6]	鄢薪, 慎利, 潘俊杰, 戴延帅, 王继成, 郑晓莉, 李志林. 多尺度特征融合与空间优化的弱监督高分遥感建筑变化检测[J]. 测绘学报, 2024, 53(8): 1586-1597.
[7]	徐涛, 杨元维, 高贤君, 王志威, 潘越, 李少华, 许磊, 王艳军, 刘波, 余静, 吴凤敏, 孙浩宇. 融合图卷积与多尺度特征的接触网点云语义分割[J]. 测绘学报, 2024, 53(8): 1624-1633.
[8]	蒋亚楠, 郑林枫, 许强, 汤明高, 朱星. 机理引导下的阶跃型滑坡位移预测深度学习模型[J]. 测绘学报, 2024, 53(6): 1128-1139.
[9]	彭代锋, 翟晨晨, 周顶蔚, 张永军, 管海燕, 臧玉府. 基于金字塔语义token全局信息增强的高分光学遥感影像变化检测[J]. 测绘学报, 2024, 53(6): 1195-1211.
[10]	丁少鹏, 卢秀山, 刘如飞, 杨懿, 顾海燕, 李海涛. 联合目标特征引导与多重注意力的建筑物变化检测[J]. 测绘学报, 2024, 53(6): 1224-1235.
[11]	纪长琦, 郭肇捷, 孙海丽, 钟若飞. 基于移动激光扫描的地铁隧道渗漏水定位及快速检测方法[J]. 测绘学报, 2024, 53(6): 1236-1250.
[12]	林云浩, 王艳军, 李少春, 蔡恒藩. 一种耦合DeepLab与Transformer的农作物种植类型遥感精细分类方法[J]. 测绘学报, 2024, 53(2): 353-366.
[13]	张彩丽, 向隆刚, 李雅丽, 高松峰, 潘传姣. 路段级导航属性信息挖掘[J]. 测绘学报, 2024, 53(2): 367-378.
[14]	庞世燕, 郝京京, 左志奇, 兰晶晶, 胡翔云. 融合密集连接和自注意力机制的高分辨率遥感影像变化检测方法[J]. 测绘学报, 2024, 53(12): 2244-2253.
[15]	江宝得, 黄威, 许少芬, 巫勇. 融合分散自适应注意力机制的多尺度遥感影像建筑物实例细化提取[J]. 测绘学报, 2023, 52(9): 1504-1514.