Multi-label scene classification method based on fusion of SAR and optical remote sensing images

doi:10.11947/j.AGCS.2025.20240281

Abstract

Abstract:

Deep convolutional neural networks have proven to be one of the most effective methods for scene classification of high-resolution remote sensing images. Most previous studies focus on scene-level classification of single optical remote sensing images and are primarily limited to single-label classification. However, single optical remote sensing images are often constrained by weather conditions, and single-label annotations cannot fully describe complex image contents. Therefore, in this paper, we constructed a multimodal, multi-label scene classification dataset called SEN12-MLRS, using SAR and optical remote sensing images acquired by the European Space Agency in 2020. We proposed a parallel dual attention fusion network (PDANet) for multi-label scene classification. PDANet achieves optical and SAR image feature extraction as well as multi-modal and multilevel feature fusion through two-branch feature extraction, adaptive feature fusion, and multilevel feature fusion. Experimental results demonstrate that PDANet achieves superior performance compared to many state-of-the-art models on the SEN12-MLRS dataset. The effectiveness of the proposed network and its modules is further validated through ablation experiments.

Key words: multi-modal remote sensing image fusion, attention mechanism, multi-label classification, feature fusion

CLC Number:

P236

Yiming ZHAO, Kelin HU, Kelong TU, Yaxian QING, Chao YANG, Kunlun QI, Huayi WU. Multi-label scene classification method based on fusion of SAR and optical remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(5): 911-923.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Tab. 1

Tab. 2

Tab. 3

Tab. 4

Tab. 5

Tab. 6

References 47

[1]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vega: IEEE, 2016: 770-778.
[2]	TAN Mingxing, LE Q V. EfficientNet: rethinking model scaling for convolutional neural networks[EB/OL]. [2024-02-03]. https://arxiv.org/abs/1905.11946.
[3]	LIU Zhuang, MAO Hanzi, WU Chaoyuan, et al. A ConvNet for the 2020s[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 11966-11976.
[4]	白坤, 慕晓冬, 陈雪冰, 等. 融合半监督学习的无监督遥感影像场景分类[J]. 测绘学报, 2022, 51(5): 691-702. DOI . doi: 10.11947/j.AGCS.2022.20210270
	BAI Kun, MU Xiaodong, CHEN Xuebing, et al. Unsupervised remote sensing image scene classification based on semi-supervised learning[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(5): 691-702. DOI . doi: 10.11947/j.AGCS.2022.20210270
[5]	李彦胜, 吴敏郎, 张永军. 知识图谱约束深度网络的高分辨率遥感影像场景分类[J]. 测绘学报, 2024, 53(4): 677-688. DOI . doi: 10.11947/j.AGCS.2024.20230125
	LI Yansheng, WU Minlang, ZHANG Yongjun. Knowledge graph-guided deep network for high-resolution remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(4): 677-688. DOI . doi: 10.11947/j.AGCS.2024.20230125
[6]	ZHANG Wei, TANG Ping, ZHAO Lijun. Remote sensing image scene classification using CNN-CapsNet[J]. Remote Sensing, 2019, 11(5): 494.
[7]	ZHANG Xinchang, SHI Qian, SUN Ying, et al. The review of land use/land cover mapping AI methodology and application in the era of remote sensing big data[J]. Journal of Geodesy and Geoinformation Science, 2024, 7(3):1-23.
[8]	尹国应, 张洪艳, 张良培. 2001—2019年长江中下游农业干旱遥感监测及植被敏感性分析[J]. 武汉大学学报(信息科学版), 2022, 47(8): 1245-1256.
	YIN Guoying, ZHANG Hongyan, ZHANG Liangpei. Remote sensing monitoring of agricultural drought in the middle and lower reaches of the Yangtze River and vegetation sensitivity analysis from 2001 to 2019[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1245-1256.
[9]	YUAN Gang, WANG Yunjia, ZHAO Feng, et al. Spatiotemporal correlation characteristics between thermal infrared remote sensing obtained surface thermal anomalies and reconstructed 4D temperature fields of underground coal fires[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-18.
[10]	LI Yansheng, ZHANG Yongjun, ZHU Zhihui. Error-tolerant deep learning for remote sensing image scene classification[J]. IEEE Transactions on Cybernetics, 2021, 51(4): 1756-1768.
[11]	BOUTELL M R, LUO Jiebo, SHEN Xipeng, et al. Learning multi-label scene classification[J]. Pattern Recognition, 2004, 37(9): 1757-1771.
[12]	DAI O E, DEMIR B, SANKUR B, et al. A novel system for content-based retrieval of single and multi-label high-dimensional remote sensing images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(7): 2473-2490.
[13]	QI Kunlun, WU Huayi, SHEN Chen, et al. Land-use scene classification in high-resolution remote sensing images using improved correlatons[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(12): 2403-2407.
[14]	LOWE D G. Distinctive image features from scale-invariant key points[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[15]	HARALICK R M, SHANMUGAM K, DINSTEIN I. Textural features for image classification[J]. IEEE Transactions on Systems, Man, and Cybernetics, 1973, SMC-3(6): 610-621.
[16]	JAIN A K, RATHA N K, LAKSHMANAN S. Object detection using Gabor filters[J]. Pattern Recognition, 1997, 30(2): 295-309.
[17]	OJALA T, PIETIKAINEN M, MAENPAA T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7): 971-987.
[18]	SWAIN M J, BALLARD D H. Color indexing[J]. International Journal of Computer Vision, 1991, 7(1): 11-32.
[19]	DALAL N, TRIGGS B. Histograms of oriented gradients for human detection[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego: IEEE, 2005: 886-893.
[20]	ZHU Qiqi, SUN Xiongli, ZHONG Yanfei, et al. High-resolution remote sensing image scene understanding: a review[C]//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE, 2019: 3061-3064.
[21]	CHENG Gong, XIE Xingxing, HAN Junwei, et al. Remote sensing image scene classification meets deep learning: challenges, methods, benchmarks, and opportunities[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 3735-3756.
[22]	PENATTI O A B, NOGUEIRA K, DOS SANTOS J A. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston: IEEE, 2015: 44-51.
[23]	龚希, 吴亮, 谢忠, 等. 融合全局和局部深度特征的高分辨率遥感影像场景分类方法[J]. 光学学报, 2019, 39(3): 19-29.
	GONG Xi, WU Liang, XIE Zhong, et al. Classification method of high-resolution remote sensing scenes based on fusion of global and local deep features[J]. Acta Optica Sinica, 2019, 39(3): 19-29.
[24]	LU Xiaoqiang, SUN Hao, ZHENG Xiangtao. A feature aggregation convolutional neural network for remote sensing scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(10): 7894-7906.
[25]	王梨名, 祁昆仑, 杨超, 等. 弱监督尺度自适应增强的高分辨率遥感影像场景分类[J]. 遥感学报, 2023, 27(12):2815-2830.
	WANG Liming, QI Kunlun, YANG Chao, et al. Weakly supervised scale adaptive enhancement for scene classification of high-resolution remote sensing images[J]. National Remote Sensing Bulletin, 2023, 27(12):2815-2830.
[26]	WANG Qi, LIU Shaoteng, CHANUSSOT J, et al. Scene classification with recurrent attention of VHR remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(2): 1155-1167.
[27]	LIN Daoyu, FU Kun, WANG Yang, et al. MARTA GANs: unsupervised representation learning for remote sensing image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(11): 2092-2096.
[28]	YU Yunlong, LI Xianzhi, LIU Fuxian. Attention GANs: unsupervised deep feature learning for aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(1): 519-531.
[29]	LÜ Pengyuan, WU Wenjun, ZHONG Yanfei, et al. SCViT: a spatial-channel feature preserving vision transformer for remote sensing image scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3157671.
[30]	吴倩倩, 倪康, 郑志忠. 基于双阶段高阶Transformer的遥感图像场景分类[J]. 遥感学报, 2025, 29(3): 792-807.
	WU Qianqian, NI Kang, ZHENG Zhizhong. Remote sensing image scene classification based on two-stage high-order Transformer[J]. National Remote Sensing Bulletin, 2025, 29(3): 792-807.
[31]	KARALAS K, TSAGKATAKIS G, ZERVAKIS M, et al. Land classification using remotely sensed data: going multilabel[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(6): 3548-3563.
[32]	QI Xiaoman, ZHU Panpan, WANG Yuebin, et al. MLRSNet: a multi-label high spatial resolution remote sensing dataset for semantic scene understanding[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 169: 337-350.
[33]	LI Yansheng, CHEN Ruixian, ZHANG Yongjun, et al. Multi-label remote sensing image scene classification by combining a convolutional neural network and a graph neural network[J]. Remote Sensing, 2020, 12(23): 4003.
[34]	KING M D, PLATNICK S, MENZEL W P, et al. Spatial and temporal distribution of clouds observed by MODIS onboard the terra and aqua satellites[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(7): 3826-3852.
[35]	LI Yansheng, WEI Fanyi, ZHANG Yongjun, et al. HS2P: hierarchical spectral and structure-preserving fusion network for multimodal remote sensing image cloud and shadow removal[J]. Information Fusion, 2023, 94: 215-228.
[36]	LI Xue, ZHANG Guo, CUI Hao, et al. MCANet: a joint semantic segmentation framework of optical and SAR images for land use classification[J]. International Journal of Applied Earth Observation and Geoinformation, 2022, 106: 102638.
[37]	ZHU Xiaoxiang, TUIA D, MOU Lichao, et al. Deep learning in remote sensing: a comprehensive review and list of resources[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(4): 8-36.
[38]	AUDEBERT N, LE SAUX B, LEFÈVRE S. Beyond RGB: very high resolution urban remote sensing with multimodal deep networks[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 140: 20-32.
[39]	HOFFMANN D S, CLASEN K N, DEMIR B. Transformer-based multi-modal learning for multi-label remote sensing image classification[C]//Proceedings of 2023 IEEE International Geoscience and Remote Sensing Symposium. Pasadena: IEEE, 2023: 4891-4894.
[40]	GAWLIKOWSKI J, SAHA S, NIEBLING J, et al. Handling unexpected inputs: incorporating source-wise out-of-distribution detection into SAR-optical data fusion for scene classification[J]. EURASIP Journal on Advances in Signal Processing, 2023, 2023(1): 47.
[41]	HONG Danfeng, ZHANG Bing, LI Xuyang, et al. SpectralGPT: spectral remote sensing foundation model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(8): 5227-5244.
[42]	GUO Xin, LAO Jiangwei, DANG Bo, et al. SkySense: a multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery[C]//Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 27662-27673.
[43]	ZUHLKE M, FOMFERRA N, BROCKMANN C, et al. SNAP (Sentinel application platform) and the ESA Sentinel 3 toolbox[C]//Proceedings of Sentinel-3 for Science Workshop. Venice: [s.n.], 2015.
[44]	LI Chengyang, SONG Dan, TONG Ruofeng, et al. Illumination-aware faster R-CNN for robust multispectral pedestrian detection[J]. Pattern Recognition, 2019, 85: 161-171.
[45]	YU Dingjun, WANG Hanli, CHEN Peiqiu, et al. Mixed pooling for convolutional neural networks[C]//Proceedings of 2014 Rough Sets and Knowledge Technology. Cham: Springer International Publishing, 2014: 364-375.
[46]	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2024-01-03]. https://arxiv.org/abs/1412.6980.
[47]	SUMBUL G, CHARFUELAN M, DEMIR B, et al. Bigearthnet: a large-scale benchmark archive for remote sensing image understanding[C]//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE, 2019: 5901-5904.

方法	指标	水体	道路	建筑物	裸地	绿地	农田	平均值
SCTFusionViT^[39]	Precision	92.38	33.33	95.70	63.05	88.11	74.86	74.57
	Recall	96.22	15.13	80.10	76.54	95.60	69.83	72.24
	F₁值	94.26	20.81	87.21	69.14	91.70	72.26	72.56
	Micro-F₁值	—	—	—	—	—	—	85.93
OOD^[40]	Precision	94.26	25.53	94.15	68.06	87.61	84.59	75.70
	Recall	96.22	23.68	80.10	73.66	95.10	63.64	72.07
	F₁值	95.23	24.57	86.56	70.75	91.21	72.63	73.49
	Micro-F₁值	—	—	—	—	—	—	85.76
MCANet^[36]	Precision	96.23	58.41	98.96	78.66	90.03	88.92	85.20
	Recall	96.48	43.42	81.48	77.37	96.72	72.99	78.08
	F₁值	96.36	49.81	89.37	78.01	93.26	80.17	81.16
	Micro-F₁值	—	—	—	—	—	—	89.53
PDANet（本文方法）	Precision	95.59	74.22	98.44	85.45	91.86	91.64	89.81
	Recall	98.83	57.89	86.62	74.90	96.93	75.10	72.56
	F₁值	97.18	65.67	92.15	79.82	94.33	82.55	85.28
	Micro-F₁值	—	—	—	—	—	—	91.40

交叉注意力	自注意力	Precision	Recall	F₁值	Micro-F₁值
		86.94	82.8	84.53	90.89
√		88.08	81.75	84.57	91.08
	√	87.99	82.56	84.98	91.11
√	√	89.81	81.71	85.28	91.40

池化方法	Precision	Recall	F₁值	Micro-F₁值
平均池化	86.62	83.19	84.63	91.04
最大池化	88.48	82.87	85.31	91.14
ADR（本文方法）	89.81	81.71	85.28	91.40

数据输入	Precision	Recall	F₁值	Micro-F₁值
无云光学	84.51	82.90	83.26	90.14
有云光学	76.57	71.78	72.19	83.95
SAR	71.61	68.07	69.33	82.51
有云光学+SAR	78.5	74.29	75.90	87.45
无云光学+SAR	89.81	81.71	85.28	91.40

骨干网络	Precision	Recall	F₁值	Micro-F₁值
ResNet18	74.04	64.35	68.20	83.02
Efficient-b6	79.74	72.65	75.44	86.31
ConvNeXt-S	86.11	81.20	83.31	89.84
ResNet101	89.81	81.71	85.28	91.40