Hyperspectral remote sensing image scene classification method based on deep manifold distillation network

doi:10.11947/j.AGCS.2024.20230373

Abstract

Abstract:

Most of the current scene classification tasks are based on high-resolution remote sensing images, and the lack of spectral information limits its discrimination ability for scene classification. While hyperspectral remote sensing images have the characteristic of “spatial-spectral integration”, which has unique advantages in scene classification. To address the issue of the complex landcover distribution and the high dimension and redundancy in hyperspectral images, this paper proposes a hyperspectral scene classification manifold distillation network (HSCMDNet) to improve the performance of hyperspectral remote sensing scene classification. For the complex landcover distribution of remote sensing images, HSCMDNet employs Swin Transformer as a teacher network to reveal the long-distance dependency information of hyperspectral images and capture the relationship between different bands. After that, a manifold distillation loss is designed in the middle layer of the teacher network and the student network ResNet-18. By matching the middle layer output features of the student model and the teacher model in the manifold space, the knowledge of the teacher model is transferred to the lightweight student model effectively, which alleviates the high computational complexity caused by high-dimensional hyperspectral data. In the orbita hyperspectral image scene classification dataset (OHID-SC) and natural scene classification with Tiangong-2 remotely sensed imagery (NaSC-TG2), the best classification accuracies of the proposed HSCMDNet network reached 93.60% and 94.55%, respectively.

Key words: hyperspectral scene classification, knowledge distillation, middle layer knowledge transfer, manifold mapping, Transformer

CLC Number:

P237

Quanyi ZHAO, Fujian ZHENG, Bo XIA, Zhengying LI, Hong HUANG. Hyperspectral remote sensing image scene classification method based on deep manifold distillation network[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(12): 2404-2415.

Figures/Tables 14

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Tab. 1

Tab. 2

Fig. 8

Tab. 3

Tab. 4

Fig. 9

Fig. 10

References 36

[1]	余东行, 徐青, 赵传, 等. 注意力引导特征融合与联合学习的遥感图像场景分类[J]. 测绘学报, 2023, 52(4): 624-637. DOI:. doi: 10.11947/j.AGCS.2023.20210659
	YU Donghang, XU Qing, ZHAO Chuan, et al. Attention-guided feature fusion and joint learning for remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(4): 624-637. DOI:. doi: 10.11947/j.AGCS.2023.20210659
[2]	夏英, 李骏垚, 郭东恩. 基于GAN的半监督遥感图像场景分类[J]. 光子学报, 2022, 51(3): 28-39.
	XIA Ying, LI Junyao, GUO Dongen. Semi-supervised scene classification of remote sensing images based on GAN[J]. Acta Photonica Sinica, 2022, 51(3): 28-39.
[3]	马欣悦, 王梨名, 祁昆仑, 等. 基于多尺度循环注意力网络的遥感图像场景分类方法[J]. 地球科学, 2021, 46(10): 3740-3752.
	MA Xinyue, WANG Liming, QI Kunlun, et al. Remote sensing image scene classification method based on multi-scale cyclic attention network[J]. Earth Science, 2021, 46(10): 3740-3752.
[4]	DOS SANTOS J, PENATTI O, TORRES R. Evaluating the potential of texture and color descriptors for remote sensing image retrieval and classification[C]//Proceedings of 2010 IEEE International Conference on Computer Vision Theory and Applications. Angers: IEEE, 2010, 2: 203-208.
[5]	CHEN Chen, ZHANG Baochang, SU Hongjun, et al. Land-use scene classification using multi-scale completed local binary patterns[J]. Signal, Image and Video Processing, 2016, 10(4): 745-752.
[6]	黄鸿, 徐科杰, 石光耀. 联合多尺度多特征的高分遥感图像场景分类[J]. 电子学报, 2020, 48(9): 1824-1833.
	HUANG Hong, XU Kejie, SHI Guangyao. Scene classification of high-resolution remote sensing image by multi-scale and multi-feature fusion[J]. Acta Electronica Sinica, 2020, 48(9): 1824-1833.
[7]	TONG Xinyi, XIA Guisong, LU Qikai, et al. Land-cover classification with high-resolution remote sensing images using transferable deep models[J]. Remote Sensing of Environment, 2020, 237: 111322.
[8]	HE Hao, WANG Shuyang, WANG Shicheng, et al. A road extraction method for remote sensing image based on encoder decoder network[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(2): 16-25.
[9]	DAI Yuchao, ZHANG Jing, HE Mingyi, et al. Salient object detection from multi-spectral remote sensing images with deep residual network[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2): 101-110.
[10]	SHI Jiacheng, LIU Wei, SHAN Haoyu, et al. Remote sensing scene classification based on multibranch fusion attention network[J]. IEEE Geoscience and Remote Sensing Letters, 2023, 20: 3262407.
[11]	JIN Jianhui, ZHOU Wujie, YE Lü, et al. DASFNet: dense-attention-similarity-fusion network for scene classification of dual-modal remote-sensing images[J]. International Journal of Applied Earth Observation and Geoinformation, 2022, 115: 103087.
[12]	GUO Haonan, SHI Qian, DU Bo, et al. Scene-driven multitask parallel attention network for building extraction in high-resolution remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(5): 4287-4306.
[13]	WANG Di, LAN Jinhui. A deformable convolutional neural network with spatial-channel attention for remote sensing scene classification[J]. Remote Sensing, 2021, 13(24): 5076.
[14]	施慧慧, 徐雁南, 滕文秀, 等. 高分辨率遥感影像深度迁移可变形卷积的场景分类法[J]. 测绘学报, 2021, 50(5): 652-663. DOI:. doi: 10.11947/j.AGCS.2021.20200190
	SHI Huihui, XU Yannan, TENG Wenxiu, et al. Scene classification of high-resolution remote sensing imagery based on deep transfer deformable convolutional neural networks[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(5): 652-663. DOI:. doi: 10.11947/j.AGCS.2021.20200190
[15]	郑卓, 方芳, 刘袁缘, 等. 高分辨率遥感影像场景的多尺度神经网络分类法[J]. 测绘学报, 2018, 47(5): 620-630. DOI:. doi: 10.11947/j.AGCS.2018.20170191
	ZHENG Zhuo, FANG Fang, LIU Yuanyuan, et al. Joint multi-scale convolution neural network for scene classification of high resolution remote sensing imagery[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(5): 620-630. DOI:. doi: 10.11947/j.AGCS.2018.20170191
[16]	WAN Huiyao, CHEN Jie, HUANG Zhixiang, et al. Lightweight channel attention and multiscale feature fusion discrimination for remote sensing scene classification[J]. IEEE Access, 2021, 9: 94586-94600.
[17]	ZHANG Shichao, WANG Changying, LI Jinhua, et al. MF-Dfnet: a deep learning method for pixel-wise classification of very high-resolution remote sensing images[J]. International Journal of Remote Sensing, 2022, 43(1): 330-348.
[18]	WANG Kexian, ZHENG Shunyi, LI Rui, et al. A deep double-channel dense network for hyperspectral image classification[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(4): 46-62.
[19]	刘康, 周壮, 李盛阳, 等. 天宫一号高光谱遥感场景分类数据集及应用[J]. 遥感学报, 2020, 24(9): 1077-1087.
	LIU Kang, ZHOU Zhuang, LI Shengyang, et al. Scene classification dataset using the Tiangong-1 hyperspectral remote sensing imagery and its applications[J]. Journal of Remote Sensing, 2020, 24(9): 1077-1087.
[20]	ZHOU Zhuang, LI Shengyang, WU Wei, et al. NaSC-TG2: natural scene classification with Tiangong-2 remotely sensed imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 3228-3242.
[21]	徐科杰, 邓培芳, 黄鸿. HSRS-SC：面向遥感场景分类的高光谱图像数据集[J]. 中国图象图形学报, 2021, 26(8): 1809-1822.
	XU Kejie, DENG Peifang, HUANG Hong. HSRS-SC: a hyperspectral image dataset for remote sensing scene classification[J]. Journal of Image and Graphics, 2021, 26(8): 1809-1822.
[22]	SINGH A, BRUZZONE L. WIANet: a wavelet-inspired attention-based convolution neural network for land cover classification[J]. IEEE Geoscience and Remote Sensing Letters, 2023, 20: 1-5.
[23]	XU Kejie, HUANG Hong, DENG Peifang. Attention-based deep feature learning network for scene classification of hyperspectral images[C]//Proceedings of 2021 IEEE Asilomar Conference on Signals, Systems, and Computers. Pacific Grove: IEEE, 2021: 1690-1693.
[24]	LIU Hao, QU Ying, ZHANG Liqiang. Multispectral scene classification via cross-modal knowledge distillation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3174352.
[25]	WANG Guoqing, ZHANG Ning, LIU Wenchao, et al. MFST: a multi-level fusion network for remote sensing scene classification[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 3205417.
[26]	ZHENG Fujian, LIN Shuai, ZHOU Wei, et al. A lightweight dual-branch swin transformer for remote sensing scene classification[J]. Remote Sensing, 2023, 15(11): 2865.
[27]	HAO Siyuan, LI Nan, YE Yuanxin. Inductive biased swin-transformer with cyclic regressor for remote sensing scene classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023, 16: 6265-6278.
[28]	ZHANG Zizhao, ZHANG Han, ZHAO Long, et al. Nested hierarchical transformer: towards accurate, data-efficient and interpretable visual understanding[C]//Proceedings of 2022 IEEE AAAI Conference on Artificial Intelligence. Virtual Event: IEEE, 2022, 36(3): 3417-3425.
[29]	XU Kejie, DENG Peifang, HUANG Hong. Vision transformer: an excellent teacher for guiding small networks in remote sensing image scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3152566.
[30]	LIU Ze, LIN Yutong, CAO Yue, et al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Virtual Event: IEEE, 2021: 10012-10022.
[31]	HE Kaiming, ZHANG Xianyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[32]	TAO Zeyi, XIA Qi, LI Qun. Neuron manifold distillation for edge deep learning[C]//Proceedings of 2021 International Symposium on Quality of Service. Tokyo: IEEE, 2021: 1-10.
[33]	刘英旭, 蒲春宇, 许典坤, 等. 面向高光谱影像场景分类的轻量化深度全局-局部知识蒸馏网络[J]. 光学精密工程, 2023, 31(17): 2598-2610.
	LIU Yingxu, PU Chunyu, XU Diankun, et al. Lightweight deep global-local knowledge distillation network for hyperspectral image scene classification[J]. Optics and Precision Engineering, 2023, 31(17): 2598-2610.
[34]	CAO Ran, FANG Leyuan, LU Ting, et al. Self-attention-based deep feature fusion for remote sensing scene classification[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(1): 43-47.
[35]	CHEN Sibao, WEI Qingsong, WANG Wenzhong, et al. Remote sensing scene classification via multi-branch local attention network[J]. IEEE Transactions on Image Processing, 2022, 31: 99-109.
[36]	YAO Dazhi, SHAO Yunxue. A data efficient transformer based on Swin Transformer[J]. The Visual Computer, 2024, 40(4): 2589-2598.

应用阶段		优化器	初始学习率	学习率更新策略
教师模型（第一阶段）	OHID-SC数据集	Adam；权重衰减参数0.000 1	0.000 1	保持初始学习率训练300轮
教师模型（第一阶段）	NaSC-TG2数据集	Adam；权重衰减参数0.0001	0.0001	保持初始学习率训练40轮
教师模型和学生模型（第二阶段）	OHID-SC和NaSC-TG2数据集	Adam；权重衰减参数0.000 1	0.000 1	共计训练60轮，每15轮学习率衰减为0.1倍

α	总体精度/（%）		T	总体精度/（%）		β	总体精度/（%）
α	OHID-SC数据集	NaSC-TG2数据集	T	OHID-SC数据集	NaSC-TG2数据集	β	OHID-SC数据集	NaSC-TG2数据集
0.1	92.78	93.36	1	92.34	93.46	0.1	92.29	93.78
0.2	92.51	94.44	2	92.23	94.27	0.2	92.89	94.30
0.5	92.29	94.19	5	92.61	94.26	0.5	93.11	94.55
0.8	92.83	94.48	10	93.00	94.55	0.8	93.60	91.20
1.0	93.11	94.20	20	92.67	94.29	1.0	92.56	92.60
1.25	92.61	94.41	30	92.56	94.39	1.25	93.22	82.70
2.0	91.74	94.28	40	93.11	94.27	2.0	92.29	92.90
5.0	91.96	92.66	50	92.56	94.26	5.0	92.94	90.80
10.0	88.89	88.53	100	92.67	94.02	10.0	92.23	68.80

算法	出版年份	总体精度/（%）
算法	出版年份	OHID-SC数据集	NaSC-TG2数据集
VGG-16	2014	80.58	87.40
ResNet-101	2015	92.29	86.18
GoogLeNet	2014	88.07	86.31
ConvNeXt	2022	90.59	85.73
RegNet	2020	90.86	80.93
ViT	2020	85.18	91.83
DeiT	2020	83.81	90.81
CaiT	2021	83.10	88.61
VGG-16+SAFF^[34]	2020	72.11	74.67
MBLANet^[35]	2021	91.50	93.78
LDGLKD^[33]	2023	91.62	87.47
HSCMDNet	2023	93.60	94.55

算法	测试时间/s		参数量/MB	FLOPs/GB
算法	OHID-SC数据集	NaSC-TG2数据集	参数量/MB	FLOPs/GB
VGG-16	9.43	159.98	131.95	5.13
ResNet-101	9.01	141.20	40.55	2.57
GoogLeNet	8.31	144.63	9.50	0.49
ConvNeXt	6.98	137.58	83.52	5.02
RegNet	7.40	139.12	35.93	2.63
ViT	8.33	143.19	84.17	5.70
DeiT	9.27	158.77	83.80	5.70
CaiT	7.69	144.31	46.41	2.86
VGG-16+SAFF^[34]	31.56	208.21	—	—
MBLANet^[35]	8.54	141.52	22.43	5.26
LDGLKD^[33]	8.49	145.18	14.04	5.01
HSCMDNet	6.67	132.59	10.70	0.74