Small-sample classification of hyperspectral images based on mixed CNN-ViT feature optimization

doi:10.11947/j.AGCS.2025.20250293

Abstract

Abstract:

Hyperspectral image classification is a key technology for achieving fine-grained recognition of ground objects. With the advancement of imaging technology, the spatial resolution of hyperspectral images acquired by UAV platforms has significantly improved, bringing new opportunities and challenges to fine-grained land cover classification. However, existing deep neural networks still exhibit insufficiently comprehensive feature learning for high spatial-resolution hyperspectral images under small-sample conditions. To address this issue, this paper proposes a mixed feature optimization method of convolutional neural networks (CNN) and vision Transformer (ViT), including three aspects: adaptive spatial-spectral feature learning, bidirectional feature integration and multi-segment feature interaction enhancement. First, multi-scale 3D spatial-spectral features and 2D local selfattention features are incorporated into a cascaded residual structure to achieve global-local multi-scale spatial-spectral feature extraction, enhancing the feature richness. Then, spatial and channel features are integrated from two directions to extract correlations across both dimensions, thereby complementing and enhancing the features extracted by CNN and ViT. After fusing these multi-stage features, they are fed into a factorized second order pooling layer to address the issues of large discrepancies and insufficient interaction among multi-stage features. Finally, the fine-grained fused features are input into a fully connected layer for classification. Small-sample classification experiments were conducted on three hyperspectral image datasets with high spatial resolution, namely LongKou, HanChuan, and HongHu. Only five samples per land-cover class are used for model training. The proposed method achieved classification accuracies of 94.00%, 83.24%, and 87.63%, respectively, demonstrating its effectiveness under small-sample conditions.

Key words: hyperspectral image classification, mixed convolutional network, local selfattention, factorized second order pooling, multi-feature optimization, small sample

CLC Number:

P237

Jin ZHANG, Fan FENG, Chenguang DAI, Zhenchao ZHANG, Ying YU, Bing LIU. Small-sample classification of hyperspectral images based on mixed CNN-ViT feature optimization[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(12): 2233-2246.

Figures/Tables 20

Fig. 1

Tab. 1

Fig. 2

Fig. 3

Fig. 4

Tab. 2

Tab. 3

Tab. 4

Tab. 5

Tab. 6

Tab. 7

Tab. 8

Fig. 5

Fig. 6

Fig. 7

Tab. 9

Tab. 10

Tab. 11

Tab. 12

Tab. 13

References 30

[1]	乔智, 姜群鸥, 律可心, 等. 无人机高光谱遥感和集成深度置信神经网络算法用于密云水库水质参数反演[J]. 光谱学与光谱分析, 2024, 44(7): 2066-2074.
	QIAO Zhi, JIANG Qunou, LÜ Kexin, et al. Retrieval model for water quality parameters of Miyun Reservoir based on UAV hyperspectral remote sensing data and deep neural network algorithm[J]. Spectroscopy and Spectral Analysis, 2024, 44(7): 2066-2074.
[2]	周冰, 李秉璇, 贺宣, 等. 陆基条件下典型地物和伪装光谱影响因子分析[J]. 光谱学与光谱分析, 2021, 41(9): 2956-2961.
	ZHOU Bing, LI Bingxuan, HE Xuan, et al. Study on the spectral characteristics of ground objects in land-based hyperspectral imaging[J]. Spectroscopy and Spectral Analysis, 2021, 41(9): 2956-2961.
[3]	HONG D, LI C, YOKOYA N, et al. Hyperspectral imaging[EB/OL]. [2025-11-13]. https://arxiv.org/abs/2508.08107.
[4]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[EB/OL]. [2025-11-13]. https://arxiv.org/abs/2010.11929.
[5]	刘冰, 余旭初, 张鹏强, 等. 联合空-谱信息的高光谱影像深度三维卷积网络分类[J]. 测绘学报, 2019, 48(1): 53-63. DOI: . doi: 10.11947/j.AGCS.2019.20170578
	LIU Bing, YU Xuchu, ZHANG Pengqiang, et al. Deep 3D convolutional network combined with spatial-spectral features for hyperspectral image classification[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(1): 53-63. DOI: . doi: 10.11947/j.AGCS.2019.20170578
[6]	ZHONG Zilong, LI J, LUO Zhiming, et al. Spectral-spatial residual network for hyperspectral image classification: a 3D deep learning framework[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(2): 847-858.
[7]	ROY S K, MANNA S, SONG Tiecheng, et al. Attention-based adaptive spectral-spatial kernel ResNet for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(9): 7831-7843.
[8]	FENG Fan, ZHANG Yongsheng, ZHANG Jin, et al. Low-rank constrained attention-enhanced multiple spatial-spectral feature fusion for small sample hyperspectral image classification[J]. Remote Sensing, 2023, 15(2): 304.
[9]	HONG Danfeng, HAN Zhu, YAO Jing, et al. SpectralFormer: rethinking hyperspectral image classification with Transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 5518615.
[10]	SUN Le, ZHAO Guangrui, ZHENG Yuhui, et al. Spectral-spatial feature tokenization Transformer for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5522214.
[11]	LI Bing, WANG Qiwen, LIANG Jiahong, et al. SquconvNet: deep sequencer convolutional network for hyperspectral image classification[J]. Remote Sensing, 2023, 15(4): 983.
[12]	ZHANG Mengmeng, LI Wei, ZHAO Xudong, et al. Morphological transformation and spatial-logical aggregation for tree species classification using hyperspectral imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5501212.
[13]	SHI Hao, CAO Guo, ZHANG Youqiang, et al. F3Net: fast Fourier filter network for hyperspectral image classification[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5502218.
[14]	JAMALI A, ROY S K, GHAMISI P. WetMapFormer: a unified deep CNN and vision Transformer for complex wetland mapping[J]. International Journal of Applied Earth Observation and Geoinformation, 2023, 120: 103333.
[15]	冯凡, 张永生, 张津, 等. 基于混合卷积网络的高光谱图像自监督特征学习方法[J]. 光学学报, 2024, 44(18): 1828007.
	FENG Fan, ZHANG Yongsheng, ZHANG Jin, et al. Self-supervised feature learning method for hyperspectral images based on mixed convolutional networks[J]. Acta Optica Sinica, 2024, 44(18): 1828007.
[16]	LIU Bing, YU Xuchu, YU Anzhu, et al. Deep few-shot learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(4): 2290-2304.
[17]	GAO Kuiliang, LIU Bing, YU Xuchu, et al. Unsupervised meta learning with multiview constraints for hyperspectral image small sample set classification[J]. IEEE Transactions on Image Processing, 2022, 31: 3449-3462.
[18]	HUANG Lingbo, CHEN Yushi, HE Xin. Spectral-spatial masked Transformer with supervised and contrastive learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5508718.
[19]	JAMALI A, ROY S K, HONG D, et al. Spatial gated multi-Layer perceptron for land use and land cover mapping[EB/OL]. [2025-11-13]. https://arxiv.org/abs/2308.05235.
[20]	TOLSTIKHIN I O, HOULSBY N, KOLESNIKOV A, et al. MLP-mixer: an all-MLP architecture for vision[EB/OL]. [2025-11-13]. https://arxiv.ogr/abs/2105.01601.
[21]	HOWARD A, SANDLER M, CHEN Bo, et al. Searching for MobileNetV3[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 1314-1324.
[22]	HU X, ZHONG Y, LUO C, et al. WHU-Hi: UAV-borne hyperspectral with high spatial resolution (H2) benchmark datasets for hyperspectral image classification[EB/OL]. [2025-11-13]. https://arxiv.org/abs/2012.13920.
[23]	ZHAN Jialei, XIE Yuhang, GUO Jiajia, et al. DGPF-RENet: a low data dependence network with low training iterations for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5520521.
[24]	LIU Baisen, LIU Yuanjia, ZHANG Wulin, et al. Spectral swin Transformer network for hyperspectral image classification[J]. Remote Sensing, 2023, 15(15): 3721.
[25]	HENDRYCKS D, GIMPEL K. Gaussian error linear units (Gelus)[EB/OL]. [2025-11-13]. https://arxiv.org/abs/1606.08415.
[26]	RAMACHANDRAN P, ZOPH B, LE Q V. Searching for activation functions[EB/OL]. [2025-11-13]. https://arxiv.org/abs/1710.05941.
[27]	LI Zhaokui, LIU Ming, CHEN Yushi, et al. Deep cross-domain few-shot learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 5501618.
[28]	XI Bobo, ZHANG Yun, LI Jiaojiao, et al. CTF-SSCL: CNN-Transformer for few-shot hyperspectral image classification assisted by semisupervised contrastive learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5532617.
[29]	WANG Zhuowei, ZHAO Shihui, ZHAO Genping, et al. Dual-branch domain adaptation few-shot learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5506116.
[30]	LIU Bing, YU Anzhu, YU Xuchu, et al. Deep multiview learning for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(9): 7758-7772.

类别	模块名	输出维度	核尺寸	个数
模型	Input	15×15×16
输入	Conv3D-BN-ReLU	15×15×16，24	3×3×3	24
3D空谱特征学习-1	MultiRes3D-1	15×15×16，24	1×1×7	24
	MultiRes3D-1	15×15×16，24	1×1×3	4，8，12
	LWSA-1	15×15×16，24
	Add-1	15×15×16，24
	Reshape-1	15×15×384
3D空谱特征学习-2	MultiRes3D-2	15×15×16，24	7×7×1	24
	MultiRes3D-2	15×15×16，24	3×3×1	4，8，12
	LWSA-2	15×15×16，24
	Add-2	15×15×16，24
	Reshape-2	15×15×384
2D重组特征整合	2D-Mixer	225×128
	Reshape-3	15×15×128
	Concatenate	15×15×896
多段特征交互	FSOP	2664
模型	Flatten	2664
输出	FC-Softmax	地物类别数

项目	LK数据集	HC数据集	HH数据集
图像大小/像素	550×440	1217×303	940×475
光谱范围/μm	0.4~1.0	0.4~1.0	0.4~1.0
波段数	270	274	270
空间分辨率/m	0.463	0.109	0.043
类别数	9	16	22
标记样本数	204 542	257 530	386 693

序号	地物名称	标记样本数	训练样本数	测试样本数
1	玉米	34 511	5	34 506
2	棉花	8374	5	8369
3	芝麻	3031	5	3026
4	宽叶大豆	63 212	5	63 207
5	窄叶大豆	4151	5	4146
6	水稻	11 854	5	11 849
7	水体	67 056	5	67 051
8	道路和房屋	7124	5	7119
9	混合杂草	5229	5	5224
	总数	204 542	45	204 497

序号	地物名称	标记样本数	训练样本数	测试样本数
1	草莓	44 735	5	44 730
2	豇豆	22 753	5	22 748
3	大豆	10 287	5	10 282
4	高粱	5353	5	5348
5	空心菜	1200	5	1195
6	西瓜	4533	5	4528
7	绿色植物	5903	5	5898
8	树	17 978	5	17 973
9	草	9469	5	9464
10	红色屋顶	10 516	5	10 511
11	灰色屋顶	16 911	5	16 906
12	塑料	3679	5	3674
13	裸土	9116	5	9111
14	道路	18 560	5	18 555
15	明亮物体	1136	5	1131
16	水	75 401	5	75 396
	总数	257 530	80	257 450

序号	地物名称	标记样本数	训练样本数	测试样本数
1	红色屋顶	14 041	5	14 036
2	路	3512	5	3507
3	裸土	21 821	5	21 816
4	棉花	163 285	5	163 280
5	棉花柴	6218	5	6213
6	油菜	44 557	5	44 552
7	白菜	24 103	5	24 098
8	小白菜	4054	5	4049
9	卷心菜	10 819	5	10 814
10	榨菜	12 394	5	12 389
11	菜心	11 015	5	11 010
12	青菜	8954	5	8949
13	上海青	22 507	5	22 502
14	莴苣	7356	5	7351
15	莴笋	1002	5	997
16	生菜	7262	5	7257
17	罗马生菜	3010	5	3005
18	胡萝卜	3217	5	3212
19	白萝卜	8712	5	8707
20	蒜薹	3486	5	3481
21	蚕豆	1328	5	1323
22	柿子树	4040	5	4035
	总数	386 693	110	386 583