Hyperspectral image classification method based on hierarchical transformer network

doi:10.11947/j.AGCS.2023.20220540

Abstract

Abstract: Hyperspectral image classification, which assigns each pixel to predefined land cover categories, is of crucial importance in various Earth science tasks such as environmental mapping and other related fields. In recent years, scholars have attempted to utilize deep learning frameworks for hyperspectral image classification and achieved satisfactory results. However, these methods still have certain deficiencies in extracting spectral features. This paper proposes a hierarchical self-attention network (HSAN) for hyperspectral image classification based on the self-attention mechanism. Firstly, a skip-layer self-attention module is constructed for feature learning, leveraging the self-attention mechanism of Transformer to capture contextual information and enhance the contribution of relevant information. Secondly, a hierarchical fusion method is designed to further alleviate the loss of relevant information during the feature learning process and enhance the interplay of features at different hierarchical levels. Experimental results on the Pavia University and Houston2013 datasets demonstrate that the proposed framework outperforms other state-of-the-art hyperspectral image classification frameworks.

Key words: hyperspectral image classification, transformer, self-attention mechanism, hierarchical fusion

CLC Number:

P237

ZHANG Yichao, ZHENG Xiangtao, LU Xiaoqiang. Hyperspectral image classification method based on hierarchical transformer network[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(7): 1139-1147.

References

[1] AHMAD M, SHABBIR S, ROY S K, et al.Hyperspectral image classification-traditional to deep models:a survey for future prospects[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021.
[2] RASTI B, HONG Danfeng, HANG Renlong, et al. Feature extraction for hyperspectral imagery:the evolution from shallow to deep:overview and toolbox[J]. IEEE Geoscience and Remote Sensing Magazine, 2020, 8(4):60-88.
[3] XUE Zhaohui, NIE Xiangyu. Low-rank and sparse representation with adaptive neighborhood regularization for hyperspectral image classification[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(1):73-90. DOI:10.11947/j.JGGS.2022.0108
[4] 左溪冰,刘冰,余旭初,等.高光谱影像小样本分类的图卷积网络方法[J].测绘学报, 2021, 50(10):1358-1369. DOI:10.11947/j.AGCS.2021.20200155. ZUO Xibing, LIU Bing, YU Xuchu, et al. Graph convolutional network method for small sample classification of hyperspectral images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(10):1358-1369. DOI:10.11947/j. AGCS.2021.20200155.
[5] HAM J, CHEN Yangchi, CRAWFORD M M, et al. Investigation of the random forest framework for classification of hyperspectral data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2005, 43(3):492-501.
[6] CAMPS-VALLS G, BRUZZONE L. Kernel-based methods for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2005, 43(6):1351-1362.
[7] MA Li, CRAWFORD M M, TIAN Jinwei. Local manifold learning-based K-nearest-neighbor for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(11):4099-4109.
[8] HU Wei, HUANG Yangyu, WEI Li, et al. Deep convolutional neural networks for hyperspectral image classification[J]. Journal of Sensors, 2015.
[9] ZHAO Wenzhi, DU Shihong. Spectral-spatial feature extraction for hyperspectral image classification:a dimension reduction and deep learning approach[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(8):4544-4554.
[10] WANG Kexian, ZHENG Shunyi, LI Rui, et al. A deep double-channel dense network for hyperspectral image classification[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(4):46-62.
[11] LIU Xun, DENG Chenwei, CHANUSSOT J, et al. StfNet:a two-stream convolutional neural network for spatiotemporal image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(9):6552-6564.
[12] WU Hao, PRASAD S. Convolutional recurrent neural networks for hyperspectral data classification[J]. Remote Sensing, 2017, 9(3):298.
[13] YU Shiqi, JIA Sen, XU Chunyan. Convolutional neural networks for hyperspectral image classification[J]. Neurocomputing, 2017, 219:88-98.
[14] GAO Hongmin, YANG Yao, LI Chenming, et al. Joint alternate small convolution and feature reuse for hyperspectral image classification[J]. ISPRS International Journal of Geo-Information, 2018, 7(9):349.
[15] RAN Lingyan, ZHANG Yanning, WEI Wei, et al. A hyperspectral image classification framework with spatial pixel pair features[J]. Sensors, 2017, 17(10):2421.
[16] ZHONG Zilong, LI Jin, LUO Zhinming, et al. Spectral-spatial residual network for hyperspectral image classification:a 3D deep learning framework[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(2):847-858.
[17] LI Simin, ZHU Xueyu, LIU Yang, et al. Adaptive spatial-spectral feature learning for hyperspectral image classification[J]. IEEE Access, 2019, 7:61534-61547.
[18] ROY S K, MANNA S, SONG Tiecheng, et al. Attention-based adaptive spectral-spatial kernel ResNet for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(9):7831-7843.
[19] MOU Lichao, LU Xiaoqiang, LI Xuelong, et al. Nonlocal graph convolutional networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12):8246-8257.
[20] WAN Sheng, GONG Chen, ZHONG Ping, et al.Multiscale dynamic graph convolutional network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 58(5):3162-3177.
[21] WAN Sheng, PAN Shirui, ZHONG Ping, et al. Dual interactive graph convolutional networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:1-14.
[22] HONG Danfeng, GAO Lianru, YAO Jing, et al. Graph convolutional networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(7):5966-5978.
[23] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.
[24] HAN Kai, WANG Yunhe, CHEN Hanting, et al. A survey on vision transformer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(1):87-110.
[25] SUN Hao, ZHENG Xiangtao, LU Xiaoqiang, et al. Spectral-spatial attention network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 58(5):3232-3245.
[26] HONG Danfeng, HAN Zhu, YAO Jing, et al. Spectralformer:rethinking hyperspectral image classification with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:1-15.
[27] MA Wenping, Yang Qifan, Wu Yue, et al. Double-branch multi-attention mechanism network for hyperspectral image classification[J]. Remote Sensing, 2019, 11(11):1307.
[28] WOO S, PARK J, LEE J Y, et al.Cbam:convolutional block attention module[C]//Proceedings of 2018 European conference on computer vision. Switzerland:Springer, 2018:3-19.
[29] YANG Kai, SUN Hao, ZOU Chunbo, et al. Cross-attention spectral-spatial network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:1-14.
[30] LI Rui, ZHENG Shunyi, DUAN C, et al. Classification of hyperspectral image based on double-branch dual-attention mechanism network[J]. Remote Sensing, 2020, 12(3):582.
[31] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE conference on computer vision and pattern recognition. Cham:IEEE 2016:770-778.