A capsule network for hyperspectral image classification employing spatial-spectral feature

doi:10.11947/j.AGCS.2023.20220565

Abstract

Abstract: An efficient and stable deep learning classifier can improve the classification accuracy of hyperspectral remote sensing images. In order to deal with the insufficiency of the scalar neuron's limited feature expression ability and the inability to effectively model the spatial hierarchical relationship among features in convolutional neural networks, an end-to-end hyperspectral capsule network (H-CapsNet) was designed considering the characteristics of hyperspectral image. The main body of H-CapsNet is composed of encoder (Conv, PrimaryCaps and DigitCats) and decoder (fully connection layer). It mainly embeds channel and spatial attention modules at the network input to enhance the model's capture and recognition of spatial and spectral features, thereby improving the network's ability to focus and express features. Taking the hyperspectral images of Zhang-jia-gang city and two public datasets:University of Pavia and University of Houston, as examples, the performance of the proposed H-CapsNet was compared with traditional machine learning algorithms and several deep neural networks. The experimental results show that the H-CapsNet has achieved the best classification accuracy on three hyperspectral images with different resolutions, and the overall accuracy is improved by 2.36%~7.67%, 0.16%~11.8% and 1.75%~15.58% compared with other methods. In particular, the H-CapsNet has good adaptability to small pixel neighborhoods. When the image patch size is limited, it can still achieve relatively ideal classification results.

Key words: capsule network, deep learning, hyperspectral remote sensing, ZY-1 02D, land cover classification

CLC Number:

TP751:P237

DU Peijun, ZHANG Wei, ZHANG Peng, LIN Cong, GUO Shanchuan, HU Zezhou. A capsule network for hyperspectral image classification employing spatial-spectral feature[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(7): 1090-1104.

References

[1] 童庆禧, 张兵, 张立福. 中国高光谱遥感的前沿进展[J]. 遥感学报, 2016, 20(5):689-707. TONG Qingxi, ZHANG Bing, ZHANG Lifu. Current progress of hyperspectral remote sensing in China[J]. Journal of Remote Sensing, 2016, 20(5):689-707.
[2] GOETZ A F H. Three decades of hyperspectral remote sensing of the Earth:a personal view[J]. Remote Sensing of Environment, 2009, 113:S5-S16.
[3] 张良培, 李家艺. 高光谱图像稀疏信息处理综述与展望[J]. 遥感学报, 2016, 20(5):1091-1101. ZHANG Liangpei, LI Jiayi. Development and prospect of sparse representation-based hyperspectral image processing and analysis[J]. Journal of Remote Sensing, 2016, 20(5):1091-1101.
[4] 杜培军, 夏俊士, 薛朝辉, 等. 高光谱遥感影像分类研究进展[J]. 遥感学报, 2016, 20(2):236-256. DU Peijun, XIA Junshi, XUE Zhaohui, et al. Review of hyperspectral remote sensing image classification[J]. Journal of Remote Sensing, 2016, 20(2):236-256.
[5] GHAMISI P, MAGGIORI E, LI S T, et al. New frontiers in spectral-spatial hyperspectral image classification the latest advances based on mathematical morphology, Markov random fields, segmentation, sparse representation, and deep learning[J]. IEEE Geoscience and Remote Sensing Magazine, 2018, 6(3):10-43.
[6] ZHONG Y, HU X, LUO C, et al. WHU-Hi:UAV-borne hyperspectral with high spatial resolution (H²) benchmark datasets and classifier for precise crop identification based on deep convolutional neural network with CRF[J]. Remote Sensing of Environment, 2020, 250:112012.
[7] TAN K, WANG H, CHEN L, et al. Estimation of the spatial distribution of heavy metal in agricultural soils using airborne hyperspectral imaging and random forest[J]. Journal of Hazardous Materials, 2020, 382:120987.
[8] DARVISHZADEH R, ATZBERGER C, SKIDMORE A, et al. Mapping grassland leaf area index with airborne hyperspectral imagery:a comparison study of statistical approaches and inversion of radiative transfer models[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2011, 66(6):894-906.
[9] NASRABADI N M. Hyperspectral target detection:an overview of current and future challenges[J]. IEEE Signal Processing Magazine, 2014, 31(1):34-44.
[10] XUE Z, NIE X. Low-rank and sparse representation with adaptive neighborhood regularization for hyperspectral image classification[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(1):73-90.
[11] FU P, ZHANG W, YANG K, et al. A novel spectral analysis method for distinguishing heavy metal stress of maize due to copper and lead:RDA and EMD-PSD[J]. Ecotoxicology and Environmental Safety, 2020, 206:111211.
[12] LIU S, MARINELLI D, BRUZZONE L, et al. A review of change detection in multitemporal hyperspectral images:current techniques, applications, and challenges[J]. IEEE Geoscience and Remote Sensing Magazine, 2019, 7(2):140-58.
[13] CHANG S, DU B, ZHANG L. A subspace selection-based discriminative forest method for hyperspectral anomaly detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(6):4033-4046.
[14] JIA S, TANG G, ZHU J, et al. A novel ranking-based clustering approach for hyperspectral band selection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(1):88-102.
[15] TARAMELLI A, VALENTINI E, INNOCENTI C, et al. FHYL:field spectral libraries, airborne hyperspectral images and topographic and bathymetric LiDAR data for complex coastal mapping[C]//Proceedings of 2013 IEEE International Geoscience and Remote Sensing Symposium. Melbourne:IEEE, 2013.
[16] SU H, YAO W, WU Z, et al. Kernel low-rank representation with elastic net for China coastal wetland land cover classification using GF-5 hyperspectral imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171:238-252.
[17] LI Z, HUANG H, DUAN Y, et al. DLPNet:a deep manifold network for feature extraction of hyperspectral imagery[J]. Neural Networks, 2020, 129:7-18.
[18] KUMAR B, DIKSHIT O, GUPTA A, et al. Feature extraction for hyperspectral image classification:a review[J]. International Journal of Remote Sensing, 2020, 41(16):6248-6287.
[19] GHAMISI P, PLAZA J, CHEN Y, et al. Advanced spectral classifiers for hyperspectral images:a review[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(1):8-32.
[20] DU P J, XIA J S, ZHANG W, et al. Multiple classifier system for remote sensing image classification:a review[J]. Sensors (Basel), 2012, 12(4):4764-4792.
[21] BENEDIKTSSON J A, PESARESI M, ARNASON K. Classification and feature extraction for remote sensing images from urban areas based on morphological transformations[J]. IEEE Transactions on Geoscience and Remote Sensing, 2003, 41(9):1940-1949.
[22] DALLA M M, VILLA A, BENEDIKTSSON J A, et al. Classification of hyperspectral images by using extended morphological attribute profiles and independent component analysis[J]. IEEE Geoscience and Remote Sensing Letters, 2011, 8(3):542-546.
[23] JIA S, SHEN L L, LI Q Q. Gabor feature-based collaborative representation for hyperspectral imagery classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(2):1118-1129.
[24] ZHANG W, DU P J, LIN C, et al. An improved feature set for hyperspectral image classification:harmonic analysis optimized by multiscale guided filter[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:3903-3916.
[25] XIA J S, DU P J, HE X Y, et al. Hyperspectral remote sensing image classification based on rotation forest[J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(1):239-243.
[26] 魏立飞, 余铭, 钟燕飞, 等. 空-谱融合的条件随机场高光谱影像分类方法[J]. 测绘学报, 2020, 49(03):343-354. DOI:10.11947/j.AGCS.2020.20190042. WEI Lifei, YU Ming, ZHONG Yanfei, et al. Hyperspectral image classification method based on space-spectral fusion conditional random field[J].Acta Geodaetica et Cartographica Sinica, 2020, 49(3):343-354. DOI:10.11947/j.AGCS.2020.20190042.
[27] YUAN Q, SHEN H, LI T, et al. Deep learning in environmental remote sensing:achievements and challenges[J]. Remote Sensing of Environment, 2020, 241:111716.
[28] MA L, LIU Y, ZHANG X, et al. Deep learning in remote sensing applications:a meta-analysis and review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152:166-177.
[29] AUDEBERT N, LE SAUX B, LEFEVRE S. Deep learning for classification of hyperspectral data:a comparative review[J]. IEEE Geoscience and Remote Sensing Magazine, 2019, 7(2):159-173.
[30] HONG D, GAO L, YAO J, et al. Graph convolutional networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7):5966-5978.
[31] XUE Z, ZHANG M, LIU Y, et al. Attention-based second-order pooling network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021:1-16.
[32] HINTON G E, KRIZHEVSKY A, WANG S D. Transforming auto-encoders[C]//Proceedings of the 21st international conference on artificial neural networks. Espoo:Springer Berlin Heidelberg, 2011:44-51.
[33] SABOUR S, FROSST N, HINTON G E. Dynamic routing between capsules[J]. Advances in Neural Information Processing Systems, 2017, 30.
[34] HINTON G E, SABOUR S, FROSST N. Matrix capsules with EM routing[C]//Proceedings of 2018 International Conference on Learning Representations. Vancouver:OpenReview.net, 2018:1-15.
[35] PAOLETTI M E, HAUT J M, FERNANDEZ B, et al. Capsule networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(4):2145-2160.
[36] WANG X, TAN K, DU Q, et al. Caps-TripleGAN:GAN-assisted CapsNet for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(9):7232-7245.
[37] JIANG X, LIU W, ZHANG Y, et al. Spectral-spatial hyperspectral image classification using dual-channel capsule networks[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(6):1094-1098.
[38] YIN J, LI S, ZHU H, et al. Hyperspectral image classification using CapsNet with well-initialized shallow layers[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16(7):1095-1099.
[39] LI H C, WANG W Y, PAN L, et al. Robust capsule network based on maximum correntropy criterion for hyperspectral image classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:738-751.
[40] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553):436-444.
[41] GU J, WANG Z, KUEN J, et al. Recent advances in convolutional neural networks[J]. Pattern Recognition, 2018, 77:354-377.
[42] WOO S, PARK J, LEE J-Y, et al. CBAM:convolutional block attention module[C]//Proceedings of 2018 European Conference on Computer Vision. Munich:Springer, 2018:3-19.
[43] WANG Q, WU B, ZHU P, et al. ECA-Net:efficient channel attention for deep convolutional neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle:IEEE, 2020:11534-11542.
[44] DEBES C, MERENTITIS A, HEREMANS R, et al. Hyperspectral and LiDAR data fusion:outcome of the 2013 GRSS data fusion contest[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(6):2405-2418.
[45] 高奎亮, 余旭初, 张鹏强, 等. 利用胶囊网络实现高光谱影像空谱联合分类[J]. 武汉大学学报(信息科学版), 2022, 47(03):428-437. GAO Kuiliang, YU Xuchu, ZHANG Pengqiang, et al. Hyperspectral image spatial-spectral classification using a capsule network based method[J]. Geomatics and Information Science of Wuhan University, 2022, 47(3):428-437.
[46] MOUNTRAKIS G, IM J, OGOLE C. Support vector machines in remote sensing:a review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2011, 66(3):247-259.
[47] BELGIU M, DRǍGUŢ L. Random forest in remote sensing:a review of applications and future directions[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 114:24-31.
[48] XU Y, ZHANG L, DU B, et al. Spectral-spatial unified networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018:1-17.
[49] HU W, HUANG Y, WEI L, et al. Deep convolutional neural networks for hyperspectral image classification[J]. Journal of Sensors, 2015:1-12.
[50] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words:transformers for image recognition at scale[C]//Proceedings of 2021 International Conference on Learning Representations. Vienna:OpenReview.net, 2021:1-22.
[51] SHARMA V, DIBA A, TUYTELAARS T, et al. Hyperspectral CNN for image classification & band selection, with application to face recognition[R]. Belgium:KU Leuven, 2016.
[52] CHEN Y, JIANG H, LI C, et al. Deep feature extraction and classification of hyperspectral images based on convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(10):6232-6251.