Attention-guided feature fusion and joint learning for remote sensing image scene classification

doi:10.11947/j.AGCS.2023.20210659

Abstract

Abstract: Aiming at the difficulties of high-precision remote sensing image classification caused by scale variations, inter-class similarity and intra-class difference, a method with attention-guided feature fusion and joint learning is proposed for remote sensing image scene classification to make full use of multi-scale features extracted from the images. First, the deep convolutional neural network is used to extract three levels of feature maps from the images. Then, the residual attention mechanism is designed to enhance the semantic information and suppress the noise information of the feature maps. Finally, global average pooling is used to obtain the global information of the feature maps and to construct the feature vectors. Then the three levels of feature vectors are fused by connection.The three levels of feature vectors and the fusion result are classified in independent fully connected layers, respectively.During the training process, the joint loss is calculated to optimize the model's parameters. And multi-classifier decision-level fusion is adopted to improve the robustness of prediction. Experimental results on the UC Merced, AID and NWPU-RESISC45 datasets show that the proposed method can significantly improve the discrimination on similar scenes and scenes with intra-class difference. And compared with the similar method using multi-scale features, the overall accuracies are improved by 0.84%, 4.04% and 4.43%, respectively.

Key words: remote sensing image, scene classification, convolutional neural network, feature fusion, joint learning, attention mechanism

CLC Number:

P237

YU Donghang, XU Qing, ZHAO Chuan, GUO Haitao, LU Jun, LIN Yuzhun, LIU Xiangyun. Attention-guided feature fusion and joint learning for remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(4): 624-637.

References

[1] CHENG Gong, HAN Junwei, LU Xiaoqiang. Remote sensing image scene classification: benchmark and state of the art[J]. Proceedings of 2017 IEEE, 2017, 105(10): 1865-1883.
[2] YANG Y, NEWSAM S. Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems.San Jose, CA,USA: ACM Press, 2010: 270-279.
[3] XIA Guisong, HU Jingwen, HU Fan, et al. AID:a benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7): 3965-3981.
[4] 闫利, 朱睿希, 刘异, 等. 顾及遥感影像场景类别信息的视觉单词优化分类[J]. 遥感学报, 2017, 21(2): 280-290. YAN Li, ZHU Ruixi, LIU Yi, et al. Scene classification of remote sensing images by optimizing visual vocabulary concerning scene label information[J]. Journal of Remote Sensing, 2017, 21(2): 280-290.
[5] ZHAO Lijun, TANG Ping, HUO Lianzhi. Land-use scene classification using a concentric circle-structured multiscale bag-of-visual-words model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,2014, 7(12): 4620-4631.
[6] ZHU Qiqi, ZHONG Yanfei, ZHAO Bei, et al. Bag-of-visual-words scene classifier with local and global features for high spatial resolution remote sensing imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(6): 747-751.
[7] ZUO Zongcheng, ZHANG Wen, ZHANG Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(3): 39-49.
[8] PENATTI O A B, NOGUEIRA K, DOS SANTOS J A. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains? [C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Boston, MA, USA: IEEE, 2015: 44-51.
[9] HU Fan, XIA Guisong, HU Jingwen, et al. Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery[J]. Remote Sensing, 2015,7(11):14680-14707.
[10] DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA:IEEE, 2009: 248-255.
[11] GAO Yue, SHI Jun, LI Jun, et al. Remote sensing scene classification based on high-order graph convolutional network[J]. European Journal of Remote Sensing, 2021, 54(sup1):141-155.
[12] YU Yunlong, LI Xianzhi, LIU Fuxian. Attention GANs: unsupervised deep feature learning for aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing,2020, 58(1): 519-531.
[13] ZHANG Wei, TANG Ping, ZHAO Lijun. Remote sensing image scene classification using CNN-CapsNet[J]. Remote Sensing, 2019, 11(5):494.
[14] BAZI Y, BASHMAL L, RAHHAL M M A, et al. Vision transformers for remote sensing image classification[J]. Remote Sensing, 2021, 13(3): 516.
[15] HE Nanjun, FANG Leyuan, LI Shutao, et al. Skip-connected covariance network for remote sensing scene classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020,31(5): 1461-1474.
[16] ZHANG Bin, ZHANG Yongjun, WANG Shugen. A lightweight and discriminative model for remote sensing scene classification with multidilation pooling module[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(8): 2636-2653.
[17] 叶利华, 王磊, 张文文, 等. 高分辨率光学遥感场景分类的深度度量学习方法[J]. 测绘学报, 2019, 48(6): 698-707.DOI: 10.11947/j.AGCS.2019.20180434. YE Lihua, WANG Lei, ZHANG Wenwen, et al. Deep metric learning method for high resolution remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 698-707.DOI: 10.11947/j.AGCS.2019.20180434.
[18] YU Donghang, GUO Haitao, XU Qing, et al. Hierarchical attention and bilinear fusion for remote sensing image scene classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 6372-6383.
[19] DU Peijun, LI Erzhu, XIA Junshi, et al. Feature and model level fusion of pretrained CNN for remote sensing scene classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(8): 2600-2611.
[20] WANG Q, LIU S, CHANUSSOT J, et al. Scene classification with recurrent attention of VHR remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(2): 1155-1167.
[21] ZHANG Dong, LI Nan, YE Qiaolin. Positional context aggregation network for remote sensing scene classification[J]. IEEE Geoscience and Remote Sensing Letters,2020, 17(6): 943-947.
[22] XUE Wei, DAI Xiangyang, LIU Li. Remote sensing scene classification based on multi-structure deep features fusion[J]. IEEE Access, 2020, 8: 28746-28755.
[23] YU Donghang, XU Qing, GUO Haitao, et al. Aggregating features from dual paths for remote sensing image scene classification[J]. IEEE Access, 2022,10:16740-16755.
[24] 马欣悦, 王梨名, 祁昆仑, 等. 基于多尺度循环注意力网络的遥感影像场景分类方法[J]. 地球科学, 2021, 46(10): 3740-3752. MA Xinyue, WANG Liming, QI Kunlun, et al. Remote sensing image scene classification method based on multi-scale cyclic attention network[J]. Earth Science, 2021, 46(10): 3740-3752.
[25] 张桐, 郑恩让, 沈钧戈, 等. 基于深度多分支特征融合网络的光学遥感场景分类[J]. 光子学报, 2020, 49(5): 166-177. ZHANG Tong, ZHENG Enrang, SHEN Junge, et al. Remote sensing image scene classification based on deep multi-branch feature fusion network[J]. Acta Photonica Sinica, 2020, 49(5):166-177.
[26] SUN Xiongli, ZHU Qiqi, QIN Qianqing. A multi-level convolution Pyramid semantic fusion framework for high-resolution remote sensing image scene classification and annotation[J].IEEE Access, 2021,9: 18195-18208.
[27] CAO Ran, FANG Leyuan, LU Ting, et al. Self-attention-based deep feature fusion for remote sensing scene classification[J]. IEEE Geoscience and Remote Sensing Letters, 2021,18(1):43-47.
[28] CHAIB Souleyman, LIU Huan, GU Yanfeng, et al. Deep feature fusion for VHR remote sensing scene classification[J].IEEE Transactions on Geoscience and Remote Sensing, 2017,55(8):4775-4784.
[29] WANG Fei, JIANG Mengqing, QIAN Chen, et al. Residual attention network for image classification[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Honolulu, HI, USA: IEEE, 2017: 6450-6458.
[30] DU R, CHANG D, BHUNIA A K, et al. Fine-grained visual classification via progressive multi-granularity training of jigsaw patches[C]//Proceedings of 2020 European Conference on Computer Vision.Glasgow, UK:Springer, 2020: 153-168.
[31] SONG Jianwei, YANG Ruoyu. Feature boosting, suppression, and diversification for fine-grained visual classification[C]//Proceedings of 2021 International Joint Conference on Neural Networks (IJCNN).Shenzhen, China:IEEE, 2021: 1-8.
[32] YU Donghang, XU Qing, LIU Xiangyun, et al. Joint learning using multiscale attention-enhanced features for remote sensing image scene classification[J]. Journal of Applied Remote Sensing, 2022, 16(3): 036506.
[33] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2023-01-25]. https://arxiv.org/abs/1409.1556.
[34] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 770-778.
[35] RADOSAVOVIC I, KOSARAJUR P, GIRSHICK R, et al. Designing network design spaces[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Seattle, WA, USA: IEEE, 2020:10425-10433.
[36] SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 2818-2826.
[37] MVLLER R, KORNBLITH S, HINTON G. When does label smoothing help? [EB/OL].[2022-03-15]. https://arxiv.org/abs/1906.02629.
[38] SELVARAJU R, COGSWELL M, DAS A, et al. Grad-CAM:visual explanations from deep networks via gradient-based localization[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy:IEEE, 2017:618-626.
[39] LAURENS V, HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(2605):2579-2605.
[40] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023.
[41] WOO S, PARK J, LEE J. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018:3-19.
[42] XIE Jie, HE Nanjun, FANG Leyuan, et al. Scale-free convolutional neural network for remote sensing scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(9): 6916-6928.
[43] BI Qi, QIN Kun, ZHANG Han, et al. RADC-Net: a residual attention based convolution network for aerial scene classification[J]. Neurocomputing, 2020, 377: 345-359.
[44] 郭东恩, 夏英, 罗小波, 等. 基于有监督对比学习的遥感图像场景分类[J]. 光子学报, 2021, 50(7): 87-98. GUO Dongen, XIA Ying, LUO Xiaobo, et al. Remote sensing image scene classification based on supervised contrastive learning[J]. Acta Photonica Sinica, 2021, 50(7):87-98.
[45] 施慧慧, 徐雁南, 滕文秀, 等. 高分辨率遥感影像深度迁移可变形卷积的场景分类法[J]. 测绘学报, 2021, 50(5): 652-663.DOI: 10.11947/j.AGCS.2021.20200190. SHI Huihui, XU Yannan, TENG Wenxiu, et al. Scene classification of high-resolution remote sensing imagery based on deep transfer deformable convolutional neural networks[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(5): 652-663.DOI: 10.11947/j.AGCS.2021.20200190.
[46] TANG Xu, MA Qiushuo, ZHANG Xiangrong, et al. Attention consistent network for remote sensing scene classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 2030-2045.
[47] WANG Xin, DUAN Lin, SHI Aiye, et al. Multilevel feature fusion networks with adaptive channel dimensionality reduction for remote sensing scene classification[J]. IEEE Geoscience and Remote Sensing Letters,2022, 19: 1-5.