Scene classification of high-resolution remote sensing imagery based on deep transfer deformable convolutional neural networks

doi:10.11947/j.AGCS.2021.20200190

Abstract

Abstract: In recent years, scene classification of high-resolution remote sensing images based on deep convolutional neural networks has become the focus of attention. Because of the existing deep convolution neural network is not robust to the geometric deformation of remote sensing scene image, we proposed a novel scene classification method for high-resolution remote sensing image, based on the deep transfer deformable convolutional neural networks (DTDCNN). Specifically, the depth features of remote sensing image are extracted by using the trained depth model on the large-scale natural scene dataset (ImageNet), then, the deformable convolution layer is introduced to learn the depth features which are robust to the geometric deformation of remote sensing scene.The results show that:the accuracy of DTDCNN on AID, UC-Merced and NWPU-RESISC45 datasets is improved by 4.25%, 1.9% and 4.83% after adding the deformable convolution, respectively. By the adaptive adjustment of the receptive field for different objects in the scene, DTDCNN enhances the ability of spatial sampling position, and, effectively improves the accuracy of remote sensing scene classification.

Key words: remote sensing, scene classification, convolutional neural networks, deformable convolutional, transfer learning

CLC Number:

P237

SHI Huihui, XU Yannan, TENG Wenxiu, WANG Ni. Scene classification of high-resolution remote sensing imagery based on deep transfer deformable convolutional neural networks[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(5): 652-663.

References

[1] 李德仁, 童庆禧, 李荣兴, 等. 高分辨率对地观测的若干前沿科学问题[J]. 中国科学(地球科学), 2012, 42(6):805-813. LI Deren, TONG Qingxi, LI Rongxing, et al. Current issues in high-resolution Earth observation technology[J]. Science China Earth Sciences, 2012, 42(6):805-813.
[2] YANG Yi, NEWSAM S. Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems.[S.l.]:ACM, 2010:270-279.
[3] CHERIYADAT A M. Unsupervised feature learning for aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2014, 52(1):439-451.
[4] 张帆. 面向高分辨率遥感影像分析的深度学习方法研究[D]. 武汉:武汉大学, 2017. ZHANG Fan. Deep learning for very high resolution remote sensing data analysis[D]. Wuhan:Wuhan University, 2017.
[5] ZHU Qiqi, ZHONG Yanfei, ZHAO Bei, et al. Bag-of-Visual-Words scene classifier with local and global features for high spatial resolution remote sensing imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(6):747-751.
[6] 钱晓亮, 李佳, 程塨, 等. 特征提取策略对高分辨率遥感图像场景分类性能影响的评估[J]. 遥感学报, 2018, 22(5):758-776. QIAN Xiaoliang, LI Jia, CHENG Gong, et al. Evaluation of the effect of feature extraction strategy on the performance of high-resolution remote sensing image scene classification[J]. Journal of Remote Sensing, 2018, 22(5):758-776.
[7] 闫苗, 赵红东, 李宇海, 等. 基于卷积神经网络的高光谱遥感地物多分类识别[J]. 激光与光电子学进展, 2019, 56(2):021702. YAN Miao, ZHAO Hongdong, LI Yuhai, et al. Multi-classification and recognition of hyperspectral remote sensing objects based on convolutional neural network[J]. Laser & Optoelectronics Progress, 2019, 56(2):021702.
[8] 龚希, 吴亮, 谢忠, 等. 融合全局和局部深度特征的高分辨率遥感影像场景分类方法[J]. 光学学报, 2019, 39(3):0301002. GONG Xi, WU Liang, XIE Zhong, et al. Classification method of high-resolution remote sensing scenes based on fusion of global and local deep features[J]. Acta Optica Sinica, 2019, 39(3):0301002.
[9] MARMANIS D, DATCU M, ESCH T, et al. Deep learning earth observation classification using ImageNet pretrained networks[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(1):105-109.
[10] 庄福振,罗平,何清,等.远移学习研究进展[J].软件学报,2015,26(1):26-39. ZHUANG Fuzhen, LUO Ping, HE Qing, et al. Survey on transter learning research[J]. Journal of Software, 2015, 26(1):26-39.
[11] DENG Jia, DONG Wei, SOCHER R, et al. ImageNet:a large-Scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2009). Miami, FL:IEEE, 2009.
[12] 谭琨, 王雪, 杜培军. 结合深度学习和半监督学习的遥感影像分类进展[J]. 中国图象图形学报, 2019, 24(11):1823-1841. TAN Kun, WANG Xue, DU Peijun. Research progress of the remote sensing classification combining deep learning and semi-supervised learning[J]. Journal of Image and Graphics, 2019, 24(11):1823-1841.
[13] PENATTI O A B, NOGUEIRA K, SANTOS J A D. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Boston, MA:IEEE, 2015.
[14] BASU S, GANGULY S, MUKHOPADHYAY S, et al. DeepSat:a learning framework for satellite imagery[C]//Proceedings of the 23rd SIGSPATIAL International Conference on Advances in Geographic Information Systems.[S.l.]:ACM, 2015.
[15] 蔡博文, 王树根, 王磊, 等. 基于深度学习模型的城市高分辨率遥感影像不透水面提取[J]. 地球信息科学学报, 2019, 21(9):1420-1429. CAI Bowen, WANG Shugen, WANG Lei, et al. Extraction of urban impervious surface from high-resolution remote sensing imagery based on deep learning[J]. Journal of Geo-information Science, 2019, 21(9):1420-1429.
[16] DAI Jifeng, QI Haozhi, XIONG Yuwen, et al. Deformable convolutional networks[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy:IEEE, 2017.
[17] ZUO Zongcheng, ZHANG Wen, ZHANG Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(3):39-49. DOI:10.11947/j.JGGS.2020.0304.
[18] ZHANG Wei, TANG Ping, ZHAO Lijun. Remote sensing image scene classification using CNN-CapsNet[J]. Remote Sensing, 2019, 11(5):494.
[19] QI Kunlun, GUAN Qingfeng, YANG Chao, et al. Concentric circle pooling in deep convolutional networks for remote sensing scene classification[J]. Remote Sensing, 2018, 10(6):934.
[20] GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Journal of Geodesy and Geoinformation Science, 2018, 1(1):1-15. DOI:10.11947/j.JGGS.2018.0101.
[21] 滕文秀, 温小荣, 王妮, 等. 基于深度迁移学习的无人机高分影像树种分类与制图[J]. 激光与光电子学进展, 2019, 56(7):072801. TENG Wenxiu, WEN Xiaorong, WANG Ni, et al. Tree species classification and mapping based on deep transfer learning with unmanned aerial vehicle high resolution images[J]. Laser & Optoelectronics Progress, 2019, 56(7):072801.
[22] PAN S J, YANG Qiang. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10):1345-1359.
[23] ZHU Xizhou, HU Han, LIN S, et al. Deformable ConvNets V2:more deformable, better results[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA:IEEE, 2019:9308-9316.
[24] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV:IEEE, 2016.
[25] 叶利华, 王磊, 张文文, 等. 高分辨率光学遥感场景分类的深度度量学习方法[J]. 测绘学报, 2019, 48(6):698-707. DOI:10.11947/j.AGCS.2019.20180434. YE Lihua, WANG Lei, ZHANG Wenwen, et al. Deep metric learning method for high resolution remote sensing image scene classification[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6):698-707. DOI:10.11947/j.AGCS.2019.20180434.
[26] XIA Guisong, HU Jingwen, HU Fan, et al. AID:a benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7):3965-3981.
[27] 滕文秀, 王妮, 陈泰生, 等. 基于深度对抗域适应的高分辨率遥感影像跨域分类[J]. 激光与光电子学进展, 2019, 56(11):112801. TENG Wenxiu, WANG Ni, CHEN Taisheng, et al. Deep adversarial domain adaptation method for cross-domain classification in high-resolution remote sensing images[J]. Laser & Optoelectronics Progress, 2019, 56(11):112801.
[28] CHENG Gong, HAN Junwei, LU Xiaoqiang. Remote sensing image scene classification:benchmark and state of the art[J]. Proceedings of the IEEE, 2017, 105(10):1865-1883.
[29] WEI Yufan, LUO Xiaobo, HU Lixin, et al. An improved unsupervised representation learning generative adversarial network for remote sensing image scene classification[J]. Remote Sensing Letters, 2020, 11(6):598-607.
[30] LIU Xuning, ZHOU Yong, ZHAO Jiaqi, et al. Siamese convolutional neural networks for remote sensing scene classification[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16(8):1200-1204.
[31] CHAIB S, LIU Huan, GU Yanfeng, et al. Deep feature fusion for VHR remote sensing scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(8):4775-4784.
[32] WANG Guoli, FAN Bin, XIANG Shiming, et al. Aggregating rich hierarchical features for scene classification in remote sensing imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2017, 10(9):4104-4115.
[33] HUANG Gao, LIU Zhuang, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI:IEEE, 2018.
[34] WANG Chen, LIN Wei, TANG Pengfei. Multiple resolution block feature for remote-sensing scene classification[J]. International Journal of Remote Sensing, 2019, 40(17-18):6884-6904.
[35] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//Proceedings of International Conference on Learning Representations. San Diego:ICLR, 2015:1-5.
[36] WANG Yuliang, LI Mingshi. Urban impervious surface detection from remote sensing images:a review of the methods and challenges[J]. IEEE Geoscience and Remote Sensing Magazine, 2019, 7(3):64-93.
[37] WANG Yuliang, SU Huiyi, LI Ming. An improved model based detection of urban impervious surfaces using multiple features extracted from Rosis-3 hyperspectral images[J]. Remote Sensing, 2019, 11(2):136.