A coupled DeepLab and Transformer approach for fine classification of crop cultivation types in remote sensing

doi:10.11947/j.AGCS.2024.20220692

Abstract

Abstract: How to accurately monitor the planting of different types of complex farmland crops by remote sensing is the key to the realization of agricultural area survey and crop yield estimation in the area of smart rural agriculture. In the current pixel level semantic segmentation of crop planting in high-resolution images, the deep convolution neural network is difficult to take into account the spatial multi-scale global features and local details, which leads to problems such as blurring boundary contours between various farmland plots and low internal integrity of the same farmland area. In view of these shortcomings, this paper designs and proposes a dual branch parallel feature fusion network (FDTNet) that couples DeepLabv3+and Transformer encoders to achieve fine remote sensing monitoring of crop planting. Firstly, DeepLabv3+and Transformer are embedded in FDTNet in parallel to capture the local and global features of farmland image respectively. Secondly, the coupled attention fusion module (CAFM) is used to effectively fuse the characteristics of the two features. Then, in the decoder stage, the convolutional block attention module (CBAM) is applied to enhance the weight of the effective features of the convolutional layer. Finally, the progressive multi-level feature fusion strategy is adopted to fully fuse the effective features in the encoder and deco-der, and output the feature map to achieve high-precision classification and recognition of late rice, middle rice, lotus root field, vegetable field and greenhouse. In order to verify the effectiveness of FDTNet network model in high-resolution crop classification application, this paper selects different high-resolution Yuhu dataset and Zhejiang dataset and experimental results of mIoU reach 74.7% and 81.4%, respectively. The mIoU of FDTNet can be 2.2% and 3.6% respectively higher than the existing deep learning methods, such as UNet, DeepLabv3, DeepLabv3+, ResT and Res-Swin. The results show that FDTNet has better classification performance than the compared methods in two types of farmland scenes, which have single texture and large sample size, or multiple texture and small sample size. The proposed FDTNet has a comprehensive ability to extract effective features of multiple category crops.

Key words: high-resolution remote sensing image, crop planting type, semantic segmentation, feature fusion, deep learning

CLC Number:

P237

LIN Yunhao, WANG Yanjun, LI Shaochun, CAI Hengfan. A coupled DeepLab and Transformer approach for fine classification of crop cultivation types in remote sensing[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(2): 353-366.

References

[1] ADÃO T, HRUŠKA J, PÁDUA L, et al. Hyperspectral imaging:a review on UAV-based sensors, data processing and applications for agriculture and forestry[J]. Remote Sensing, 2017, 9(11):1110.
[2] ZHONG Yanfei, WANG Xinyu, XU Yao, et al. Mini-UAV-borne hyperspectral remote sensing:from observation and processing to applications[J]. IEEE Geoscience and Remote Sensing Magazine, 2018, 6(4):46-62.
[3] 刘海启. 以精准农业驱动农业现代化加速现代农业数字化转型[J]. 中国农业资源与区划, 2019, 40(1):1-6, 73. LIU Haiqi. Accelerating the digital transformation of modern agriculture by driving the agricultural modernization with precision agriculture[J]. Chinese Journal of Agricultural Resources and Regional Planning, 2019, 40(1):1-6, 73.
[4] 唐华俊, 吴文斌, 杨鹏, 等. 农作物空间格局遥感监测研究进展[J]. 中国农业科学, 2010, 43(14):2879-2888. TANG Huajun, WU Wenbin, YANG Peng, et al. Recent progresses in monitoring crop spatial patterns by using remote sensing technologies[J]. Scientia Agricultura Sinica, 2010, 43(14):2879-2888.
[5] 赵春江. 农业遥感研究与应用进展[J]. 农业机械学报, 2014, 45(12):277-293. ZHAO Chunjiang. Advances of research and application in remote sensing for agriculture[J]. Transactions of the Chinese Society for Agricultural Machinery, 2014, 45(12):277-293.
[6] ZHONG Yanfei, HU Xin, LUO Chang, et al. WHU-Hi:UAV-borne hyperspectral with high spatial resolution (H²) benchmark datasets and classifier for precise crop identification based on deep convolutional neural network with CRF[J]. Remote Sensing of Environment, 2020, 250:112012.
[7] 刘巍, 吴志峰, 骆剑承, 等. 深度学习支持下的丘陵山区耕地高分辨率遥感信息分区分层提取方法[J]. 测绘学报, 2021, 50(1):105-116. DOI:10.11947/j.AGCS.2021.20190448. LIU Wei, WU Zhifeng, LUO Jiancheng, et al. A divided and stratified extraction method of high-resolution remote sensing information for cropland in hilly and mountainous areas based on deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(1):105-116.DOI:10.11947/j.AGCS.2021.20190448.
[8] 叶思菁. 大数据环境下遥感图谱应用方法研究:以作物干旱监测为例[J]. 测绘学报, 2018, 47(6):892.DOI:10.11947/j.AGCS.2018.20170535. YE Sijing. Research on application of remote sensing tupu-take monitoring of meteorological disaster for example[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):892.DOI:10.11947/j.AGCS.2018.20170535.
[9] CHAMORRO M J A, CUÉ LA ROSA L E, FEITOSA R Q, et al. Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171:188-201.
[10] 陈劲松, 黄健熙, 林珲, 等. 基于遥感信息和作物生长模型同化的水稻估产方法研究[J]. 中国科学:信息科学, 2010, 40(S1):173-183. CHEN Jinsong, HUANG Jianxi, LIN Hui, et al.Study on rice yield estimation method based on assimilation of remote sensing information and crop growth model[J]. Scientia Sinica (Informationis), 2010, 40(S1):173-183.
[11] DE LA CASA A, OVANDO G, BRESSANINI L, et al. Soybean crop coverage estimation from NDVI images with different spatial resolution to evaluate yield variability in a plot[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 146:531-547.
[12] BEERI O, PELED A. Geographical model for precise agriculture monitoring with real-time remote sensing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2009, 64(1):47-54.
[13] ASHAPURE A, JUNG J, YEOM J, et al. A novel framework to detect conventional tillage and no-tillage cropping system effect on cotton growth and development using multi-temporal UAS data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152:49-64.
[14] 蒙继华, 吴炳方, 杜鑫, 等. 遥感在精准农业中的应用进展及展望[J]. 国土资源遥感, 2011, 23(3):1-7. MENG Jihua, WU Bingfang, DU Xin, et al. A review and outlook of applying remote sensing to precision agriculture[J]. Remote Sen-sing for Land & Resources, 2011, 23(3):1-7.
[15] 李德仁, 童庆禧, 李荣兴, 等. 高分辨率对地观测的若干前沿科学问题[J]. 中国科学:地球科学, 2012, 42(6):805-813. LI Deren, TONG Qingxi, LI Rongxing, et al.Some frontier scientific problems of high-resolution earth observation[J]. Scientia Sinica (Terrae), 2012, 42(6):805-813.
[16] LICHTBLAU E, OSWALD C J. Classification of impervious land-use features using object-based image analysis and data fusion[J]. Computers, Environment and Urban Systems, 2019, 75:103-116.
[17] SALMAN N H, LIU C Q. Image segmentation and edge detection based on watershed techniques[J]. International Journal of Compu-ters and Applications, 2003, 25(4):258-263.
[18] RYDBERG A, BORGEFORS G. Integrated method for boundary delineation of agricultural fields in multispectral satellite images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2001, 39(11):2514-2520.
[19] MOUNTRAKIS G, IM J, OGOLE C. Support vector machines in remote sensing:a review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2011, 66(3):247-259.
[20] PIIROINEN R, HEISKANEN J, MÕTTUS M, et al. Classification of crops across heterogeneous agricultural landscape in Kenya using Aisa EAGLE imaging spectroscopy data[J]. International Journal of Applied Earth Observation and Geoinformation, 2015, 39:1-8.
[21] BELGIU M, DRǍGUŢ L. Random forest in remote sensing:a review of applications and future directions[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 114:24-31.
[22] ZHONG Yanfei, LIN Xuemei, ZHANG Liangpei. A support vector conditional random fields classifier with a mahalanobis distance boundary constraint for high spatial resolution remote sensing imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(4):1314-1330.
[23] ZHAO Ji, ZHONG Yanfei, ZHANG Liangpei. Detail-preserving smoothing classifier based on conditional random fields for high spatial resolution remote sensing imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(5):2440-2452.
[24] ZHAO Ji, ZHONG Yanfei, JIA Tianyi, et al. Spectral-spatial classification of hyperspectral imagery with cooperative game[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 135:31-42.
[25] WANG Hong, CHEN Xianzhong, ZHANG Tianxiang, et al. CCTNet:coupled CNN and transformer network for crop segmentation of remote sensing images[J]. Remote Sensing, 2022, 14(9):1956.
[26] WANG Wang, LI Shaochun, WANG Wang, et al. A simple deep learning network for classification of 3D mobile LiDAR point clouds[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(3):49-59.
[27] YANG S, GU L, LI X, et al. Crop classification method based on optimal feature selection and hybrid CNN-RF networks for multi-temporal remote sensing imagery[J]. Remote sensing, 2020, 12(19):3119.
[28] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA:IEEE, 2016:770-778.
[29] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston:IEEE, 2015:3431-3440.
[30] 胡明洪, 李佳田, 姚彦吉, 等. 结合多路径的高分辨率遥感影像建筑物提取SER-UNet算法[J]. 测绘学报, 2023, 52(5):808-817. DOI:10.11947/j.AGCS.2023.20210691. HU Minghong, LI Jiatian, YAO Yanji, et al. SER-UNet algorithm for building extraction from high-resolution remote sensing image combined with multipath[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(5):807-817, DOI:10.11947/j.AGCS.2023.20210691.
[31] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas:IEEE, 2016:770-778.
[32] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab:semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4):834-848.
[33] REN Y, ZHANG X, MA Y, et al. Full convolutional neural network based on multi-scale feature fusion for the class imbalance remote sensing image classification[J]. Remote Sensing, 2020, 12(21):3547.
[34] CHEN L C, ZHU Yukun, PAPANDREOU G, et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of 2018 Computer Vision European Conference. Munich:ACM Press, 2018:833-851.
[35] 许泽宇, 沈占锋, 李杨, 等. 增强型DeepLab算法和自适应损失函数的高分辨率遥感影像分类[J]. 遥感学报, 2022, 26(2):406-415. XU Zeyu, SHEN Zhanfeng, LI Yang, et al. Classification of high-resolution remote sensing images based on enhanced Deep Lab algorithm and adaptive loss function[J]. National Remote Sensing Bulletin, 2022, 26(2):406-415.
[36] REEDHA R, DERICQUEBOURG E, CANALS R, et al. Transformer neural network for weed and crop classification of high resolution UAV images[J]. Remote Sensing, 2022, 14(3):592.
[37] YUAN Li, HOU Qibin, JIANG Zihang, et al. VOLO:vision outlooker for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(5):6575-6586.
[38] LIU Ze, LIN Yutong, CAO Yue, et al. Swin Transformer:hierarchical Vision Transformer using Shifted Windows[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal:IEEE, 2021:9992-10002.
[39] DONG Xiaoyi, BAO Jianmin, CHEN Dongdong, et al. CSWin transformer:a general vision transformer backbone with cross-shaped windows[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans:IEEE, 2022:12114-12124.
[40] XIAO T, SINGH M, MINTUN E, et al. Early convolutions help transformers see better[J]. Advances in Neural Information Processing Systems, 2021, 34:30392-30400.
[41] ZHANG Q, YANG Y B. ResT:an efficient transformer for visual recognition[J]. Advances in Neural Information Processing Systems, 2021, 34:15475-15485.
[42] SRINIVAS A, LIN T Y, PARMAR N, et al. Bottleneck transformers for visual recognition[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville:IEEE, 2021:16514-16524.
[43] DAI Z, LIU H, LE Q V, et al. Coatnet:marrying convolution and attention for all data sizes[J]. Advances in Neural Information Processing Systems, 2021, 34:3965-3977.
[44] NIU B, FENG Q, CHEN B, et al. HSI-TransUNet:a transformer based semantic segmentation model for crop mapping from UAV hyperspectral imagery[J]. Computers and Electronics in Agriculture, 2022, 201:107297.
[45] PENG Zhiliang, HUANG Wei, GU Shanzhi, et al. Conformer:local features coupling global representations for visual recognition[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal:IEEE, 2021:357-366.
[46] ZHANG Yundong, LIU Huiye, HU Qiang. TransFuse:fusing transformers and CNNs for medical image segmentation[M]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2021. Cham:Springer International Publishing, 2021:14-24.
[47] DING L, LIN D, LIN Shaofu, et al. Looking outside the window:wide-context transformer for the semantic segmentation of high-resolution remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:1-13.
[48] WOO S, PARK J, LEE J Y, et al. CBAM:convolutional block attention module[C]//Proceedings of 2018 European Conference on Computer Vision. Cham:Springer, 2018:3-19.
[49] ZHOU Lichen, ZHANG Chuang, WU Ming. D-LinkNet:LinkNet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City:IEEE, 2018:192-1924.
[50] ZHANG Cheng, JIANG Wanshou, ZHANG Yuan, et al. Transformer and CNN hybrid deep neural network for semantic segmentation of very-high-resolution remote sensing imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:4408820.
[51] QIAN Z, CAO Y, SHI Z, et al. A semantic segmentation method for remote sensing images based on Deeplab v3[C]//Proceedings of 2021 International Conference on Big Data & Artificial Intelligence & Software Engineering. Zhuhai:IEEE, 2021:396-400.
[52] HU Jie, SHEN Li, SUN Gang. Squeeze-and-excitation networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City:IEEE, 2018:7132-7141.
[53] SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM:visual explanations from deep networks via gradient-based localization[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice:IEEE, 2017:618-626.