Improved U-Net remote sensing image semantic segmentation method

doi:10.11947/j.AGCS.2023.20210684

Abstract

Abstract: Semantic segmentation of remote sensing images by deep neural network is an important content of remote sensing intelligent interpretation, which plays a very important role in urban planning, disaster assessment, agricultural production and other fields. High resolution remote sensing images are characterized by complex background, diverse scales and irregular shape, etc. Therefore, using natural scene semantic segmentation methods to process remote sensing images often has the problem of low segmentation accuracy. Based on the U-Net model, a multi-scale skip connection method is proposed to integrate semantic features of different levels and obtain accurate segmentation boundary and location information. Attention mechanism and pyramid pooling are introduced to solve the problem of fine segmentation in complex background. In order to verify the effectiveness of our proposed method, experiments were carried out on the WHDLD and LandCover.ai dataset and compared with the mainstream semantic segmentation methods. The experimental results show that the proposed method outperforms other comparison methods, with mIoU reaching 74.28% and 82.04% respectively, and with average of F₁ score reaching 84.47% and 89.76% respectively; compared with the segmentation results of U-Net, the value of IoU improves significantly in buildings, roads and other categories with a relatively small proportion, and is superior to other comparison methods.

Key words: remote sensing semantic segmentation, U-Net, attention mechanism, multi-scale skip connetion, pyramid pooling

CLC Number:

P237

HU Gongming, YANG Chuncheng, XU Li, SHANG Haibin, WANG Zefan, QIN Zhilong. Improved U-Net remote sensing image semantic segmentation method[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(6): 980-989.

References

[1] 李道纪, 郭海涛, 卢俊, 等. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020,49(8):1051-1064. DOI: 10.11947/j.AGCS.2020.20190407. LI Daoji, GUO Haitao, LU Jun,et al. A remote sensing image classfication procedure based on multilevel attention fusion U-Net[J].Acta Geodaetica et Cartographica Sinica, 2020,49(8):1051-1064. DOI: 10.11947/j.AGCS.2020.20190407.
[2] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[3] SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651.
[4] KAMPFFMEYER M, SALBERG A B, JENSSEN R. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Las Vegas:IEEE,2016: 680-688.
[5] RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]//Proceedings of 2015 International Conference on Medical Image Computing and Computer-assisted Intervention.[S.l.]: Springer,2015.
[6] PAN Xuran, YANG Fan, GAO Lianru, et al. Building extraction from high-resolution aerial imagery using a generative adversarial network with spatial and channel attention mechanisms[J]. Remote Sensing, 2019,11(8):917.
[7] 张玉鑫, 颜青松, 邓非. 高分辨率遥感影像建筑物提取多路径RSU网络法[J]. 测绘学报,2022,51(1):135-144. DOI:10.11947/j.AGCS.2021.20200508. ZAHNG Yuxin,YAN Qingsong,DENG Fei.Multi-path RSU network method for high-resolution remote sensing image building extraction[J]. Acta Geodaetica et Cartographica Sinica,2022,51(1):135-144.DOI:10.11947/j.AGCS.2021.20200508.
[8] HE Hao, WANG Wang, WANG Wang, et al.A road extraction method for remote sensing image based on encoder-de-coder network[J]. Journal of Geodesy and Geoinformation Science,2020,3(2): 415-426.
[9] ZHOU Z, SIDDIQUEE M M R, TAJBAKHSH N, et al. UNet: redesigning skip connections to exploit multiscale features in image segmentation[J]. IEEE Transactions on Medical Imaging, 2020, 39(6): 1856-1867.
[10] MNIH V, HEESS N, GRAVES A. Recurrent models of visual attention: advances in neural information processing systems[EB/OL]. [2022-11-20].https://arxiv.org/abs/1406.6247.
[11] HU Jie, SHEN Li, SUN Gang. Squeeze-and-excitation networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City:IEEE, 2018: 7132-7141.
[12] ROY A G, NAVAB N, WACHINGER C. Concurrent spatial and channel 'squeeze & excitation’ in fully convolutional networks[C]//Proceedings of 2018 Medical Image Computing and Computer Assisted Intervention. Cham: Springer, 2018: 421-429.
[13] OKTAY O, SCHLEMPER J, FOLGOC L L, et al. Attention U-net: learning where to look for the pancreas[EB/OL]. [2022-11-24]. https://arxiv.org/abs/1804.03999.
[14] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.
[15] ZHAO Hengshuang, SHI Jianping, QI Xiaojuan, et al. Pyramid scene parsing network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu:IEEE, 2017: 6230-6239.
[16] YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[EB/OL].[2022-12-04].https://arxiv.org/abs/1511.07122.
[17] IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on International Conference on Machine Learning. New York: ACM, 2015: 448-456.
[18] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[EB/OL].[2023-01-04]. https://arxiv.org/abs/1412.7062.
[19] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[20] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL].[2023-03-24]. https://arxiv.org/abs/1706.05587.
[21] CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of 15th European Conference of Computer Vision.Munich:ACM, 2018: 833-851.
[22] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J].Computer Science, 2014,10(1409):1556.
[23] ZHOU B, KHOSLA A, LAPEDRIZA A, et al. Object detectors emerge in deep scene CNNs[C]//Proceedings of 2015 International Conference on Learning Representations. San Diego:ICLR,2015.
[24] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV).Venice:IEEE, 2017: 2999-3007.
[25] SHAO Zhenfeng, YANG Ke, ZHOU Weixun. Performance evaluation of single-label and multi-label remote sensing image retrieval using a dense labeling dataset[J]. Remote Sensing, 2018, 10(6): 964.
[26] SHAO Zhenfeng, ZHOU Weixun, DENG Xueqing, et al. Multilabel remote sensing image retrieval based on fully convolutional network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 318-328.
[27] BOGUSZEWSKI A, BATORSKI D, ZIEMBA-JANKOWSKA N, et al. LandCover.ai: dataset for automatic mapping of buildings, woodlands and water from aerial imagery[J].Journal of the Indian Society of Remote Sensing,2018, 46:2057-2068.
[28] CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Honolulu:IEEE, 2017: 1800-1807.