Multi-scale building instance refinement extraction from remote sensing images by fusing with decentralized adaptive attention mechanism

doi:10.11947/j.AGCS.2023.20220322

Abstract

Abstract: The accurate and efficient automatic extraction of building footprints from remote sensing images has a wide range of applications. Since the buildings in remote sensing images have different types, scales, shapes and backgrounds, the existing methods, to varying degrees, suffer from the problems of missing small-scale buildings, blurred contour boundaries, and inability to distinguish individual building instances. Therefore, this paper proposed a multi-scale building instance refinement extraction convolutional neural network(MBRef-CNN) fusing with decentralized adaptive attention mechanism for remote sensing images. First, a feature pyramid network fused with split-attention and adaptive attention mechanism (SA-FPN) was used to learn multi-scale building features. Then, according to the multi-scale features, the region proposal network (RPN) was used to detect the location of individual building instances. Finally, the boundary refinement network (BndRN) was used to iteratively acquire the precise building masks. On WHU aerial imagery dataset, the comparison experiments were conducted with the existing popular segmentation methods. The results show that the accuracy of the proposed method in this paper is higher than the others. Moreover, the MBRef-CNN shows good comprehensive performance in multi-scale building extraction, and has obvious accuracy advantages in small-scale building extraction.

Key words: building refinement extraction, split-attention networks, adaptive attention mechanism, multi-scale, remote sensing images, deep learning

CLC Number:

P231

JIANG Baode, HANG Wei, XU Shaofen, WU Yong. Multi-scale building instance refinement extraction from remote sensing images by fusing with decentralized adaptive attention mechanism[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(9): 1504-1514.

References

[1] RATHORE M M, AHMAD A, PAUL A, et al. Urban planning and building smart cities based on the Internet of Things using big data analytics[J]. Computer Networks, 2016, 101: 63-80.
[2] XU Hao, ZHU Panpan, LUO Xiaobo, et al. Extracting buildings from remote sensing images using a multitask encoder-decoder network with boundary refinement[J]. Remote Sensing, 2022, 14(3): 564.
[3] YU Yongtao, LIU Chao, GAO Junyong, et al. Building extraction from remote sensing imagery with a high-resolution capsule network[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5.
[4] LUO Lin, LI Pengpeng, YAN Xuesong. Deep learning-based building extraction from remote sensing images: a comprehensive review[J]. Energies, 2021, 14(23): 7982.
[5] SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651.
[6] XU Yongyang, WU Liang, XIE Zhong, et al. Building extraction in very high resolution remote sensing imagery using deep learning and guided filters[J]. Remote Sensing, 2018, 10(1): 144.
[7] SHAO Zhenfeng, TANG Penghao, WANG Zhongyuan, et al. BRRNet: a fully convolutional neural network for automatic building extraction from high-resolution remote sensing images[J]. Remote Sensing, 2020, 12(6): 1050.
[8] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 936-944.
[9] GUO Haonan, SHI Qian, DU Bo, et al. Scene-driven multitask parallel attention network for building extraction in high-resolution remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(5): 4287-4306.
[10] HU Qintao, ZHEN Liangli, MAO Yao, et al. Automated building extraction using satellite remote sensing imagery[J]. Automation in Construction, 2021, 123: 103509.
[11] HU Jie, SHEN Li, ALBANIE S, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023.
[12] ZHU Qing, LIAO Cheng, HU Han, et al. MAP-net: multiple attending path neural network for building footprint extraction from remote sensed imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 6169-6181.
[13] HE Kaiming, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]// Proceedings of 2017 IEEE International Conference on Computer Vision. Venice:IEEE, 2017: 2980-2988.
[14] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
[15] CHEN Kai, PANG Jiangmiao, WANG Jiaqi, et al. Hybrid task cascade for instance segmentation[C]// Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 4969-4978.
[16] GAO Naiyu, SHAN Yanhu, WANG Yupei, et al. SSAP: single-shot instance segmentation with affinity pyramid[C]// Proceedings of 2019 IEEE International Conference on Computer Vision. Seoul: IEEE, 2019: 642-651.
[17] ARNAB A, TORR P H S. Pixelwise instance segmentation with a dynamically instantiated network[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 441-450.
[18] SHRESTHA S, VANNESCHI L. Improved fully convolutional network with conditional random fields for building extraction[J]. Remote Sensing, 2018, 10(7): 1135.
[19] YANG Guang, ZHANG Qian, ZHANG Guixu. EANet: edge-aware network for the extraction of buildings from aerial images[J]. Remote Sensing, 2020, 12(13): 2161.
[20] JUNG H, CHOI H S, KANG M. Boundary enhancement semantic segmentation for building extraction from remote sensed image[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-12.
[21] ZHANG Hang, WU Chongruo, ZHANG Zhongyue, et al. ResNeSt: split-attention networks[C]// Proceedings of 2022 IEEE Conference on Computer Vision and Pattern Recognition Workshops. New Orleans: IEEE, 2022: 2735-2745.
[22] LI Xiang, WANG Wenhai, HU Xiaolin, et al. Selective kernel networks[C]// Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 510-519.
[23] KIM J H, LEE H, HONG S J, et al. Objects segmentation from high-resolution aerial images using U-Net with pyramid pooling layers[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16(1): 115-119.
[24] GIRSHICK R. Fast R-CNN[C]// Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago:IEEE, 2015: 1440-1448.
[25] 季顺平, 魏世清. 遥感影像建筑物提取的卷积神经元网络与开源数据集方法[J]. 测绘学报, 2019, 48(4): 448-459.DOI: 10.11947/j.AGCS.2019.20180206. JI Shunping, WEI Shiqing. Building extraction via convolutional neural networks from an open remote sensing building dataset[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(4): 448-459.DOI: 10.11947/j.AGCS.2019.20180206.
[26] MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Can semantic labeling methods generalize to any city?The inria aerial image labeling benchmark[C]// Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). Fort Worth: IEEE, 2017: 3226-3229.
[27] ZHANG Hang, DANA K, SHI Jianping, et al. Context encoding for semantic segmentation[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7151-7160.
[28] CHINCHOR N. MUC-4 evaluation metrics[C]//Proceedings of the 4th Conference on Message Understanding. New York: ACM Press, 1992: 22-29.