Two-stream boundary constraints and relativistic generation adversarial network for building contour regularization

doi:10.11947/j.AGCS.2024.20230056

Abstract

Abstract:

Building extraction from high-resolution remote sensing images is still a hot and difficult research topic in the field of remote sensing application and cartography. Although the introduction of deep learning method has greatly improved the accuracy of building segmentation, the problems of irregular contour and unclear boundary in building segmentation mask still exist. In order to obtain regular building contours and clear boundaries, this paper proposes a method of building contour regularization based on two-stream boundary constraint and relativistic generation adversarial network. The network is composed of a two-stream boundary constraint generator and a relativistic average discriminator. The two-stream boundary constraint generator fuses the boundary details of remote sensing images and input labels through the two-stream network architecture and boundary loss function, thus generating regular building contours. The relativistic average discriminator forces the generator to generate a more realistic building mask by evaluating the quality difference between the ground truth label and the generated regularization result. To verify the performance of the model and explore the reasons for the performance improvement, two groups of experiments were designed on WHU building dataset and Inria aerial image labeling dataset, including a comparative experiment and ablation experiment. The experimental results show that this method can generate regularization results that match ground truth labels, and it has obvious advantages in solving the problems of blurred boundaries and irregular contours of segmentation masks.

Key words: building contour regularization, boundary constraint, generative adversarial networks, high-resolution remote sensing image

CLC Number:

P283

Jichong YIN, Fang WU, Renjian ZHAI, Yue QIU, Xianyong GONG, Ruixing XING. Two-stream boundary constraints and relativistic generation adversarial network for building contour regularization[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(7): 1444-1457.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Tab.1

Fig.7

Fig.8

Fig.9

Fig.10

Tab.3

References 44

[1]	龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6):693-704. DOI: 10.11947/j.AGCS.2018.20170640.
	GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):693-704. DOI: 10.11947/j.AGCS.2018.20170640.
[2]	龚健雅. 人工智能时代测绘遥感技术的发展机遇与挑战[J]. 武汉大学学报(信息科学版), 2018, 43(12):1788-1796.
	GONG Jianya. Chances and challenges for development of surveying and remote sensing in the age of artificial intelligence[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12):1788-1796.
[3]	ALSHEHHI R, MARPU P R, WOON W L, et al. Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 130:139-149.
[4]	DORNAIKA F, MOUJAHID A, EL MERABET Y, et al. Building detection from orthophotos using a machine learning approach: an empirical study on image segmentation and descriptors[J]. Expert Systems with Applications, 2016, 58:130-142.
[5]	CHEN Z, LI S, XU Y, et al. Correg-YOLOV3: a method for dense buildings detection in high-resolution remote sensing images[J]. Journal of Geodesy and Geoinformation Science, 2023, 6(2):51-61.
[6]	JI Shunping, WEI Shiqing, LU Meng. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(1):574-586.
[7]	张玉鑫, 颜青松, 邓非. 高分辨率遥感影像建筑物提取多路径RSU网络法[J]. 测绘学报, 2022, 51(1):135-144. DOI: 10.11947/j.AGCS.2021.20200508.
	ZHANG Yuxin, YAN Qingsong, DENG Fei. Multi-path RSU network method for high-resolution remote sensing image building extraction[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(1):135-144. DOI: 10.11947/j.AGCS.2021.20200508.
[8]	季顺平, 魏世清. 遥感影像建筑物提取的卷积神经元网络与开源数据集方法[J]. 测绘学报, 2019, 48(4):448-459. DOI: 10.11947/j.AGCS.2019.20180206.
	JI Shunping, WEI Shiqing. Building extraction via convolutional neural networks from an open remote sensing building dataset[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(4):448-459. DOI: 10.11947/j.AGCS.2019.20180206.
[9]	YUAN Jiangye. Learning building extraction in aerial scenes with convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(11):2793-2798.
[10]	MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Convolutional neural networks for large-scale remote-sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(2):645-657.
[11]	KANG Wenchao, XIANG Yuming, WANG Feng, et al. EU-net: an efficient fully convolutional network for building extraction from optical remote sensing images[J]. Remote Sensing, 2019, 11(23):2813.
[12]	何直蒙, 丁海勇, 安炳琪. 高分辨率遥感影像建筑物提取的空洞卷积E-Unet算法[J]. 测绘学报, 2022, 51(3):457-467. DOI: 10.11947/j.AGCS.2022.20200601.
	HE Zhimeng, DING Haiyong, AN Bingqi. E-Unet: a atrous convolution-based neural network for building extraction from high-resolution remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(3):457-467. DOI: 10.11947/j.AGCS.2022.20200601.
[13]	QIU Yue, WU Fang, QIAN Haizhong, et al. AFL-net: attentional feature learning network for building extraction from remote sensing images[J]. Remote Sensing, 2022, 15(1):95.
[14]	CHEN J, LI Z, LI S, et al. From digitalized to intelligentized surveying and mapping: fundamental issues and research agenda[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(2):148-160.
[15]	ZHENG X, HUAN L, XIA G S, et al. Parsing very high-resolution urban scene images by learning deep ConvNets with edge-aware loss[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 170:15-28.
[16]	YIN Jichong, WU Fang, QIU Yue, et al. A multiscale and multitask deep learning framework for automatic building extraction[J]. Remote Sensing, 2022, 14(19):4744.
[17]	GUO Haonan, DU Bo, ZHANG Liangpei, et al. A coarse-to-fine boundary refinement network for building footprint extraction from remote sensing imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 183:240-252.
[18]	LI Qingyu, SHI Yilei, HUANG Xin, et al. Building footprint generation by integrating convolution neural network with feature pairwise conditional random field (FPCRF)[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(11):7502-7519.
[19]	ZHAO K, KANG J, JUNG J, et al. Building extraction from satellite images using mask R-CNN with building boundary regularization[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE, 2018: 247-251.
[20]	HE Kaiming, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2):386-397.
[21]	WEN Qi, JIANG Kaiyu, WANG Wei, et al. Automatic building extraction from google earth images under complex backgrounds based on deep instance segmentation network[J]. Sensors, 2019, 19(2):333.
[22]	朱盼盼, 李帅朋, 张立强, 等. 基于多任务学习的高分辨率遥感影像建筑提取[J]. 地球信息科学学报, 2021, 23(3):514-523.
	ZHU Panpan, LI Shuaipeng, ZHANG Liqiang, et al. Multitask learning-based building extraction from high-resolution remote sensing images[J]. Journal of Geo-Information Science, 2021, 23(3):514-523.
[23]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[24]	DING Lei, TANG Hao, LIU Yahui, et al. Adversarial shape learning for building extraction in VHR remote sensing images[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2022, 31:678-690.
[25]	GUO Haonan, SHI Qian, MARINONI A, et al. Deep building footprint update network: a semi-supervised method for updating existing building footprint from bi-temporal remote sensing images[J]. Remote Sensing of Environment, 2021, 264:112589.
[26]	WANG Zhenqing, ZHOU Yi, WANG Futao, et al. A multi-scale edge constraint network for the fine extraction of buildings from remote sensing images[J]. Remote Sensing, 2023, 15(4):927.
[27]	DOUGLAS D H, PEUCKER T K. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature[J]. Cartographica: the International Journal for Geographic Information and Geovisualization, 1973, 10(2):112-122.
[28]	TASAR O, MAGGIORI E, ALLIEZ P, et al. Polygonization of binary classification maps using mesh approximation with right angle regularity[C]//Proceedings of 2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia: IEEE, 2018: 6404-6407.
[29]	MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Polygonization of remote sensing classification maps by mesh approximation[C]//Proceedings of 2017 IEEE International Conference on Image Processing. Beijing: IEEE, 2017: 560-564.
[30]	GIRARD N, TARABALKA Y. End-to-end learning of polygons for remote sensing image classification[C]//Proceedings of 2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia: IEEE, 2018: 2083-2086.
[31]	WEI Shiqing, JI Shunping, LU Meng. Toward automatic building footprint delineation from aerial images using CNN and regularization[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(3):2178-2189.
[32]	ZORZI S, FRAUNDORFER F. Regularization of building boundaries in satellite images using adversarial and regularized losses[C]//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE, 2019: 5140-5143.
[33]	LI M, LAFARGE F, MARLET R. Approximating shapes in images with low-complexity polygons[C]//Proceedings of 2020 IEEE Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 8630-8638.
[34]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing System. Cambridge: MIT Press, 2014: 2672-2680.
[35]	LIU Q, MENG X, SHAO F, et al. PSTAF-GAN: progressive spatio-temporal attention fusion method based on generative adversarial network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5408513.
[36]	BENZENATI T, KESSENTINI Y, KALLEL A. Pansharpening approach via two-stream detail injection based on relativistic generative adversarial networks[J]. Expert Systems with Applications, 2022, 188:115996.
[37]	TANG Meng, PERAZZI F, DJELOUAH A, et al. On regularized losses for weakly-supervised CNN segmentation[C]//Proceedings of 2018 European Conference on Computer Vision. Munich: Springer, 2018: 507-522.
[38]	TANG Meng, DJELOUAH A, PERAZZI F, et al. Normalized cut loss for weakly-supervised CNN segmentation[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 1818-1827.
[39]	CHEN Zhao, BADRINARAYANAN V, LEE Chenyu, et al. GradNorm: gradient normalization for adaptive loss balancing in deep multitask networks[C]//Proceedings of the 35th International Conference on Machine Learning. Stockholm: PMLR, 2018: 794-803.
[40]	MAGGIORI E, TARABALKA Y, CHARPIAT G, et al. Can semantic labeling methods generalize to any city？The Inria aerial image labeling benchmark[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium. Fort Worth: IEEE, 2017: 3226-3229.
[41]	XIE Yakun, ZHU Jun, CAO Yungang, et al. Refined extraction of building outlines from high-resolution remote sensing imagery based on a multifeature convolutional neural network and morphological filtering[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:1842-1855.
[42]	YE Su, PONTIUS R G J, RAKSHIT R. A review of accuracy assessment for object-based image analysis: from per-pixel to per-polygon approaches[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 141:137-147.
[43]	PERSELLO C, BRUZZONE L. A novel protocol for accuracy assessment in classification of very high resolution images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(3):1232-1244.
[44]	LIZARAZO I. Accuracy assessment of object-based image classification: another STEP[J]. International Journal of Remote Sensing, 2014, 35(16):6135-6156.

数据集	覆盖面积/km²	空间分辨率/m	样本尺寸	样本数量/张			样本形式	建筑物数量/栋	影像来源
数据集	覆盖面积/km²	空间分辨率/m	样本尺寸	训练集	测试集	验证集	样本形式	建筑物数量/栋	影像来源
Inria	810	0.3	5000	180	180	—	栅格	—	航空
WHU	450	0.3	512	4736	2416	1036	矢量/栅格	220000	航空/卫星

方法	Recall/（%）	Precision/（%）	F₁值/（%）	IoU/（%）	M_r/（%）	E_cur	E_shp
分割掩膜	94.06	93.50	93.78	88.29	84.47	5.34	4.85
方法①	91.42	94.23	93.25	87.43	83.04	2.63	5.23
方法②	93.47	94.30	93.21	87.58	84.34	3.25	3.43
方法③	93.64	94.47	93.56	87.97	84.13	3.72	3.51
本文方法	93.79	94.72	94.01	88.77	85.32	3.17	2.79

方法	Recall/（%）	Precision/（%）	F₁值/（%）	IoU/（%）	M_r/（%）	E_cur	E_shp
分割掩膜	89.09	88.77	88.93	80.06	63.76	7.65	6.45
方法①	89.17	87.89	88.53	77.41	63.25	3.27	6.87
方法②	89.53	89.51	89.52	81.03	64.46	5.34	3.84
方法③	88.47	88.16	88.32	79.09	64.18	4.86	4.66
本文方法	90.13	90.11	90.12	82.01	65.53	3.46	3.23