Weakly supervised building change detection integrating multi-scale feature fusion and spatial refinement for high resolution remote sensing images

doi:10.11947/j.AGCS.2024.20230118

Abstract

Abstract:

To alleviate the heavy dependence of deep learning methods on large-scale high-cost pixel-level annotations, in this paper, we propose a novel weakly supervised method, named MDF-LSR-Net, for high-resolution remote sensing building change detection. Specifically, the proposed method first designs a multi-scale difference feature aggregation module to make better use of multi-scale difference features to generate change heatmaps. Then, by utilizing the local spatial consistency of the low-level fused difference features, MDF-LSR-Net presents a local spatial refinement module to enhance the integrity and accuracy of change regions in heatmaps. Finally, the change detection model is trained based on the high-quality change heatmaps. Experimental results on publicly available datasets, including WHU and LEVIR, demonstrate that our proposed method can obtain more integral and accurate change heatmaps, leading to significantly improved detection performance of the final change detection model. The final model has achieved over 65% points in IOU and over 79% points in F₁ on the WHU dataset.

Key words: high-resolution remote sensing imagery, building change detection, deep learning, weakly supervised learning, multi-scale feature fusion

CLC Number:

P237

Xin YAN, Li SHEN, Junjie PAN, Yanshuai DAI, Jicheng WANG, Xiaoli ZHENG, Zhi-lin LI. Weakly supervised building change detection integrating multi-scale feature fusion and spatial refinement for high resolution remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(8): 1586-1597.

Figures/Tables 8

Fig.1

Tab.1

Fig.2

Tab.2

Fig.3

Tab.3

Fig.4

Fig.5

References 36

[1]	胡明洪, 李佳田, 姚彦吉, 等. 结合多路径的高分辨率遥感影像建筑物提取SER-UNet算法[J]. 测绘学报, 2023, 52(5): 808-817. DOI: 10.11947/j.AGCS.2023.20210691.
	HU Minghong, LI Jiatian, YAO Yanji, et al. SER-UNet algorithm for building extraction from high-resolution remote sensing image combined with multipath[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(5): 808-817. DOI: 10.11947/j.AGCS.2023.20210691.
[2]	刘帅, 李笑迎, 于梦, 等. 高分辨率遥感图像双解耦语义分割网络模型[J]. 测绘学报, 2023, 52(4): 638-647. DOI: 10.11947/j.AGCS.2023.20210455.
	LIU Shuai, LI Xiaoying, YU Meng, et al. Dual decoupling semantic segmentation model for high-resolution remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(4): 638-647. DOI: 10.11947/j.AGCS.2023.20210455.
[3]	乔文凡, 慎利, 戴延帅, 等. 联合膨胀卷积残差网络和金字塔池化表达的高分影像建筑物自动识别[J]. 地理与地理信息科学, 2018, 34(5): 56-62.
	QIAO Wenfan, SHEN Li, DAI Yanshuai, et al. Building extraction from high resolution remote sensing images by combining dilated convolutional residual networks and pyramid pooling representation[J]. Geography and Geo-Information Science, 2018, 34(5): 56-62.
[4]	吴纹辉, 慎利, 董新丰, 等. 面向高分辨率遥感影像建筑物变化检测的边缘感知网络[J]. 地理与地理信息科学, 2021, 37(3): 21-28.
	WU Wenhui, SHEN Li, DONG Xinfeng, et al. Edge sensing network for building change detection in high resolution remote sensing images[J]. Geography and Geo-Information Science, 2021, 37(3): 21-28.
[5]	吴纹辉, 雷添杰. 基于孪生网络和典型语义分割模型的遥感影像变化检测方法框架研究[J]. 国土资源信息化, 2021(2): 15-21.
	WU Wenhui, LEI Tianjie. A framework research of remote sensing image change detection method based on Siamese network and typical semantic segmentation models[J]. Land and Resources Informatization, 2021(2): 15-21.
[6]	LIU Yi, PANG Chao, ZHAN Zongqian, et al. Building change detection for remote sensing images using a dual-task constrained deep Siamese convolutional network model[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(5): 811-815.
[7]	叶沅鑫, 孙苗苗, 周亮, 等. 面向建筑物变化检测的主体边缘分解与重组神经网络[J]. 测绘学报, 2023, 52(1): 71-81. DOI: 10.11947/j.AGCS.2023.20210350.
	YE Yuanxin, SUN Miaomiao, ZHOU Liang, et al. Main body, edge decomposition and reorganization network for building change detection[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(1): 71-81. DOI: 10.11947/j.AGCS.2023.20210350.
[8]	张志强, 张新长, 辛秦川, 等. 结合像元级和目标级的高分辨率遥感影像建筑物变化检测[J]. 测绘学报, 2018, 47(1): 102-112. DOI: 10.11947/j.AGCS.2018.20170483.
	ZHANG Zhiqiang, ZHANG Xinchang, XIN Qinchuan, et al. Combining the pixel-based and object-based methods for building change detection using high-resolution remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(1): 102-112. DOI: 10.11947/j.AGCS.2018.20170483.
[9]	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]//Proceedings of 2015 International Conference on Medical Image Computing and Computer-assisted intervention. Cham: Springer, 2015: 234-241.
[10]	CHEN L C, ZHU Yukun, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of 2018 European Conference on Computer Vision. Cham: Springer, 2018: 833-851.
[11]	JIANG Huiwei, HU Xiangyun, LI Kun, et al. PGA-SiamNet: pyramid feature-based attention-guided Siamese network for remote sensing orthoimagery building change detection[J]. Remote Sensing, 2020, 12(3): 484.
[12]	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 618-626.
[13]	MCEVER R A, MANJUNATH B S. PCAMs: weakly supervised semantic segmentation using point supervision[EB/OL]. [2023-02-02]. http://arxiv.org/abs/2007.05615.
[14]	LEE J, YI Jihun, SHIN C, et al. BBAM: bounding box attribution map for weakly supervised semantic and instance segmentation[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 2643-2651.
[15]	ARASLANOV N, ROTH S. Single-stage semantic segmentation from image labels[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020.
[16]	WANG Yude, ZHANG Jie, KAN Meina, et al. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 12272-12281.
[17]	AHN J, CHO S, KWAK S. Weakly supervised learning of instance segmentation with inter-pixel relations[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 2204-2213.
[18]	李鑫伟, 李彦胜, 张永军. 弱监督深度语义分割网络的多源遥感影像水体检测[J]. 中国图象图形学报, 2021, 26(12): 3015-3026.
	LI Xinwei, LI Yansheng, ZHANG Yongjun. Weakly supervised deep semantic segmentation network for water body extraction based on multi-source remote sensing imagery[J]. Journal of Image and Graphics, 2021, 26(12): 3015-3026.
[19]	YAN Xin, SHEN Li, WANG Jicheng, et al. MSG-SR-Net: a weakly supervised network integrating multiscale generation and superpixel refinement for building extraction from high-resolution remotely sensed imageries[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 1012-1023.
[20]	CHEN Jie, HE Fen, ZHANG Yi, et al. SPMF-Net: weakly supervised building segmentation by combining superpixel pooling and multi-scale feature fusion[J]. Remote Sensing, 2020, 12(6): 1049.
[21]	LI Zhenshi, ZHANG Xueliang, XIAO Pengfeng, et al. On the effectiveness of weakly supervised semantic segmentation for building extraction from high-resolution remote sensing imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 3266-3281.
[22]	WEI Yao, JI Shunping. Scribble-based weakly supervised deep learning for road surface extraction from remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5602312.
[23]	LIAN Renbao, HUANG Liqin. Weakly supervised road segmentation in high-resolution remote sensing images using point annotations[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 4501013.
[24]	LU Ming, FANG Leyuan, LI Muxing, et al. NFANet: a novel method for weakly supervised water extraction from high-resolution remote-sensing imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5617114.
[25]	FU Kun, LU Wanxuan, DIAO Wenhui, et al. WSF-NET: weakly supervised feature-fusion network for binary segmentation in remote sensing image[J]. Remote Sensing, 2018, 10(12): 1970.
[26]	KHAN S, HE Xuming, PORIKLI F, et al. Learning deep structured network for weakly supervised change detection[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence. Melbourne: International Joint Conferences on Artificial Intelligence Organization, 2017: 2008-2015.
[27]	ANDERMATT P, TIMOFTE R. A weakly supervised convolutional network for change segmentation and classification[C]//Proceedings of the 15th Asian Conference on Computer Vision. Cham: Springer, 2020: 103-119.
[28]	KALITA I, KARATSIOLIS S, KAMILARIS A. Land use change detection using deep siamese neural networks and weakly supervised learning[C]//Proceedings of the 19th International Conference. Cham: Springer, 2021: 24-35.
[29]	WU Chen, DU Bo, ZHANG Liangpei. Fully convolutional change detection framework with generative adversarial network for unsupervised, weakly supervised and regional supervised change detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 9774-9788.
[30]	HUANG Rui, WANG Ruofei, GUO Qing, et al. Background-mixed augmentation for weakly supervised change detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(7): 7919-7927.
[31]	王超, 王帅, 陈晓, 等. 联合UNet++和多级差分模块的多源光学遥感影像对象级变化检测[J]. 测绘学报, 2023, 52(2): 283-296. DOI: 10.11947/j.AGCS.2023.20220202.
	WANG Chao, WANG Shuai, CHEN Xiao, et al. Object-level change detection of multi-sensor optical remote sensing images combined with UNet++and multi-level difference module[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(2): 283-296. DOI: 10.11947/j.AGCS.2023.20220202.
[32]	JI Shunping, WEI Shiqing, LU Meng. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(1): 574-586.
[33]	CHEN Hao, SHI Zhenwei. A spatial-temporal attention-based method and a new dataset for remote sensing image change detection[J]. Remote Sensing, 2020, 12(10): 1662.
[34]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[35]	DURAND T, MORDAN T, THOME N, et al. WILDCAT: weakly supervised learning of deep ConvNets for image classification, pointwise localization and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5957-5966.
[36]	FAN Junsong, ZHANG Zhaoxiang, TAN Tieniu. Employing multi-estimations for weakly-supervised semantic segmentation[C]//Proceedings 2020 of ECCV. Cham: Springer, 2020: 332-348.

方法	LEVIR数据集				WHU数据集
方法	P	R	F₁值	IOU	P	R	F₁值	IOU
基准方法	38.95	36.16	37.50	23.08	66.27	66.71	66.49	49.80
MDF_base	61.94	60.10	61.01	43.89	87.52	65.84	75.15	60.19
MDF	69.29	71.36	70.31	54.21	83.88	74.38	78.85	65.08
本文方法	69.91	72.35	71.11	55.17	82.91	79.37	81.10	68.21

方法	LEVIR数据集				WHU数据集
方法	P	R	F₁值	IOU	P	R	F₁值	IOU
WILDCAT	31.97	71.03	44.10	28.28	54.42	69.63	61.09	43.98
SEAM	56.89	74.11	64.37	47.46	78.40	73.84	76.05	61.36
WSF－Net	51.51	79.38	62.48	44.65	67.54	81.25	73.76	58.43
MSG－SR－Net	59.28	77.33	67.11	50.50	76.62	77.99	77.30	62.99
本文方法	69.91	72.35	71.11	55.17	82.91	79.37	81.10	68.21

方法	LEVIR数据集				WHU数据集
方法	P	R	F₁值	IOU	P	R	F₁值	IOU
WILDCAT	38.10	46.88	42.04	26.61	56.58	52.91	54.68	37.63
SEAM	49.58	82.77	62.01	44.94	58.56	83.74	68.93	52.59
WSF－Net	48.87	66.61	56.38	39.26	60.53	74.86	66.93	50.30
MSG－SR－Net	52.14	83.98	64.34	47.42	64.60	79.10	71.12	55.18
本文方法	53.08	86.06	65.66	48.88	80.18	78.48	79.32	65.73