High-resolution optical images change detection based on global information enhancement by pyramid semantic token

doi:10.11947/j.AGCS.2024.20230415

Abstract

Abstract:

Due to the influence of complex background and spectral changes, missing detection of small objects and incomplete detection of geometric structures and details easily arise in remote sensing change detection (CD) domain. To address these issues, this paper proposes a pyramid semantic token guided global information enhancement change detection network (PST-GIENet) by combining the advantages of convolutional neural network (CNN) and Transformer network. Firstly, ResNet18 network without max-pooling layer is adopted to generate bi-temporal deep features, which are fused and refined by joint attention mechanism and deep supervision strategy. Secondly, image features are represented as multi-scale semantic token through spatial pyramid pooling, a Transformer encoder-decoder is subsequently employed to model the global context of the fused features. Finally, change map is produced through a layer-wise up-sampling decoder. To verify the effectiveness of the proposed method, extensive experiments and analysis were conducted on three publicly available CD datasets, including LEVIR-CD, CDD, and WHU-CD. The quantitative results showed that PST-GIENet achieved the highest metric scores in all the three datasets, with F₁ scores of 91.71%, 96.16%, and 94.08%, respectively. In addition, visual results indicate that PST-GIENet can effectively suppress the interference from complex backgrounds and spectral distortions, which significantly enhances the network's ability to capture edge structures and multi-scale changes of ground objects, achieving the best visual performance.

Key words: high-resolution remote sensing images, change detection, pyramid semantic tokens, global dependency, attention mechanism

CLC Number:

P237

Daifeng PENG, Chenchen ZHAI, Dingwei ZHOU, Yongjun ZHANG, Haiyan GUAN, Yufu ZANG. High-resolution optical images change detection based on global information enhancement by pyramid semantic token[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(6): 1195-1211.

Figures/Tables 18

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Tab.1

Tab.2

Fig.7

Fig.8

Fig.9

Tab.3

Fig.10

Tab.4

Fig.11

Tab.5

Fig.12

Tab.6

References 50

[1]	BRUZZONE L, BOVOLO F. A novel framework for the design of change-detection systems for very-high-resolution remote sensing images[J]. Proceedings of the IEEE, 2013, 101(3):609-630.
[2]	TEWKESBURY A P, COMBER A J, TATE N J, et al. A critical synthesis of remotely sensed optical image change detection techniques[J]. Remote Sensing of Environment, 2015, 160:1-14.
[3]	DEMIR B, BOVOLO F, BRUZZONE L. Updating land-cover maps by classification of image time series: a novel change-detection-driven transfer learning approach[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(1):300-312.
[4]	JIN Suming, YANG Limin, DANIELSON P, et al. A comprehensive change detection method for updating the national land cover database to circa 2011[J]. Remote Sensing of Environment, 2013, 132:159-175.
[5]	LE HÉGARAT-MASCLE S, OTTLÉC , GUÉRIN C. Land cover change detection at coarse spatial scales based on iterative estimation and previous state information[J]. Remote Sensing of Environment, 2005, 95(4):464-479.
[6]	BRUZZONE L, PRIETO D F. Automatic analysis of the difference image for unsupervised change detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2000, 38(3):1171-1182.
[7]	FALCO N, MARPU P R, BENEDIKTSSON J A. Comparison of ITPCA and IRMAD for automatic change detection using initial change mask[C]//Proceedings of 2012 IEEE International Geoscience and Remote Sensing Symposium. Munich: IEEE, 2012: 6769-6772.
[8]	WU Chen, DU Bo, CUI Xiaohui, et al. A post-classification change detection method based on iterative slow feature analysis and Bayesian soft fusion[J]. Remote Sensing of Environment, 2017, 199:241-255.
[9]	CAO Guo, LI Yupeng, LIU Yazhou, et al. Automatic change detection in high-resolution remote-sensing images by means of level set evolution and support vector machine classification[J]. International Journal of Remote Sensing, 2014, 35(16):6255-6270.
[10]	CELIK T. Unsupervised change detection in satellite images using principal component analysis and k-means clustering[J]. IEEE Geoscience and Remote Sensing Letters, 2009, 6(4):772-776.
[11]	BENEDEK C, SZIRANYI T. Change detection in optical aerial images by a multilayer conditional mixed Markov model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2009, 47(10):3416-3430.
[12]	LV Pengyuan, ZHONG Yanfei, ZHAO Ji, et al. Unsupervised change detection based on hybrid conditional random field model for high spatial resolution remote sensing imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(7):4002-4015.
[13]	ZHANG Yongjun, PENG Daifeng, HUANG Xu. Object-based change detection for VHR images based on multiscale uncertainty analysis[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(1):13-17.
[14]	GIL-YEPES J L, RUIZ L A, RECIO J A, et al. Description and validation of a new set of object-based temporal geostatistical features for land-use/land-cover change detection[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 121:77-91.
[15]	DAI Yuchao, ZHANG Jing, HE Mingyi, et al. Salient object detection from multi-spectral remote sensing images with deep residual network[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):101-110.
[16]	CHAI Junyi, ZENG Hao, LI Anming, et al. Deep learning in computer vision: a critical review of emerging techniques and application scenarios[J]. Machine Learning with Applications, 2021, 6:100134.
[17]	WANG Min, WANG Peidong. CFM-UNet: a joint CNN and transformer network via cross feature modulation for remote sensing images segmentation[J]. Journal of Geodesy and Geoinformation Science, 2023, 6(4):40-47.
[18]	YANG Yuanxi, REN Xia, WANG Jianrong. Development of integrated and intelligent geodetic and photogrammetry satellites with corresponding key technologies[J]. Journal of Geodesy and Geoinformation Science, 2023, 6(4):3-12.
[19]	MA Lei, LIU Yu, ZHANG Xueliang, et al. Deep learning in remote sensing applications: a meta-analysis and review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152:166-177.
[20]	张祖勋, 姜慧伟, 庞世燕, 等. 多时相遥感影像的变化检测研究现状与展望[J]. 测绘学报, 2022, 51(7):1091-1107. DOI:10.11947/j.AGCS.2022.20220070.
	ZHANG Zuxun, JIANG Huiwei, PANG Shiyan, et al. Review and prospect in change detection of multi-temporal remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7):1091-1107. DOI:10.11947/j.AGCS.2022.20220070.
[21]	EL AMIN A M, LIU Qingjie, WANG Yunhong. Zoom out CNNs features for optical remote sensing change detection[C]//Proceedings of the 2nd International Conference on Image, Vision and Computing. Chengdu: IEEE, 2017: 812-817.
[22]	ZHANG Hui, GONG Maoguo, ZHANG Puzhao, et al. Feature-level change detection using deep representation and feature change analysis for multispectral imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(11):1666-1670.
[23]	LEI Yu, LIU Xiaodong, SHI Jiao, et al. Multiscale superpixel segmentation with deep features for change detection[J]. IEEE Access, 2019, 7:36600-36616.
[24]	GONG Maoguo, ZHAO Jiaojiao, LIU Jia, et al. Change detection in synthetic aperture radar images based on deep neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 27(1):125-138.
[25]	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]//Proceedings of 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[26]	BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[27]	CAYE DAUDT R, LE SAUX B, BOULCH A. Fully convolutional Siamese networks for change detection[C]//Proceedings of 2018 IEEE International Conference on Image Processing. Athens: IEEE, 2018: 4063-4067.
[28]	CAYE DAUDT R, LE SAUX B, BOULCH A, et al. Multitask learning for large-scale semantic change detection[J]. Computer Vision and Image Understanding, 2019, 187:102783.
[29]	ZHOU Zongwei, SIDDIQUEE M M R, TAJBAKHSH N, et al. UNet++: a nested U-net architecture for medical image segmentation[C]//Proceedings of 2018 Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer: Cham, 2018: 3-11.
[30]	PENG Daifeng, ZHANG Yongjun, GUAN Haiyan. End-to-end change detection for high resolution satellite images using improved UNet++[J]. Remote Sensing, 2019, 11(11):1382.
[31]	王超, 王帅, 陈晓, 等. 联合UNet++和多级差分模块的多源光学遥感影像对象级变化检测[J]. 测绘学报, 2023, 52(2):283-296.DOI:10.11947/j.AGCS.2023.20220202.
	WANG Chao, WANG Shuai, CHEN Xiao, et al. Object-level change detection of multi-sensor optical remote sensing images combined with UNet++ and multi-level difference module[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(2):283-296. DOI:10.11947/j.AGCS.2023.20220202.
[32]	FANG Sheng, LI Kaiyu, SHAO Jinyuan, et al. SNUNet-CD: a densely connected Siamese network for change detection of VHR images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19:8007805.
[33]	叶沅鑫, 孙苗苗, 周亮, 等. 面向建筑物变化检测的主体边缘分解与重组神经网络[J]. 测绘学报, 2023, 52(1):71-81.DOI:10.11947/j.AGCS.2023.20210350.
	YE Yuanxin, SUN Miaomiao, ZHOU Liang, et al. Main body, edge decomposition and reorganization network for building change detection[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(1):71-81. DOI:10.11947/j.AGCS.2023.20210350.
[34]	NIU Zhaoyang, ZHONG Guoqiang, YU Hui. A review on the attention mechanism of deep learning[J]. Neurocomputing, 2021, 452:48-62.
[35]	梁哲恒, 黎宵, 邓鹏, 等. 融合多尺度特征注意力的遥感影像变化检测方法[J]. 测绘学报, 2022, 51(5):668-676.DOI:10.11947/j.AGCS.2022.20200540.
	LIANG Zheheng, LI Xiao, DENG Peng, et al. Remote sensing image change detection fusion method integrating multi-scale feature attention[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(5):668-676. DOI:10.11947/j.AGCS.2022.20200540.
[36]	ZHANG Chenxiao, YUE Peng, TAPETE D, et al. A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 166:183-200.
[37]	SHI Qian, LIU Mengxi, LI Shengchen, et al. A deeply supervised attention metric-based network and an open aerial image dataset for remote sensing change detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5604816.
[38]	CHEN Jie, YUAN Ziyang, PENG Jian, et al. DASNet: dual attentive fully convolutional Siamese networks for change detection in high-resolution satellite images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14:1194-1206.
[39]	CHEN Hao, SHI Zhenwei. A spatial-temporal attention-based method and a new dataset for remote sensing image change detection[J]. Remote Sensing, 2020, 12(10):1662.
[40]	CHEN Hao, QI Zipeng, SHI Zhenwei. Remote sensing image change detection with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5607514.
[41]	SHI Nian, CHEN Keming, ZHOU Guangyao. A divided spatial and temporal context network for remote sensing change detection[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15:4897-4908.
[42]	SONG Ze, WEI Xiaohui, KANG Xudong, et al. Toward efficient remote sensing image change detection via cross-temporal context learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61:5404210.
[43]	FENG Yuchao, XU Honghui, JIANG Jiawei, et al. ICIF-net: intra-scale cross-interaction and inter-scale feature fusion network for bitemporal remote sensing images change detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:4410213.
[44]	BANDARA W G C, PATEL V M. A transformer-based Siamese network for change detection[C]//Proceedings of 2022 IEEE International Geoscience and Remote Sensing Symposium. Kuala Lumpur: IEEE, 2022: 207-210.
[45]	ZHANG Cui, WANG Liejun, CHENG Shuli, et al. SwinSUNet: pure transformer network for remote sensing image change detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5224713.
[46]	YAN Tianyu, WAN Zifu, ZHANG Pingping. Fully transformer network for change detection of remote sensing images[C]//Proceedings of the 16th Asian Conference on Computer Vision. Macao: ACM Press, 2022: 1691-1708.
[47]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of 2018 European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 3-19.
[48]	LEE C Y, XIE S, GALLAGHER P, et al. Deeply-supervised nets[C]//Proceedings of 2015 Conference on Artificial Intelligence and Statistics. [S.l.]: PMLR, 2015: 562-570.
[49]	LEBEDEV M A, VIZILTER Y V, VYGOLOV O V, et al. Change detection in remote sensing images using conditional adversarial networks[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, XLII-2:565-571.
[50]	JI Shunping, WEI Shiqing, LU Meng. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(1):574-586.

参数	LEVIR-CD数据集	CDD数据集	WHU-CD数据集
优化器	AdamW	AdamW	Adam W
训练轮数	100	100	100
批大小	8	8	8
初始学习率	0.000 2	0.000 2	0.000 1

方法	LEVIR-CD					CDD					WHU-CD
方法	P	R	F₁值	IoU	OA	P	R	F₁值	IoU	OA	P	R	F₁值	IoU	OA
FC-EF^[27]	87.57	78.14	82.58	70.34	96.65	76.28	71.01	73.55	58.17	93.98	81.23	73.30	77.06	62.68	98.31
FC-Siam-Conc^[27]	90.96	82.96	86.78	76.64	97.35	86.21	82.36	84.24	72.77	96.36	47.88	82.84	60.69	43.56	95.85
FC-Siam-Diff^[27]	91.44	79.92	85.29	74.35	97.01	83.88	79.27	81.51	68.79	95.76	79.33	72.40	75.70	60.91	98.20
UNet++_MSOF^[30]	89.08	85.37	87.19	77.28	97.31	89.36	87.22	88.27	79.00	97.27	88.96	82.27	85.48	74.65	98.92
SNUNet^[32]	89.98	87.63	88.79	79.83	97.79	95.45	95.14	95.29	91.01	98.89	88.58	88.68	88.63	79.58	99.12
STANet^[39]	85.00	91.40	88.10	78.70	98.70	88.00	94.30	91.10	83.60	97.80	84.92	88.57	86.71	76.53	98.95
DSIFN^[36]	93.30	86.21	89.61	81.18	97.80	88.09	96.22	91.97	85.14	98.03	88.39	83.48	85.86	75.22	98.94
BIT^[40]	90.50	89.42	89.96	81.75	98.89	95.15	92.41	93.76	88.25	98.55	90.96	89.32	90.13	82.04	99.24
ICIF-Net^[43]	91.80	88.48	90.11	82.00	99.11	95.40	92.72	94.04	88.75	98.62	93.01	85.47	88.90	80.11	98.99
ChangeFormer^[44]	92.05	88.81	90.40	82.48	99.04	95.32	95.50	95.41	91.22	98.92	91.59	87.72	89.62	81.18	99.21
FTNet^[46]	92.71	89.37	91.01	83.51	99.06	92.00	77.22	83.97	72.37	96.03	95.43	89.33	92.28	85.67	99.24
PST-GIENet	92.32	91.11	91.71	84.69	99.16	95.86	96.47	96.16	92.60	99.05	95.39	92.79	94.08	88.81	99.41

方法	F₁值/（%）	参数量	测试时间/s
FC-EF^[27]	82.58	1.35	34
FC-Siam-Conc^[27]	86.78	1.55	38
FC-Siam-Diff^[27]	85.29	1.35	38
UNet++_MSOF^[30]	87.19	9.05	65
SNUNet^[32]	88.79	12.03	245
DSIFN^[36]	88.10	50.44	117
STANet^[39]	89.61	16.93	64
BIT^[40]	89.96	3.49	63
ICIF-Net^[43]	90.11	23.84	88
ChangeFormer^[44]	90.40	41.01	140
FTNet^[46]	91.01	164.45	375
PST-GIENet	91.71	28.60	67

方法	LEVIR-CD	CDD	WHU-CD
方法	F₁值IoU	F₁值IoU	F₁值IoU
w/o JAM	91.50 84.34	95.90 92.12	93.28 87.41
DSw/o	91.58 84.46	96.04 92.39	93.41 87.64
w/o TS	91.37 84.11	95.08 90.61	92.81 86.59
w/Maxpool	90.82 83.19	95.28 90.99	92.23 85.58
PST-GIENet	91.71 84.69	96.16 92.60	94.08 88.81

ED	DD	LEVIR-CD		CDD		WHU-CD
ED	DD	F₁值	IoU	F₁值	IoU	F₁值	IoU
0	0	91.37	84.11	95.08	90.61	92.81	86.59
1	1	91.67	83.94	95.55	91.47	93.86	88.43
2	1	91.40	84.16	95.58	91.54	92.70	86.39
4	1	91.36	84.10	95.05	90.56	92.02	85.22
8	1	91.29	83.97	95.36	91.28	93.90	88.51
1	2	91.60	84.51	95.59	91.54	93.56	87.90
1	4	91.71	84.69	95.99	92.29	93.55	87.88
1	8	91.70	84.67	96.16	92.60	94.08	88.81