Spatio-temporal fusion algorithm based on adaptive reference feature incorporation and multi-scale feature aggregation

doi:10.11947/j.AGCS.2025.20240457

Abstract

Abstract:

The purpose of spatio-temporal fusion algorithm is to generate dense time series images with high spatial resolution, which is very important for monitoring fine dynamic-changes of the surface. Most of the spatio-temporal fusion algorithms help to predict the target fine image by reference fine image in the adjacent time, which makes the purpose of spatio-temporal fusion algorithm is to generate dense time series images with high spatial resolution, which is very important for monitoring fine dynamic-changes of the surface. However, the existing spatio-temporal fusion algorithms are easily misled by the reference image in the area with land cover change, and the reconstruction of heterogeneous areas composed of small targets is more difficult. To this end, this paper proposes a spatio-temporal fusion algorithm based on adaptive reference feature incorporation and multi-scale feature aggregation. In the encoding stage, an adaptive reference feature incorporation module is designed. According to the change information provided by the time series coarse image pair and the gating structure, the adaptive introduction of the reference fine image is realized, which not only improves the prediction accuracy by using the reference information, but also suppresses the misleading of the reference information to the change area. In the decoder stage, a multi-scale feature aggregation strategy is designed to aggregate information of different scales for each layer of the decoder, and the channel attention mechanism is combined to filter information with important features to improve the reconstruction accuracy of heterogeneous areas. Finally, the focal frequency loss term is introduced into the loss function. From the perspective of frequency distribution, it enhances the authenticity of the generated image and focuses on the reconstruction of difficult frequency bands to make up for the deficiency of spatial spectrum loss. The experimental results on LGC, CIA and Wuhan datasets show that the proposed algorithm has better fusion results than the other six algorithms.

Key words: spatio-temporal fusion, deep learning, adaptive reference feature incorporation, multi-scale feature aggregation, focal frequency loss

CLC Number:

P237

Shuai FANG, Jiaen LIU, Jing ZHANG. Spatio-temporal fusion algorithm based on adaptive reference feature incorporation and multi-scale feature aggregation[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(8): 1476-1488.

Figures/Tables 14

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Tab. 1

Fig. 6

Tab. 2

Fig. 7

Tab. 3

Tab. 4

Tab. 5

Fig. 8

Tab. 6

References 33

[1]	张良培, 何江, 杨倩倩, 等. 数据驱动的多源遥感信息融合研究进展[J]. 测绘学报, 2022, 51(7): 1317-1337. DOI: . doi: 10.11947/j.AGCS.2022.20220171
	ZHANG Liangpei, HE Jiang, YANG Qianqian, et al. Data-driven multi-source remote sensing data fusion: progress and challenges[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7): 1317-1337. DOI: . doi: 10.11947/j.AGCS.2022.20220171
[2]	LI Jun, LI Yunfei, HE Lin, et al. Spatio-temporal fusion for remote sensing data: an overview and new benchmark[J]. Science China Information Sciences, 2020, 63(4): 140301.
[3]	ZHU Xiaolin, CAI Fangyi, TIAN Jiaqi, et al. Spatiotemporal fusion of multisource remote sensing data: literature survey, taxonomy, principles, applications, and future directions[J]. Remote Sensing, 2018, 10(4): 527.
[4]	黄波, 赵涌泉. 多源卫星遥感影像时空融合研究的现状及展望[J]. 测绘学报, 2017, 46(10): 1492-1499. DOI: . doi: 10.11947/j.AGCS.2017.20170376
	HUANG Bo, ZHAO Yongquan. Research status and prospect of spatiotemporal fusion of multi-source satellite remote sensing imagery[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(10): 1492-1499. DOI: . doi: 10.11947/j.AGCS.2017.20170376
[5]	刘建波, 马勇, 武易天, 等. 遥感高时空融合方法的研究进展及应用现状[J]. 遥感学报, 2016, 20(5): 1038-1049.
	LIU Jianbo, MA Yong, WU Yitian, et al. Review of methods and applications of high spatiotemporal fusion of remote sensing data[J]. Journal of Remote Sensing, 2016, 20(5): 1038-1049.
[6]	GAO Feng, MASEK J, SCHWALLER M, et al. On the blending of the Landsat and MODIS surface reflectance: predicting daily Landsat surface reflectance[J]. IEEE Transactions on Geoscience and Remote sensing, 2006, 44(8): 2207-2218.
[7]	ZHU Xiaolin, CHEN Jin, GAO Feng, et al. An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions[J]. Remote Sensing of Environment, 2010, 114(11): 2610-2623.
[8]	ZHU Xiaolin, HELMER E H, GAO Feng, et al. A flexible spatiotemporal method for fusing satellite images with different resolutions[J]. Remote Sensing of Environment, 2016, 172: 165-177.
[9]	XU Chen, DU Xiaoping, YAN Zhenzhen, et al. VSDF: a variation-based spatiotemporal data fusion method[J]. Remote Sensing of Environment, 2022, 283: 113309.
[10]	皮新宇, 曾永年, 王盼成. 面向非均质区域的空间增强型时空融合模型研究[J]. 测绘学报, 2023, 52(10): 1714-1723. DOI: . doi: 10.11947/j.AGCS.2023.20220519
	PI Xinyu, ZENG Yongnian, WANG Pancheng. Spatially enhanced spatio-temporal fusion model for heterogeneity regions[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(10): 1714-1723. DOI: . doi: 10.11947/j.AGCS.2023.20220519
[11]	HUANG Bo, SONG Huihui. Spatiotemporal reflectance fusion via sparse representation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50(10): 3707-3716.
[12]	SONG Huihui, LIU Qingshan, WANG Guojie, et al. Spatiotemporal satellite image fusion using deep convolutional neural networks[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3): 821-829.
[13]	TAN Zhenyu, YUE Peng, DI Liping, et al. Deriving high spatiotemporal remote sensing images using deep convolutional network[J]. Remote Sensing, 2018, 10(7): 1066.
[14]	TAN Zhenyu, DI Liping, ZHANG Mingda, et al. An enhanced deep convolutional model for spatiotemporal image fusion[J]. Remote Sensing, 2019, 11(24): 2898.
[15]	LIU Xun, DENG Chenwei, CHANUSSOT J, et al. StfNet: a two-stream convolutional neural network for spatiotemporal image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(9): 6552-6564.
[16]	YIN Zhixiang, WU Penghai, FOODY G M, et al. Spatiotemporal fusion of land surface temperature based on a convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(2): 1808-1822.
[17]	ZHANG Hongyan, SONG Yiyao, HAN Chang, et al. Remote sensing image spatiotemporal fusion using a generative adversarial network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(5): 4273-4286.
[18]	SONG Bingze, LIU Peng, LI Jun, et al. MLFF-GAN: a multilevel feature fusion with GAN for spatiotemporal remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3169916.
[19]	SHANG Cheng, LI Xinyan, YIN Zhixiang, et al. Spatiotemporal reflectance fusion using a generative adversarial network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5400915.
[20]	SONG Yiyao, ZHANG Hongyan, HUANG He, et al. Remote sensing image spatiotemporal fusion via a generative adversarial network with one prior image pair[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5528117.
[21]	TAN Zhenyu, GAO Meiling, LI Xinghua, et al. A flexible reference-insensitive spatiotemporal fusion model for remote sensing images using conditional generative adversarial network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3050551.
[22]	CHEN Jia, WANG Lizhe, FENG Ruyi, et al. CycleGAN-STF: spatiotemporal fusion via CycleGAN-based image generation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 5851-5865.
[23]	CHEN Guanyu, JIAO Peng, HU Qing, et al. SwinSTFM: remote sensing spatiotemporal fusion using swin transformer[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3182809.
[24]	LIN Liupeng, SHEN Yao, WU Jingan, et al. CAFE: a cross-attention based adaptive weighting fusion network for MODIS and Landsat spatiotemporal fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2023, 20: 3286463.
[25]	JIANG Hao, QIAN Yurong, YANG Guangqi, et al. MLKNet: multi-stage for remote sensing image spatiotemporal fusion network based on a large kernel attention[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024, 17: 1257-1268.
[26]	MA Yaobin, WANG Qi, WEI Jingbo. Spatiotemporal fusion via conditional diffusion model[J]. IEEE Geoscience and Remote Sensing Letters, 2024, 21: 5002405.
[27]	CHEN Guangsheng, LU Hailiang, DI Donglin, et al. StfMLP: spatiotemporal fusion multilayer perceptron for remote-sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2023, 20: 3230720.
[28]	XIAO Juan, AGGARWAL A K, DUC N H, et al. A review of remote sensing image spatiotemporal fusion: challenges, applications and recent trends[J]. Remote Sensing Applications: Society and Environment, 2023, 32: 101005.
[29]	HU Jie, SHEN Li, SUN Gang. Squeeze-and-excitation networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
[30]	JIANG Liming, DAI Bo, WU Wayue, et al. Focal frequency loss for image reconstruction and synthesis[C]//Proceedings of 2021 International Conference on Computer Vision. Montreal: IEEE, 2021: 13899-13909.
[31]	EMELYANOVA I V, MCVICAR T R, VAN NIEL T G, et al. Assessing the accuracy of blending Landsat-MODIS surface reflectances in two landscapes with contrasting spatial and temporal dynamics: a framework for algorithm selection[J]. Remote Sensing of Environment, 2013, 133: 193-209.
[32]	ZHANG Xingjian, XIE Linglin, LI Shuang, et al. Wuhan dataset: a high-resolution dataset of spatiotemporal fusion for remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2024, 21: 3432285.
[33]	ILYA L, FRANK H. SGDR: stochastic gradient descent with warm restarts[C]//Proceedings of 2017 International Conference on Learning Representations. Toulon: [s.n.], 2017.

算法	RMSE↓	SSIM↑	SAM↓	CC↑
STARFM	0.040 9	0.757 0	0.210 0	0.820 7
FSDAF	0.040 2	0.759 5	0.211 8	0.829 4
STFDCNN	0.033 5	0.806 9	0.160 8	0.873 9
EDCSTFN	0.032 7	0.811 3	0.159 9	0.880 8
GAN-STFM	0.033 4	0.811 7	0.164 5	0.878 7
MLFF-GAN	0.028 0	0.830 6	0.135 1	0.913 5
本文算法	0.026 5	0.847 9	0.123 6	0.923 2

算法	RMSE↓	SSIM↑	SAM↓	CC↑
STARFM	0.027 8	0.843 3	0.074 1	0.961 4
FSDAF	0.027 2	0.840 4	0.072 1	0.962 9
STFDCNN	0.026 5	0.854 5	0.067 2	0.963 6
EDCSTFN	0.023 9	0.871 0	0.060 0	0.970 4
GAN-STFM	0.026 4	0.858 2	0.072 4	0.964 0
MLFF-GAN	0.023 8	0.863 3	0.062 6	0.970 8
本文算法	0.023 1	0.874 8	0.059 1	0.972 6

算法	RMSE↓	SSIM↑	SAM↓	CC↑
STARFM	0.044 7	0.744 9	0.129 7	0.873 6
FSDAF	0.046 0	0.733 4	0.146 7	0.835 3
STFDCNN	0.018 6	0.893 5	0.064 1	0.955 9
EDCSTFN	0.018 5	0.901 9	0.074 6	0.962 0
GAN-STFM	0.019 5	0.888 6	0.084 3	0.954 2
MLFF-GAN	0.028 6	0.809 8	0.091 9	0.893 5
本文算法	0.014 7	0.918 6	0.048 1	0.971 6

数据集	评价指标	去除AFIM	去除MFAM	去除FFL	完整算法结构
LGC	RMSE↓	0.027 7	0.026 7	0.026 5	0.026 5
	SSIM↑	0.839 0	0.844 3	0.846 5	0.847 9
	SAM↓	0.130 4	0.127 9	0.124 6	0.123 6
	CC↑	0.916 8	0.921 2	0.923 0	0.923 2
CIA	RMSE↓	0.023 3	0.023 8	0.023 4	0.023 1
	SSIM↑	0.870 8	0.868 1	0.871 8	0.874 8
	SAM↓	0.061 7	0.062 2	0.059 9	0.059 1
	CC↑	0.972 3	0.970 7	0.972 1	0.972 6
Wuhan	RMSE↓	0.015 7	0.016 3	0.015 5	0.014 7
	SSIM↑	0.888 9	0.905 0	0.913 9	0.918 6
	SAM↓	0.057 8	0.055 6	0.052 8	0.048 1
	CC↑	0.968 8	0.965 9	0.969 3	0.971 6

数据集	评价指标	GANSTFM	FFLSTFM	MLFFGAN	MLFFFFL
LGC	RMSE↓	0.033 4	0.033 2	0.028 0	0.027 0
	SSIM↑	0.811 7	0.812 9	0.830 6	0.838 5
	SAM↓	0.164 5	0.159 4	0.135 1	0.130 8
	CC↑	0.878 7	0.880 1	0.913 5	0.919 0
CIA	RMSE↓	0.026 4	0.025 2	0.023 8	0.023 8
	SSIM↑	0.858 2	0.861 6	0.863 3	0.867 6
	SAM↓	0.072 4	0.063 6	0.062 6	0.061 9
	CC↑	0.964 0	0.967 2	0.970 8	0.971 0
Wuhan	RMSE↓	0.019 5	0.016 2	0.028 6	0.026 7
	SSIM↑	0.888 6	0.893 1	0.809 8	0.817 3
	SAM↓	0.084 3	0.060 9	0.091 9	0.084 2
	CC↑	0.954 2	0.961 7	0.893 5	0.906 8