DRformer: a progressive coupled multiscale CNN and condensed attention Transformer method for hyperspectral image super-resolution

doi:10.11947/j.AGCS.2025.20240485

Abstract

Abstract:

The super-resolution technology of hyperspectral image aims to enhance the spatial detail and quality of low-resolution hyperspectral images for better applications in areas such as environmental monitoring. In recent years, machine learning techniques based on deep convolutional neural networks have made significant progress in single hyperspectral image super-resolution. However, challenges remain in balancing the learning of spatial multi-scale local features and global detail features. This paper presents a fusion network, DRformer, that integrates convolutional neural networks and Transformer architecture using a progressive sampling strategy. The network employs a multi-scale adaptive weighted spectral attention module for local feature extraction and selective emphasis of spectral information, followed by an initial upsampling. Subsequently, a CADR module based on the Transformer architecture is incorporated after a second upsampling to process global image features and enhance effective information. To verify the effectiveness and robustness of the network, experiments were conducted on the Chikusei and Houston2013 datasets. The results demonstrate that DRformer outperforms existing deep learning methods, including GDRRN, SSPSR, EUNet and MSDformer in terms of super-resolution performance. Additionally, ablation experiments were carried out to validate the effectiveness of each module in the network.

Key words: hyperspectral image, super-resolution, Transformer, attention mechanism

CLC Number:

P237

Qing CHENG, Boxuan WANG, Hongyan ZHANG. DRformer: a progressive coupled multiscale CNN and condensed attention Transformer method for hyperspectral image super-resolution[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(7): 1230-1242.

Figures/Tables 14

Fig. 1

Fig. 2

Fig. 3

Tab. 1

Tab. 2

Fig. 4

Fig. 5

Tab. 3

Fig. 6

Fig. 7

Tab. 4

Tab. 5

Fig. 8

Tab. 6

References 48

[1]	SABINS F F. Remote sensing for mineral exploration[J]. Ore Geology Reviews, 1999, 14(3-4): 157-183.
[2]	BACKMAN V, WALLACE M B, PERELMAN L T, et al. Detection of preinvasive cancer cells[J]. Nature, 2000, 406(6791): 35-36.
[3]	LU Guolan, FEI Baowei. Medical hyperspectral imaging: a review[J]. Journal of Biomedical Optics, 2014, 19(1): 10901.
[4]	WANG Kexian, ZHENG Shunyi, LI Rui, et al. A deep double-channel dense network for hyperspectral image classification[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(4): 46-62.
[5]	XUE Zhaohui, NIE Xiangyu. Low-rank and sparse representation with adaptive neighborhood regularization for hyperspectral image classification[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(1): 73-90.
[6]	陈超, 梁锦涛, 杨刚, 等. 面向土地覆盖精准分类的遥感特征参数优选方法[J]. 测绘学报, 2024, 53(7): 1401-1416. DOI: . doi: 10.11947/j.AGCS.2024.20230327
	CHEN Chao, LIANG Jintao, YANG Gang, et al. Remote sensing parameters optimization for accurate land cover classification[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(7): 1401-1416. DOI: . doi: 10.11947/j.AGCS.2024.20230327
[7]	叶思菁. 大数据环境下遥感图谱应用方法研究：以作物干旱监测为例[J]. 测绘学报, 2018, 47(6): 892. DOI: . doi: 10.11947/j.AGCS.2018.20170535
	YE Sijing. Research on application of remote sensing tupu-take monitoring of meteorological disaster for example[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 892. DOI: . doi: 10.11947/j.AGCS.2018.20170535
[8]	LI Jiaxin, HONG Danfeng, GAO Lianru, et al. Deep learning in multimodal remote sensing data fusion: a comprehensive review[J]. International Journal of Applied Earth Observation and Geoinformation, 2022, 112: 102926.
[9]	张兵, 高连如, 李嘉鑫, 等. 高/多光谱遥感图像超分辨率融合研究进展与展望[J]. 测绘学报, 2023, 52(7): 1074-1089. DOI: . doi: 10.11947/j.AGCS.2023.20220499
	ZHANG Bing, GAO Lianru, LI Jiaxin, et al. Advances and prospects in hyperspectral and multispectral remote sensing image super-resolution fusion[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(7): 1074-1089. DOI: . doi: 10.11947/j.AGCS.2023.20220499
[10]	JIANG Junjun, SUN He, LIU Xianming, et al. Learning spatial-spectral prior for super-resolution of hyperspectral imagery[J]. IEEE Transactions on Computational Imaging, 2020, 6: 1082-1096.
[11]	DIAN Renwei, LI Shutao, GUO Anjing, et al. Deep hyperspectral image sharpening[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(11): 5345-5355.
[12]	LIU Jianjun, WU Zebin, XIAO Liang, et al. A truncated matrix decomposition for hyperspectral image super-resolution[J]. IEEE Transactions on Image Processing, 2020, 29: 8028-8042.
[13]	DIAN Renwei, LI Shutao, FANG Leyuan. Learning a low tensor-train rank representation for hyperspectral image super-resolution[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(9): 2672-2683.
[14]	KEYS R G. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1981, 29(6): 1153-1160.
[15]	XU Yang, WU Zebin, CHANUSSOT J, et al. Hyperspectral images super-resolution via learning high-order coupled tensor ring representation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(11): 4747-4760.
[16]	ZHANG Lefei, SONG Liangchen, DU Bo, et al. Nonlocal low-rank tensor completion for visual data[J]. IEEE Transactions on Cybernetics, 2021, 51(2): 673-685.
[17]	XU Yang, WU Zebin, CHANUSSOT J, et al. Nonlocal patch tensor sparse representation for hyperspectral image super-resolution[J]. IEEE Transactions on Image Processing, 2019, 28(6): 3034-3047.
[18]	GAO Lianru, HONG Danfeng, YAO Jing, et al. Spectral super-resolution of multispectral imagery with joint sparse and low-rank learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3): 2269-2280.
[19]	DONG Chao, LOY C C, HE Kaiming, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[20]	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[EB/OL]. [2024-10-15]. https://arxiv.org/abs/1511.04587v2.
[21]	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1132-1140.
[22]	LI Yadong, MAVROMATIS S, ZHANG Feng, et al. Single-image super-resolution for remote sensing images using a deep generative adversarial network with local and global attention mechanisms[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3000224.
[23]	MENG Yapeng, LI Wenyuan, LEI Sen, et al. Large-factor super-resolution of remote sensing images with spectra-guided generative adversarial networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5634111.
[24]	WANG Junjie, ZHANG Mengmeng, LI Wei, et al. A multistage information complementary fusion network based on flexible-mixup for HSI-X image classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(12): 17189-17201.
[25]	WANG Wuli, MA Xiaohu, LENG Linchun, et al. A hybrid CNN based on global reasoning for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5.
[26]	MEI Shaohui, YUAN Xin, JI Jingyu, et al. Hyperspectral image spatial super-resolution via 3D full convolutional neural network[J]. Remote Sensing, 2017, 9(11): 1139.
[27]	LI Qiang, WANG Qi, LI Xuelong. Mixed 2D/3D convolutional network for hyperspectral image super-resolution[J]. Remote Sensing, 2020, 12(10): 1660.
[28]	FU Ying, LIANG Zhiyuan, YOU Shaodi. Bidirectional 3D quasi-recurrent neural network for hyperspectral image super-resolution[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 2674-2688.
[29]	HAUNT J M, FERNANDEZ-BELTRANN R, PAOLETTI M E, et al. Remote sensing image super-resolution using deep residual channel attention[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(11): 9277-9289.
[30]	LIU Denghong, LI Jie, YUAN Qiangqiang. A spectral grouping and attention-driven residual dense network for hyperspectral image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(9): 7711-7725.
[31]	ZHANG Jing, SHAO Minhao, WAN Zekang, et al. Multi-scale feature mapping network for hyperspectral image super-resolution[J]. Remote Sensing, 2021, 13(20): 4180.
[32]	ALEXEY D. An image is worth 16×16 words: Transformers for image recognition at scale[EB/OL]. [2024-11-25]. https://arxiv.org/abs/2010.11929.
[33]	YIN Hongxu, VAHDAT A, ALVAREZ J M, et al. A-ViT: adaptive tokens for efficient vision transformer[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 10799-10808.
[34]	ZAMIR S W, ARORA A, KHAN S, et al. Restormer: efficient transformer for high-resolution image restoration[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 5718-5729.
[35]	WANG Min, WANG Peidong. CFM-UNet: a joint CNN and transformer network via cross feature modulation for remote sensing images segmentation[J]. Journal of Geodesy and Geoinformation Science, 2023, 6(4): 40-47.
[36]	LIANG Jingyun, CAO Jiezhang, SUN Guolei, et al. SwinIR: image restoration using swin transformer[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE, 2021: 1833-1844.
[37]	LIU Yaoting, HU Jianwen, KANG Xudong, et al. Interactformer: interactive transformer and CNN for hyperspectral image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 3183468.
[38]	CHEN Xiangyu, WANG Xintao, ZHOU Jiantao, et al. Activating more pixels in image super-resolution transformer[EB/OL]. [2024-11-25]. https://arxiv.org/abs/2205.04437v3.
[39]	HE Jiang, YUAN Qiangqiang, LI Jie, et al. DsTer: a dense spectral transformer for remote sensing spectral super-resolution[J]. International Journal of Applied Earth Observation and Geoinformation, 2022, 109: 102773.
[40]	CHEN Shi, ZHANG Lefei, ZHANG Liangpei. MSDformer: multiscale deformable transformer for hyperspectral image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-14.
[41]	LI Jiaojiao, WU Chaoxiong, SONG Rui, et al. Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle: IEEE, 2020: 1894-1903.
[42]	ZHAO Haiyu, GOU Yuanbiao, LI Boyun, et al. Comprehensive and delicate: an efficient transformer for image restoration[C]//Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 14122-14132.
[43]	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[J]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2017, 7: 1132-1140.
[44]	AGGARWAL H K, MAJUMDAR A. Hyperspectral image denoising using spatio-spectral total variation[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(3): 442-446.
[45]	LI Yong, ZHANG Lei, DINGL C, et al. Single hyperspectral image super-resolution with grouped deep recursive residual network[C]//Proceedings of 2018 IEEE Fourth International Conference on Multimedia Big Data. Xi'an: IEEE, 2018: 1-4.
[46]	LIU Denghong, LI Jie, YUAN Qiangqiang, et al. An efficient unfolding network with disentangled spatial-spectral representation for hyperspectral image super-resolution[J]. Information Fusion, 2023, 94: 92-111.
[47]	WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[48]	LONCAN L, DE ALMEIDA L B, BIOUCAS-DIAS J M, et al. Hyperspectral pansharpening: a review[J]. IEEE Geoscience and Remote Sensing Magazine, 2015, 3(3): 27-46.

模块	网络架构	输入尺寸	模型参数	输出尺寸
整体结构	MAWSA	H×W×c	—	H×W×256
	Conv-UP	H×W×256	3×3，c，Stride1	sH/2×sW/2×c
	Concatenation	sH/2×sW/2×c	—	sH/2×sW/2×C
	UP	sH/2×sW/2×C	—	sH×sW×C
	CADR-Conv	sH×sW×C	1×1，256，Stride1	sH×sW×256
	Bicubic-Conv	H×W×C	3×3，256，Stride1	sH×sW×256
	Conv	sH×sW×256	3×3，C，Stride1	sH×sW×C
MAWSA	Dconv（1，3，5）-ReLU	H×W×c	3×3，256，Stride1，dilation（1，3，5）	H×W×256
	Conv-ReLU-Conv	H×W×256	3×3，256，Stride1	H×W×256
	AWCA	H×W×256	reduction=16	H×W×256
CADR	Condensed	sH×sW×C	num＿heads=8，d=16	sH×sW×C
	Attention	sH×sW×C	num＿heads=8，d=16	sH×sW×C
	DRNet	sH×sW×C		sH×sW×C
	（PReLU-Conv）		3×3，256，Stride1
	（Resblock）		3×3，256，Stride1
	（PReLU-Conv）		3×3，C，Stride1

比例因子	方法	PSNR	SSIM	CC	RMSE	ERGAS	SAM
×4	Bicubic	37.637 7	0.895 3	0.921 2	0.015 6	6.756 3	3.403 9
	GDRRN	37.721 8	0.897 0	0.922 3	0.015 4	6.697 0	3.321 5
	SSPSR	39.505 5	0.932 8	0.947 5	0.012 6	5.473 7	2.719 0
	EUNet	38.867 0	0.926 0	0.939 7	0.013 4	5.931 8	2.751 7
	MSDformer	39.067 1	0.924 5	0.942 3	0.013 2	5.714 1	2.864 4
	本文方法	40.085 1	0.933 4	0.943 3	0.012 4	5.224 1	2.697 1
×8	Bicubic	34.504 8	0.806 8	0.831 3	0.022 3	9.697 5	5.043 5
	GDRRN	34.556 4	0.807 3	0.832 7	0.022 2	9.646 9	4.991 5
	SSPSR	34.938 0	0.822 9	0.848 5	0.021 0	9.318 6	4.863 2
	EUNet	35.115 6	0.833 4	0.853 8	0.020 7	9.073 7	4.502 8
	MSDformer	34.790 2	0.816 2	0.842 2	0.021 5	9.433 1	4.823 0
	本文方法	35.913 8	0.822 3	0.810 6	0.020 6	8.594 5	4.478 9

方法	比例因子	PSNR	SSIM	CC	RMSE	ERGAS	SAM
Bicubic	×4	33.698 4	0.801 6	0.901 4	0.025 6	6.157 1	6.459 2
GDRRN	×4	33.798 3	0.806 7	0.903 2	0.025 3	6.084 1	6.342 1
SSPSR	×4	34.207 3	0.831 4	0.910 1	0.023 9	5.748 1	5.494 4
EUNet	×4	34.264 0	0.834 3	0.911 4	0.023 7	5.698 1	5.479 8
MSDformer	×4	34.077 1	0.828 8	0.908 8	0.024 2	5.849 8	5.894 3
本文方法	×4	34.477 7	0.840 1	0.915 5	0.023 1	5.567 0	5.355 8
Bicubic	×8	31.120 9	0.673 0	0.816 3	0.034 7	8.339 1	9.365 7
GDRRN	×8	31.097 6	0.674 2	0.814 0	0.034 5	8.320 1	9.368 8
SSPSR	×8	31.221 5	0.684 1	0.820 7	0.033 9	8.174 1	8.895 9
EUNet	×8	31.349 1	0.687 7	0.824 1	0.033 5	8.055 6	8.634 9
MSDformer	×8	31.305 1	0.688 2	0.823 0	0.033 6	8.108 1	9.079 3
本文方法	×8	31.350 2	0.690 4	0.824 2	0.033 5	8.120 1	8.787 1

模型	参数量/MB	时间/s
GDRRN	0.479 2	304
SSPSR	13.674 8	1828
EUNet	2.796 7	867
MSDformer	9.121 5	1535
DRformer	12.988 2	4101

变体	参数量/MB	PSNR	SSIM	CC	RMSE	ERGAS	SAM
本文方法	12.988 2	34.477 7	0.840 1	0.915 5	0.023 1	5.567 0	5.355 8
变体1	14.241 6	34.229 5	0.830 7	0.910 9	0.023 7	5.728 5	5.654 3
变体2	12.988 2	33.968 1	0.827 6	0.907 1	0.024 1	5.883 1	6.114 6
变体3	9.095 6	34.181 7	0.828 9	0.910 4	0.023 9	5.760 4	5.558 5
变体4	5.689 8	34.112 1	0.827 5	0.909 2	0.024 1	5.818 8	5.985 3